Manage Data Quality at Scale with Dataplex and Gemini

Interested in a solution to Manage Data Quality at Scale with Dataplex and Gemini?

Data quality is non-negotiable, especially in the age of GenAI, AI Agents 🤖, and Function Calling. Traditional third-party observability tools often present challenges: increased risk ⚠️, lengthy procurement ⏳, and loss of control over your data 🔒.

In my latest video, I dive into a powerful solution: combining Dataplex's Auto DQ with Gemini to manage data quality at scale.

Here's what this video covers:

00:29 - Contents & Intro
01:48 - Why Data Quality Is Even More Important Today
03:12 - Problem with Existing Solutions
06:12 - What is Dataplex Auto DQ and Why Use It
✨ 06:12 - The 7 Data Quality Dimensions
✨ 07:12 - Define Data Quality Rules
09:34 - A Powerful Solution Using Dataplex & Gemini
✨ 09:34 - High Level Design
✨ 13:00 - Low Level Design (i.e. Rules naming convention + Terraform module)
15:04 - Code WalkThrough & Live Demo
✨ 15:04 - Code Walkthrough
✨ 22:56 - Rule Specs Documentation (important & hard to find)
✨ 24:09 - Live Demo, DQ Features
✨ 34:40 - Reverse Engineer YAML
38:49 - Outstanding Issues ❗
42:17 - Summary & Future Thoughts

Watch the full video to learn how to build a robust data quality framework and ensure your data foundation is rock-solid!

Code Repo: https://github.com/richardhe-fundamen...

Slides: https://drive.google.com/file/d/1heEa...

NEW: See the new video on how to build a powerful data quality agent to get to these insights in seconds!    • A Powerful Data Quality Agent for Dataplex