Architectural Overview of Atlassian's Next-Generation Data Lakehouse

We are rebuilding our data lakehouse from the ground up - a greenfield re-design based on everything we have learned from the last 5 years of working with Databricks. In this talk, we’ll give an architectural overview of how we’re setting up this new lake, covering topics like: How we are laying out our Databricks accounts/workspaces and AWS accounts to create the concept of “environments” Our “workbench environment” concept that separates insights work from production pipelines The benefit we get from doubling down on Unity Catalog, Delta Lake, and Managed Tables Creation and enforcement of a consistent information architecture Making our data lakehouse declarative, and keeping human users out of the production environment Supporting and governing machine learning workloads

Talk By: Chen Zhou, Senior Data Platform Engineer, Atlassian ; Perry Stephenson, Principal Data Platform Engineer, Atlassian

Here's more to explore:
Rise of the Data Lakehouse: https://dbricks.co/3NHT7CD
Lakehouse Fundamentals Training: https://dbricks.co/44ancQs

Connect with us: Website: https://databricks.com
Twitter:   / databricks  
LinkedIn:   / data…  
Instagram:   / databricksinc  
Facebook:   / databricksinc