Modern data stack project - introduction and pipeline overview

Introduction to the data engineering project that we'll build with the modern data stack

Repo
https://github.com/AIgineerAB/modern_...

This is a a video in a series where I build a modern data stack project that uses

dlt to extract load (EL) data into a duckdb which acts as a data warehouse
duckdb data warehouse is divided into staging layer, refined layer and marts layer
a star schema is created using dbdiagram which will act as mental model for the transformations
dbt is used to transform data from staging into refined layer and marts
the marts layer is used to serve streamlit dashboard
in the streamlit dashboard LLM will be connected as well as visualizations and KPIs
then everything will be orchestrated using dagster
docker will be used to dockerize the whole project
the dockerized project is deployed into azure