Spotify Real-Time Data Analysis | End-to-End Data Engineering Project

🔥 Spotify Real-Time Data Engineering Project with Snowflake, Power BI & DBT! 🔥

In this project, we’ll build a real-time Spotify data analysis pipeline — from streaming raw music data to creating a fully automated dashboard.
The best part? Once it starts, every component runs on its own — from data ingestion all the way to dashboard updates — just like a production-grade system.

We’ll design and implement an end-to-end pipeline following the Medallion Architecture (Bronze, Silver, Gold) — processing data in Snowflake and visualizing it with Power BI for rich insights.
This is a portfolio-ready project that demonstrates your expertise in data engineering, automation, and analytics.

✅ What You’ll Learn:

• Streaming raw Spotify data in real-time
• Building a data ingestion layer with Python
• Storing and managing data in Snowflake
• Transforming data using Medallion Architecture (Bronze, Silver, Gold) with DBT (Data Build Tool)
• Automating workflows for data refresh
• Connecting Snowflake to Power BI
• Designing interactive dashboards for insights

🛠 Tools & Tech Stack:
• Docker
• Snowflake
• Power BI
• DBT (Data Build Tool)
• SQL
• VS Code
• Python

📂 Project Repository:
GitHub: https://github.com/Jay61616/spotify-m...

📌 Tools Links:
Snowflake: https://signup.snowflake.com/
Docker: https://www.docker.com/products/docke...
VS code: https://code.visualstudio.com/
Power Bi: https://download.microsoft.com/downlo...

⏱ Timestamps:
0:00 intro
1:00 Architecture
5:20 Tools Required
9:55 Initialize Repo
16:15 Docker Container Creation
28:50 Data Generation
50:31 Kafka to Minio
1:13:19 Minio to Snowflake (Airflow DAG)
1:40:18 Initialize DBT
1:46:39 Silver Transform
2:00:00 Gold Transform
2:11:00 DBT run
2:16:13 Snowflake to Power BI
2:19:29 Dashboard Create
2:45:29 Outro

📣 Stay Connected:
LinkedIn:   / jayachandrakadiveti  
GitHub: https://github.com/Jay61616
Website: https://jay.unaux.com/

👉 Hashtags:
#dataengineering #spotify #snowflake #powerbi #realtimedata #datapipeline #endtoendproject #dataanalytics #portfolio #python #moderndatastack stack #dbt