This project demonstrates a Data Engineering pipeline built using Azure Databricks and Apache Spark, following the Medallion Architecture (Bronze → Silver → Gold).
The pipeline ingests data from multiple sources, performs transformations using PySpark and SparkSQL, and publishes processed data to Power BI for reporting.
- Bronze Layer – Raw data ingestion from multiple sources
- Silver Layer – Data cleaning, standardization
- Gold Layer – Aggregated, business-ready tables optimized for analytics
| Source Type | File Format | Description |
|---|---|---|
| CSV | customer.csv | Ingested and transformed using PySpark |
| JSON | products.json | Transformed using SparkSQL |
| Parquet | transaction.snappy.parquet | Processed using both SparkSQL & PySpark |
- Azure Databricks
- Apache Spark (PySpark + SparkSQL)
- Delta Lake (Merge/Upsert operations)
- Power BI (connected to Databricks for dashboarding)
