ShopVista Data Modernization Platform
End-to-End Azure Data Engineering project implementing the Medallion Architecture (Bronze → Silver → Gold) using Azure Data Lake Storage Gen2, Azure Databricks, Delta Lake, Unity Catalog, Auto Loader, Structured Streaming, Databricks Workflows, and Power BI.
Overview
The ShopVista Data Modernization Platform is a modern Azure Lakehouse implementation designed to centralize, automate, and optimize the processing of e-commerce data for business analytics.
ShopVista, a rapidly growing e-commerce organization, manages transactional and operational data generated from multiple business domains, including:
- Orders
- Shipments
- Returns
- Customers
- Products
- Categories
- Brands
- Calendar
Previously, these datasets existed as separate CSV files across different systems, requiring significant manual effort for reconciliation and reporting. This fragmented approach resulted in inconsistent reporting, duplicated processing, and limited visibility into business performance.
To overcome these challenges, this project implements an enterprise-grade Lakehouse Architecture using Microsoft Azure, enabling automated ingestion, scalable transformation, governed data management, and analytics-ready datasets.
The solution follows the Medallion Architecture (Bronze → Silver → Gold) and leverages Azure Databricks, Delta Lake, and Unity Catalog to build a reliable and scalable analytics platform.
Business Background
ShopVista is a fast-growing online retail company that sells products across multiple brands and categories to customers worldwide.
The company generates thousands of transactional records every day from multiple operational systems including:
- Customer Orders
- Product Catalog
- Shipment Tracking
- Product Returns
- Customer Information
- Product Master Data
- Brand Information
- Category Information
As business volume increased, reporting became increasingly difficult due to:
- Data duplication
- Multiple CSV files
- Manual Excel reports
- Inconsistent product information
- Slow business reporting
- Poor data governance
Management required a centralized analytics platform capable of providing reliable, near real-time insights while reducing manual intervention.
Business Requirements
The primary objective of this project is to design and implement a scalable cloud-native data platform capable of supporting enterprise reporting and analytics.
The solution addresses the following business requirements:
Data Centralization
- Consolidate all operational datasets into a centralized Azure Data Lake.
- Eliminate data silos across departments.
Automated Data Ingestion
Automatically ingest:
- Orders
- Shipments
- Returns
- Customer Data
- Product Data
- Brand Data
- Category Data
- Calendar Data
without manual intervention.
Data Quality
Improve overall data quality by:
- Schema validation
- Duplicate removal
- Null handling
- Standardized formats
- Referential integrity checks
Data Modeling
Create analytics-ready datasets using:
- Fact Tables
- Dimension Tables
- Star Schema
optimized for BI reporting.
Governance
Implement centralized governance using:
- Unity Catalog
- Managed Storage
- External Volumes
- Secure Access
Business Intelligence
Provide interactive Power BI dashboards showing:
- Revenue by Brand
- Revenue by Category
- Sales by Country
- Monthly Revenue Trend
- Return Analysis
- Product Performance
- Customer Insights
Scalability
The platform should support:
- Historical full loads
- Incremental data ingestion
- Batch processing
- Streaming workloads
- Future data growth
Solution Architecture
☁️ Azure Services Used
| Azure Service | Purpose |
|---|---|
| Azure Data Lake Storage Gen2 | Centralized Data Lake |
| Azure Databricks | Distributed Data Processing |
| Unity Catalog | Data Governance & Security |
| Delta Lake | ACID-Compliant Storage |
| Auto Loader | Incremental File Ingestion |
| Structured Streaming | Real-Time Data Processing |
| Azure Managed Identity | Secure Resource Access |
| Power BI | Reporting & Visualization |
Architecture Components
| Layer | Technology | Purpose |
|---|---|---|
| Source Layer | ShopVista Operational System | Raw Business Data |
| Landing Layer | Azure Data Lake Storage Gen2 | Raw CSV Storage |
| Governance Layer | Unity Catalog | Secure Data Management |
| Processing Layer | Azure Databricks | ETL / ELT Processing |
| Bronze Layer | Delta Lake | Raw Delta Tables |
| Silver Layer | Delta Lake | Clean & Standardized Data |
| Gold Layer | Delta Lake | Business-Ready Data |
| Reporting Layer | Power BI | Dashboards & Analytics |
Repository Structure
spark-databricks-ecommerce-pipeline/
│
├── architecture/
│
├── dashboards/
│ ├── PowerBI/
│ │ └── ecommerce_analytics.pbix
│ └── screenshots/
│ └── ecommerce_analytics_page-0001.jpg
│
├── datasets/
│ ├── historical-full-load/
│ │ ├── brands/
│ │ │ └── brands.csv
│ │ ├── category/
│ │ │ └── category.csv
│ │ ├── customers/
│ │ │ └── customers.csv
│ │ ├── date/
│ │ │ └── date.csv
│ │ ├── products/
│ │ │ └── products.csv
│ │ ├── order_items/
│ │ │ └── landing/
│ │ │ ├── order_items_2024-01-01.csv
│ │ │ ├── ...
│ │ │ └── order_items_2024-01-19.csv
│ │ ├── order_returns/
│ │ │ └── landing/
│ │ │ ├── returns_2024-01.csv
│ │ │ ├── ...
│ │ │ └── returns_2025-08.csv
│ │ └── order_shipments/
│ │ └── landing/
│ │ ├── shipments_2024-01.csv
│ │ ├── ...
│ │ └── shipments_2024-10.csv
│ │
│ └── incremental-load/
│ ├── order_items/
│ │ └── incoming/
│ │ ├── order_items_2025-09-01.csv
│ │ ├── ...
│ │ └── order_items_2025-09-15.csv
│ ├── order_returns/
│ │ └── incoming/
│ │ ├── returns_2025-09.csv
│ │ ├── ...
│ │ └── returns_2026-01.csv
│ └── order_shipments/
│ └── incoming/
│ ├── shipments_2025-09.csv
│ ├── shipments_2025-10.csv
│ ├── shipments_2025-11.csv
│ └── shipments_2025-12.csv
│
├── docs/
│ ├── 01_Azure_account_setup/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 0_free_azure.png
│ │ ├── 1_create_account.png
│ │ ├── 2_azure_account_setup.png
│ │ ├── 3_azure_account_setup.png
│ │ └── 4_azure_account.png
│ │
│ ├── 02_Azure_Databricks_setup/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 01_create_resource_group.png
│ │ ├── ...
│ │ └── 09_create_compute_cluster.png
│ │
│ ├── 03_ADLS/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 01_create_storage_account.png
│ │ ├── ...
│ │ └── 08_uploaded_files.png
│ │
│ ├── 04_Unity_Catalog/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 01_open_unity_catalog.png
│ │ ├── ...
│ │ ├── Unity_Catalog_Authentication_Flow.png
│ │ └── Unity_Catalog_Azure_Architecture.png
│ │
│ ├── 05_Raw_Schema/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 1_create_external_location_raw.png
│ │ └── 2_create_raw_schema.png
│ │
│ ├── 06_Bronze_Layer/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 1_ingest_dim_data_to_raw_bronze_layer.png
│ │ └── 2_raw_bronze_tables.png
│ │
│ ├── 07_Silver_Layer/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 1_transform_dim_data_to_silver_layer.png
│ │ ├── 2_transform_fact_data_to_silver_layer.png
│ │ └── 3_silver_tables.png
│ │
│ ├── 08_Gold_Layer/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 1_build_gold_dimension_tables.png
│ │ ├── 2_build_gold_fact_tables.png
│ │ └── 3_gold_tables.png
│ │
│ ├── 09_Create_and_Manage_Jobs/
│ │ ├── README.md
│ │ └── images/
│ │ ├── 1_create_job.png
│ │ ├── ...
│ │ └── 13_create_schedule.png
│ │
│ └── 10_PowerBI/
│ ├── README.md
│ └── images/
│
├── notebooks/
│ ├── 01_setup/
│ │ ├── 01_setup_catalog.ipynb
│ │ ├── 02_setup_raw_schema.ipynb
│ │ └── 03_create_external_volume.ipynb
│ │
│ ├── 02_bronze/
│ │ ├── dimension_tables/
│ │ │ └── 01_ingest_dimensions.ipynb
│ │ └── fact_tables/
│ │ └── 01_ingest_order_items_fact.ipynb
│ │
│ ├── 03_silver/
│ │ ├── CommonUtils.ipynb
│ │ ├── dimension_tables/
│ │ │ ├── 01_transform_brand_dim.ipynb
│ │ │ ├── 02_transform_category_dim.ipynb
│ │ │ ├── 03_transform_product_dim.ipynb
│ │ │ ├── 04_transform_customers_dim.ipynb
│ │ │ └── 05_transform_calendar_dim.ipynb
│ │ └── fact_tables/
│ │ └── 01_transform_order_items_fact.ipynb
│ │
│ └── 04_gold/
│ ├── CommonUtils.ipynb
│ ├── 01_daily_summary.ipynb
│ ├── dimension_tables/
│ │ ├── 01_build_gld_dim_products.ipynb
│ │ ├── 02_build_gld_dim_customers.ipynb
│ │ └── 03_build_gld_dim_calendar.ipynb
│ └── fact_tables/
│ └── 01_build_gld_fact_order_items.ipynb
│
├── pipelines/
│
├── .gitignore
├── README.md
└── requirements.txt
Dataset Overview
The project processes multiple business datasets representing various operational domains within the ShopVista platform.
Dimension Datasets
- Brands
- Categories
- Customers
- Products
- Calendar
Fact Datasets
- Orders
- Shipments
- Returns
Loading Strategy
Historical Load
The initial data load processes all historical records into the Bronze layer.
datasets/
historical-full-load/
Incremental Load
New files arriving daily are processed using incremental ingestion.
datasets/
incremental-load/
End-to-End Data Flow
The complete pipeline follows the Medallion Architecture.
ShopVista System
│
▼
CSV Files
│
▼
Azure Data Lake Storage Gen2
│
▼
Unity Catalog External Volume
│
▼
Azure Databricks
│
▼
Bronze Layer
│
▼
Silver Layer
│
▼
Gold Layer
│
▼
Power BI Dashboard
Detailed Medallion Pipeline
Pipeline Workflow
- Source systems generate raw CSV files.
- Files are uploaded into Azure Data Lake Storage Gen2.
- Unity Catalog provides governed access through External Volumes.
- Azure Databricks ingests raw datasets into the Bronze layer.
- Data is cleansed and standardized in the Silver layer.
- Fact and Dimension tables are built in the Gold layer.
- Business-ready datasets are consumed by Power BI dashboards.
Azure Data Lake Storage Structure
Azure Data Lake Storage
│
├── raw/
│ ├── brands/
│ ├── categories/
│ ├── customers/
│ ├── products/
│ ├── orders/
│ ├── shipments/
│ └── returns/
│
├── bronze/
│
├── silver/
│
└── gold/
Unity Catalog Structure
Unity Catalog provides centralized governance, metadata management, and secure access across all Lakehouse assets.
Catalog
│
└── ecommerce
│
├── raw
│
├── bronze
│
├── silver
│
└── gold
The catalog enables:
- Centralized metadata management
- Secure access control
- External Volume management
- Schema organization
- Table discovery
- Data governance
- Fine-grained permissions
- Unified Lakehouse architecture
Bronze Layer
Overview
The Bronze layer is the first stage of the Medallion Architecture and serves as the raw ingestion layer of the Lakehouse.
Its primary objective is to ingest source datasets into Delta Lake while preserving the original business data with minimal transformation.
The Bronze layer acts as the immutable source of truth for downstream processing.
Objectives
The Bronze layer is responsible for:
- Reading raw CSV files from Azure Data Lake Storage Gen2
- Applying predefined schemas
- Capturing ingestion metadata
- Preserving raw business data
- Storing datasets in Delta format
- Supporting historical and incremental ingestion
Source Datasets
Dimension Tables
- Brands
- Categories
- Customers
- Products
- Calendar
Fact Tables
- Orders
- Shipments
- Returns
Bronze Processing Flow
Azure Data Lake Storage
│
▼
Raw CSV Files
│
▼
Schema Validation
│
▼
Metadata Columns
│
▼
Delta Tables
│
▼
Bronze Layer
Metadata Captured
Each Bronze table stores additional metadata columns to support lineage and auditing.
| Column | Description |
|---|---|
| _source_file | Original file location |
| _ingest_time | Timestamp of ingestion |
Bronze Delta Tables
| Table |
|---|
| bronze.brz_brands |
| bronze.brz_categories |
| bronze.brz_customers |
| bronze.brz_products |
| bronze.brz_calendar |
| bronze.brz_orders |
| bronze.brz_shipments |
| bronze.brz_returns |
Bronze Notebook Structure
02_bronze/
│
├── dimension_tables/
│ └── 01_ingest_dimensions.ipynb
│
└── fact_tables/
├── 01_ingest_orders.ipynb
├── 02_ingest_shipments.ipynb
└── 03_ingest_returns.ipynb
Silver Layer
Overview
The Silver layer transforms raw business data into clean, validated, and standardized datasets suitable for analytical processing.
This layer applies business rules, performs data cleansing, removes duplicates, and enforces referential integrity.
Objectives
The Silver layer performs:
- Data cleansing
- Null handling
- Duplicate removal
- Data type standardization
- Foreign key validation
- Business rule validation
- Data enrichment
Silver Processing Flow
Bronze Layer
│
▼
Duplicate Removal
│
▼
Null Handling
│
▼
Business Validation
│
▼
Data Standardization
│
▼
Silver Layer
Data Quality Checks
The following validations are implemented.
Schema Validation
Ensures incoming datasets match expected schemas.
Null Validation
Critical business columns are checked for missing values.
Example:
- Customer ID
- Product ID
- Order ID
Duplicate Validation
Duplicate records are removed based on business keys.
Referential Integrity
Ensures relationships exist between:
Orders ↔ Customers
Orders ↔ Products
Products ↔ Categories
Products ↔ Brands
Data Type Validation
Standardizes
- Dates
- Numeric values
- Currency fields
- Boolean fields
Silver Tables
| Table |
|---|
| silver.slv_brands |
| silver.slv_categories |
| silver.slv_customers |
| silver.slv_products |
| silver.slv_calendar |
| silver.slv_orders |
| silver.slv_shipments |
| silver.slv_returns |
Silver Notebook Structure
03_silver/
│
├── dimension_tables/
│ ├── 01_transform_brand_dim.ipynb
│ ├── 02_transform_category_dim.ipynb
│ ├── 03_transform_product_dim.ipynb
│ ├── 04_transform_customer_dim.ipynb
│ └── 05_transform_calendar_dim.ipynb
│
└── fact_tables/
├── 01_transform_orders.ipynb
├── 02_transform_shipments.ipynb
└── 03_transform_returns.ipynb
Gold Layer
Overview
The Gold layer contains analytics-ready datasets optimized for reporting, business intelligence, and decision-making.
This layer integrates all cleaned datasets into a Star Schema model consisting of Fact and Dimension tables.
Objectives
The Gold layer provides
- Business-ready datasets
- Star schema
- Optimized Delta tables
- Aggregated metrics
- Daily summaries
- Power BI reporting
Gold Processing Flow
Silver Layer
│
▼
Join Fact & Dimension Tables
│
▼
Business Calculations
│
▼
Aggregation
│
▼
Star Schema
│
▼
Gold Layer
Dimension Tables
The following Dimension tables are created.
Product Dimension
Contains
- Product Details
- Category
- Brand
- Product Attributes
Customer Dimension
Contains
- Customer Information
- Country
- Region
Calendar Dimension
Contains
- Date
- Month
- Quarter
- Year
- Week
Fact Tables
Orders Fact
Stores all customer order transactions.
Shipments Fact
Stores shipment events.
Returns Fact
Stores product return information.
Gold Tables
| Table |
|---|
| gold.dim_product |
| gold.dim_customer |
| gold.dim_calendar |
| gold.fact_orders |
| gold.fact_shipments |
| gold.fact_returns |
Star Schema
The Gold layer follows a traditional Star Schema.
dim_customer
│
│
dim_product ---- fact_orders ---- dim_calendar
│
dim_brand
│
dim_category
This model provides:
- Faster BI queries
- Simpler reporting
- Better aggregation performance
Daily Summary Table
To improve reporting performance, the project generates aggregated summary tables.
Examples include
- Daily Sales
- Monthly Revenue
- Sales by Brand
- Sales by Category
- Country Sales
- Return Summary
Daily Summary Flow
Fact Orders
│
▼
Business Aggregations
│
▼
Daily Summary
│
▼
Power BI
Business KPIs
The Gold layer supports reporting for:
Sales KPIs
- Total Revenue
- Total Orders
- Average Order Value
Product KPIs
- Top Products
- Best Selling Categories
- Brand Performance
Customer KPIs
- Active Customers
- Country-wise Sales
- Customer Distribution
Return KPIs
- Return Rate
- Returned Products
- Return by Category
Shipment KPIs
- Total Shipments
- Delivered Orders
- Pending Shipments
Auto Loader & Structured Streaming
Auto Loader
Overview
Azure Databricks Auto Loader is used to automatically detect and ingest newly arrived files from Azure Data Lake Storage Gen2.
This eliminates the need for manual file management and supports scalable incremental ingestion.
Auto Loader Workflow
New CSV Files
│
▼
Azure Data Lake
│
▼
Auto Loader
│
▼
Bronze Delta Tables
Benefits
- Incremental ingestion
- Schema evolution
- File discovery
- Exactly-once processing
- Scalable ingestion
⚡ Structured Streaming
Overview
Structured Streaming enables near real-time processing of incoming datasets.
Instead of processing data only in batches, new files are automatically processed as they arrive.
Streaming Flow
New Files
│
▼
Structured Streaming
│
▼
Bronze
│
▼
Silver
│
▼
Gold
Advantages
- Near real-time processing
- Continuous ingestion
- Fault tolerance
- Checkpointing
- Exactly-once guarantees
⚙️ Databricks Workflows
The project uses Databricks Workflows to orchestrate end-to-end ETL execution.
The workflows automate:
- Bronze ingestion
- Silver transformations
- Gold processing
- Daily summary generation
Workflow Execution
Historical Load
│
▼
Bronze Pipeline
│
▼
Silver Pipeline
│
▼
Gold Pipeline
│
▼
Daily Summary
│
▼
Power BI Refresh
Workflow Benefits
- Automated scheduling
- Retry mechanisms
- Dependency management
- Monitoring
- Failure notifications
End-to-End Processing Summary
Raw Files
│
▼
Azure Data Lake Storage
│
▼
Auto Loader
│
▼
Bronze Layer
│
▼
Silver Layer
│
▼
Gold Layer
│
▼
Daily Summary
│
▼
Power BI Dashboard
📈 Power BI Dashboard
Overview
The Gold layer provides analytics-ready datasets that are consumed by Power BI to generate interactive dashboards for business users.
The dashboards enable stakeholders to monitor sales performance, customer behavior, shipment status, and product returns in near real-time.
Dashboard Architecture
Gold Layer
│
▼
Power BI Dataset
│
▼
Interactive Dashboards
Dashboard Pages
📊 Executive Dashboard
Provides a high-level overview of business performance.
KPIs include:
- Total Revenue
- Total Orders
- Total Customers
- Total Shipments
- Total Returns
- Average Order Value
📈 Sales Dashboard
Displays
- Revenue by Brand
- Revenue by Category
- Revenue by Country
- Monthly Sales Trend
- Top Selling Products
🚚 Shipment Dashboard
Provides
- Shipment Status
- Delivery Performance
- Shipment Volume
- Pending Shipments
🔄 Returns Dashboard
Displays
- Total Returns
- Return Rate
- Return Reason Analysis
- Returns by Product
- Returns by Category
👥 Customer Dashboard
Provides
- Customer Distribution
- Country-wise Customers
- Customer Purchase Analysis
- Customer Segmentation
Dashboard Features
- Interactive Filters
- Drill Through
- Cross Filtering
- KPI Cards
- Trend Analysis
- Dynamic Slicers
🔐 Unity Catalog & Data Governance
Overview
Unity Catalog provides centralized governance for all Lakehouse assets.
It manages
- Catalogs
- Schemas
- Tables
- Volumes
- Permissions
- Metadata
Unity Catalog Structure
Catalog
│
└── ecommerce
│
├── raw
├── bronze
├── silver
└── gold
Governance Features
- Centralized Metadata
- Table Lineage
- Fine-Grained Access Control
- Schema Management
- Secure Data Sharing
- Data Discovery
Security Model
Users
│
▼
Unity Catalog
│
▼
Catalog
│
▼
Schema
│
▼
Table
Benefits
- Secure Access
- Centralized Governance
- Simplified Management
- Better Compliance
- Metadata Management
📋 Data Quality Framework
The project incorporates multiple validation checks throughout the Medallion Architecture.
Bronze Validation
- Schema Validation
- Header Validation
- File Availability
- Metadata Capture
Silver Validation
- Duplicate Removal
- Null Handling
- Standardized Data Types
- Foreign Key Validation
- Business Rule Validation
Gold Validation
- Fact-Dimension Integrity
- Aggregation Validation
- KPI Validation
- Data Completeness
Data Quality Flow
Raw Data
│
▼
Schema Validation
│
▼
Duplicate Removal
│
▼
Business Rules
│
▼
Gold Validation
⚡ Performance Optimization
The project applies multiple optimization techniques to improve query performance.
Delta Lake Optimization
- OPTIMIZE
- VACUUM
- Delta Statistics
- File Compaction
Spark Optimization
- Predicate Pushdown
- Partition Pruning
- Broadcast Join
- Caching
- Adaptive Query Execution (AQE)
Storage Optimization
- Delta Format
- Column Pruning
- Data Compression
Power BI Optimization
- Star Schema
- Aggregated Tables
- Optimized Relationships
- Reduced Refresh Time
📊 Complete Project Flow
🛠 Technology Stack
| Category | Technology |
|---|---|
| Cloud Platform | Microsoft Azure |
| Data Lake | Azure Data Lake Storage Gen2 |
| Data Processing | Azure Databricks |
| Programming Language | Python |
| Big Data Framework | Apache Spark |
| Data Processing API | PySpark |
| Storage Format | Delta Lake |
| Governance | Unity Catalog |
| Streaming | Structured Streaming |
| Incremental Ingestion | Auto Loader |
| Analytics | Power BI |
| Version Control | Git & GitHub |
📊 Solution Highlights
| Metric | Value |
|---|---|
| Architecture | Lakehouse |
| Cloud Platform | Azure |
| Data Lake | ADLS Gen2 |
| Processing Engine | Databricks |
| Storage Format | Delta Lake |
| Governance | Unity Catalog |
| Streaming | Structured Streaming |
| Incremental Load | Auto Loader |
| Reporting | Power BI |
| Scalability | High |
| Availability | High |
✨ Key Features
- End-to-End Azure Lakehouse
- Azure Data Lake Storage Gen2
- Azure Databricks
- Unity Catalog
- Medallion Architecture
- Delta Lake
- Historical Data Load
- Incremental Data Load
- Auto Loader
- Structured Streaming
- Data Quality Validation
- Star Schema
- Fact & Dimension Modeling
- Daily Summary Tables
- Power BI Dashboards
- Modular Notebook Design
- Automated ETL Pipeline
🎯 Skills Demonstrated
Azure
- Azure Data Lake Storage Gen2
- Azure Databricks
- Unity Catalog
- Managed Identity
Data Engineering
- ETL Pipeline Development
- ELT Processing
- Data Lakehouse
- Medallion Architecture
- Data Modeling
- Star Schema Design
- Fact & Dimension Modeling
Big Data
- Apache Spark
- PySpark
- Delta Lake
- Structured Streaming
- Auto Loader
Analytics
- Power BI
- Business Intelligence
- KPI Reporting
- Dashboard Development
Data Governance
- Unity Catalog
- Access Control
- Metadata Management
- Data Lineage
Programming
- Python
- PySpark
- SQL
📚 Learning Outcomes
This project demonstrates practical implementation of:
- Enterprise Lakehouse Architecture
- Delta Lake
- Medallion Architecture
- Data Quality Framework
- Incremental Data Processing
- Streaming Data Processing
- Business Data Modeling
- Dashboard Development
- Azure Cloud Services
- End-to-End Data Engineering Workflow
📂 Documentation
Detailed implementation guides are available in the docs/ directory.
| Documentation | Description |
|---|---|
| Azure Account Setup | Azure subscription and resource creation |
| Azure Databricks Setup | Workspace configuration |
| ADLS Setup | Storage account and containers |
| Unity Catalog | Governance and permissions |
| Raw Schema | Source schema definitions |
| Bronze Layer | Raw ingestion implementation |
| Silver Layer | Data transformation and quality |
| Gold Layer | Data modeling and analytics |
| Databricks Workflows | Job scheduling and orchestration |
📸 Screenshots
The repository includes screenshots demonstrating:
- Azure Portal Resources
- Azure Data Lake Storage
- Azure Databricks Workspace
- Unity Catalog
- Bronze Tables
- Silver Tables
- Gold Tables
- Databricks Workflows
- Power BI Dashboard
🎉 Project Outcome
The ShopVista Data Modernization Platform demonstrates the implementation of a modern Azure Lakehouse architecture using Azure Data Lake Storage Gen2, Azure Databricks, Delta Lake, Unity Catalog, Auto Loader, Structured Streaming, and Power BI.
The solution transforms fragmented e-commerce datasets into a centralized, governed, and analytics-ready platform by applying the Medallion Architecture (Bronze → Silver → Gold). It automates data ingestion, enforces data quality, builds optimized star-schema models, and delivers interactive business dashboards through Power BI.
This project showcases enterprise-grade Data Engineering practices including scalable ETL design, cloud-native data processing, governance, performance optimization, and business intelligence integration, providing a strong foundation for real-world Azure Data Engineering solutions.