Medallion Architecture: Bronze, Silver and Gold Layers Done Right

Data lakes frequently collapse under their own weight. Without disciplined boundaries between raw payloads, cleansed structures, and consumption-ready aggregations, teams quickly find themselves maintaining an unmanageable data swamp plagued by schema drift, duplicate records, and silent pipeline failures.

In modern Databricks engineering, the medallion architecture has established itself as the gold standard pattern to solve this challenge. By logically organizing data into Bronze (raw), Silver (cleansed/conformed), and Gold (curated/aggregated) layers, this architecture guarantees data reliability, transactional integrity via Delta Lake, and full auditability from source to consumption.

Yet, implementing a medallion architecture effectively requires more than simply creating three folders in cloud object storage. It demands intentional choices around ingestion engines, schema enforcement, change data capture (CDC), and physical layout optimization.

---

The Core Philosophy of the Medallion Architecture

The medallion design pattern is an incremental data refinement pipeline. As data moves across boundaries, its quality, structure, and semantic clarity increase, while raw fidelity is systematically preserved upstream.

Source Systems ──> [ Bronze: Raw / Append-Only ] 
                       │
                       ▼
                   [ Silver: Cleaned / Conformed / CDC ] 
                       │
                       ▼
                   [ Gold: Business Logic / Star Schema / Feature Stores ]

This multi-hop approach decouples data ingestion from business transformation. Upstream changes in source application schemas do not immediately break downstream BI dashboards or machine learning feature pipelines. Instead, each layer acts as a resilient buffer with distinct responsibilities, SLAs, and access controls.

---

Bronze Layer: Raw Ingestion Done Right

The Bronze layer is an append-only archive of raw enterprise data. Its primary responsibility is high-throughput ingestion with zero data loss, preserving the raw payload exactly as it arrived from operational databases, event streams, or third-party APIs.

Key Implementation Principles

1. True Append-Only Immutability: Never run updates (UPDATE) or deletions (DELETE) on Bronze tables, except for regulatory compliance (such as GDPR/CCPA right-to-be-forgotten requests). Every batch or stream should append incoming records.
2. Metadata Enrichment: Always enrich raw records with ingestion metadata at the point of entry. Typical fields include:
   • _ingested_timestamp: UTC timestamp of write time.
   • _source_file_name: Path to the ingested cloud object.
   • _batch_id or _stream_id: Tracking identifier for debugging.
3. Handle Malformed Records with _rescued_data: When leveraging Databricks Auto Loader (cloudFiles), configure schema rescue to capture unexpected fields or malformed data instead of dropping them:

# Streaming raw JSON into Bronze using Auto Loader
(spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "dbfs:/mnt/telemetry/bronze/_schema")
    .option("cloudFiles.inferColumnTypes", "true")
    .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
    .load("s3://telemetry-raw-landing/")
    .writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "dbfs:/mnt/telemetry/bronze/_checkpoint")
    .toTable("telemetry_bronze"))

By relying on Auto Loader and schema evolution modes, your Bronze ingestion remains resilient to sudden upstream schema drift while preserving raw JSON/AVRO/CSV payloads for historical replay.

---

Silver Layer: Curated, Cleansed, and Conformed

The Silver layer is where data transformation begins. In this stage, raw Bronze records are filtered, de-duplicated, typed, validated, and normalized to build an enterprise-wide "single source of truth."

Operational Patterns for Silver

• Change Data Capture (CDC) via Delta Lake MERGE: Source systems rarely emit clean event logs. Most produce state changes. The Silver layer processes CDC logs (using tools like Debezium, Fivetran, or AWS DMS) and applies upserts into target Delta tables using MERGE INTO operations, maintaining current-state operational views alongside SCD Type 2 history.
• Data Quality Constraints: Utilize Delta Lake Constraints and Delta Live Tables (DLT) expectations to enforce business validation rules:

ALTER TABLE customers_silver 
ADD CONSTRAINT valid_email 
CHECK (email RLIKE '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$');

• Schema Conformation: Stringified dates become canonical timestamps (TIMESTAMP_NTZ), string numbers become explicit floats or decimals, and inconsistent casing across source systems is normalized.
• Data Modeling: Silver typically adopts third normal form (3NF) or Data Vault 2.0 structures to maintain normalized entities (e.g., dim_user, dim_device, fact_session) before domain-specific aggregations are applied.

---

Gold Layer: Business-Level Aggregation & Performance Tuning

The Gold layer transforms conformed Silver data into business-ready products: Star Schema data marts, dimensional models, pre-computed executive aggregates, and feature stores for machine learning models.

Gold tables are optimized for read performance, fast concurrency, and low latency querying by tools like Power BI, Tableau, and Databricks SQL Serverless.

Optimizing the Gold Layer

1. Dimensional Modeling: Structure data using denormalized facts and dimension tables (Kimball methodology) to simplify reporting queries for analytics engineers.
2. Liquid Clustering over Rigid Partitioning: Traditional Hive-style directory partitioning (e.g., /year=2024/month=10/) often leads to small-file problems or partition skew. In modern Databricks engineering, implement Liquid Clustering to dynamically adapt physical data layouts to query workloads:

-- Creating an optimized Gold table with Liquid Clustering
CREATE TABLE analytics_gold.daily_revenue_mart (
    transaction_date DATE,
    merchant_id STRING,
    region STRING,
    total_sales_amount DECIMAL(18, 2),
    transaction_count INT
)
USING DELTA
CLUSTER BY (transaction_date, region);

3. Feature Stores for ML: Feature sets derived in Gold should register directly with Unity Catalog's Feature Store, ensuring consistent offline training sets and low-latency online inference tables.

---

Architectural Comparison Across Layers

The following breakdown illustrates how data characteristics, storage patterns, and consumer access evolve across the medallion architecture:

---

Critical Antipatterns to Avoid

Even seasoned platform teams make foundational mistakes when structuring their lakehouse architecture:

• Skipping the Silver Layer: Directly aggregating Bronze data into Gold is a common shortcut that creates unmaintainable technical debt. When business rules change or bugs occur in Gold transformations, recalculating directly from raw, uncleaned Bronze data without a conformed Silver baseline requires complex, fragile refactoring.
• Over-Partitioning Early: Partitioning small tables by high-cardinality keys creates thousands of sub-megabyte Parquet files. Delta Lake’s file compactor (OPTIMIZE) and auto-compaction features must be properly balanced with compaction jobs to keep file sizes within the recommended 128 MB to 1 GB range.
• Neglecting Data Retention and Vacuum: Running VACUUM too aggressively can break active streaming checkpoints and time-travel queries. Conversely, never vacuuming leads to ballooning cloud object storage costs. Establish automated maintenance workflows to run VACUUM (typically with a 7-day or 14-day safety threshold).
• Vendor Lock-in in Lakehouse Code: Avoid writing transformation code that only runs on proprietary runtimes. Building modular transformation steps using standard PySpark, SQL, and open Delta Lake guarantees portability. At Izan Technologies, we believe in technical independence: our clients retain 100% intellectual property ownership of all data assets, pipelines, and architectural frameworks we construct.

---

Operationalizing with Unity Catalog

A modern medallion implementation in Databricks must be secured through Unity Catalog. Rather than managing access through storage credentials or IAM roles, apply a 3-level namespace (catalog.schema.table) that mirrors the architecture:

• prod_catalog.bronze.* – Accessible only by platform service principals and data engineering pipelines.
• prod_catalog.silver.* – Accessible by data engineers and authorized data scientists performing exploratory data analysis.
• prod_catalog.gold.* – Accessible by the broader organization, analytical query engines, and BI tools via column- and row-level security controls.

---

How Izan Technologies Helps

Designing a resilient lakehouse requires deep expertise across cloud platforms, streaming engines, and analytical patterns. Izan Technologies designs and deploys custom AI, ML, and Databricks engineering solutions that modernize legacy data warehouses and unstable lakes into enterprise-grade Delta Lake architectures.

When you partner with Izan Technologies, you maintain complete strategic control: our customers own 100% of the IP, code, and infrastructure we deliver.

Ready to optimize your lakehouse architecture? Try our Smart AI DataStudio to accelerate your data transformation, or talk to an expert to review your Databricks strategy today.