Data Lakehouse Architecture: Delta Lake ACID Transactions, Time Travel, and Schema Evolution

The data lakehouse architecture merges the low-cost, flexible storage of data lakes with the reliability and performance guarantees of data warehouses. Delta Lake, the open-source storage layer developed by Databricks, is the most widely adopted lakehouse format. This guide covers Delta Lake ACID transactions, time travel, schema evolution, merge operations, and integration with Apache Spark for production workloads.

Why Lakehouse? The Problem with Plain Data Lakes

Traditional data lakes suffer from four critical reliability issues:

1. No atomicity: A failed write leaves partial data, corrupting downstream reads
2. No isolation: Concurrent reads during a write see inconsistent data
3. Schema chaos: Anyone can write any schema, breaking downstream pipelines
4. No history: Deleting or overwriting data is irreversible

Delta Lake solves all four by adding a transaction log (_delta_log) alongside Parquet files, transforming an ordinary object-storage path into a fully ACID-compliant table.

Delta Lake Architecture

Every Delta table consists of:

• Data files: Parquet files in the table directory
• Transaction log: _delta_log/ directory containing JSON commit files
• Checkpoints: Parquet snapshots of the log, created every 10 commits

The transaction log records every operation (add file, remove file, metadata change) with a monotonically increasing version number.

Creating and Writing Delta Tables

from delta import DeltaTable, configure_spark_with_delta_pip
from pyspark.sql import SparkSession

spark = SparkSession.builder     .appName("DeltaLake")     .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")     .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")     .getOrCreate()

# Write a Delta table
df.write.format("delta").mode("overwrite").save("s3://data-lake/delta/orders/")

# Or use SQL DDL
spark.sql('''
    CREATE TABLE IF NOT EXISTS orders (
        order_id    STRING NOT NULL,
        customer_id BIGINT,
        amount      DOUBLE,
        order_date  DATE,
        status      STRING
    )
    USING delta
    PARTITIONED BY (order_date)
    LOCATION 's3://data-lake/delta/orders/'
''')

ACID Transactions

Delta Lake provides full ACID guarantees through its transaction log and optimistic concurrency control.

Atomic writes: Either all files in a write succeed and appear in the log, or none do.

# This is atomic — either all rows are written or none
df_new_orders.write     .format("delta")     .mode("append")     .save("s3://data-lake/delta/orders/")

Serializable isolation: Concurrent writers detect conflicts and one retries automatically.

# Safe concurrent writes with partition-level conflict detection
df.write     .format("delta")     .mode("overwrite")     .option("replaceWhere", "order_date = '2026-05-28'")     .save("s3://data-lake/delta/orders/")

Merge (Upsert) Operations

Delta Lake's MERGE is ideal for CDC (Change Data Capture) and SCD patterns.

from delta.tables import DeltaTable

delta_table = DeltaTable.forPath(spark, "s3://data-lake/delta/orders/")

# Upsert: update existing rows, insert new ones
delta_table.alias("target").merge(
    updates_df.alias("source"),
    "target.order_id = source.order_id"
).whenMatchedUpdate(set={
    "status":     "source.status",
    "amount":     "source.amount",
    "updated_at": "source.updated_at",
}).whenNotMatchedInsertAll(
).whenNotMatchedBySourceUpdate(condition="target.status != 'completed'",
    set={"status": "'stale'"}
).execute()

For high-volume CDC, optimize merge performance:

# Partition and Z-order by merge key columns
spark.sql("OPTIMIZE delta.`s3://data-lake/delta/orders/` ZORDER BY (order_id, customer_id)")

Time Travel

Delta Lake retains all historical versions of data. You can query any past version.

# Query by version number
df_v5 = spark.read     .format("delta")     .option("versionAsOf", 5)     .load("s3://data-lake/delta/orders/")

# Query by timestamp
df_yesterday = spark.read     .format("delta")     .option("timestampAsOf", "2026-05-27 00:00:00")     .load("s3://data-lake/delta/orders/")

# SQL syntax
spark.sql('''
    SELECT * FROM orders TIMESTAMP AS OF '2026-05-27'
    WHERE order_date = '2026-05-27'
''')

Restore to a previous version (destructive rollback):

delta_table = DeltaTable.forPath(spark, "s3://data-lake/delta/orders/")
delta_table.restoreToVersion(10)
# or: delta_table.restoreToTimestamp("2026-05-25 12:00:00")

Schema Evolution and Enforcement

Delta Lake enforces schema by default — writes that don't match the table schema fail.

Schema evolution: Opt-in to allow additive schema changes.

# Allow adding new columns
df_with_new_cols.write     .format("delta")     .mode("append")     .option("mergeSchema", "true")     .save("s3://data-lake/delta/orders/")

# Full schema overwrite (use cautiously)
df_new_schema.write     .format("delta")     .mode("overwrite")     .option("overwriteSchema", "true")     .save("s3://data-lake/delta/orders/")

View schema history:

delta_table = DeltaTable.forPath(spark, "s3://data-lake/delta/orders/")
delta_table.history().select("version", "timestamp", "operation", "operationParameters").show()

Optimization: OPTIMIZE and Z-Ordering

Delta tables accumulate many small files from streaming writes. OPTIMIZE compacts them.

# Compact small files
spark.sql("OPTIMIZE delta.`s3://data-lake/delta/orders/`")

# Z-order: co-locate related data for faster multi-column filters
spark.sql('''
    OPTIMIZE delta.`s3://data-lake/delta/orders/`
    ZORDER BY (customer_id, order_date)
''')

OPTIMIZE is most effective when run after a batch of streaming micro-batch writes or after large MERGE operations.

Vacuum: Cleaning Up Old Files

Old data files are retained for time travel. VACUUM removes files outside the retention window.

# Default retention: 7 days
spark.sql("VACUUM delta.`s3://data-lake/delta/orders/`")

# Custom retention (minimum 168 hours unless safety check disabled)
spark.sql("VACUUM delta.`s3://data-lake/delta/orders/` RETAIN 240 HOURS")

Set a retention policy globally:

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
spark.sql("VACUUM delta.`s3://data-lake/delta/orders/` RETAIN 48 HOURS")
# Re-enable safety check immediately after
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "true")

Streaming Writes and Reads

Delta Lake integrates natively with Spark Structured Streaming.

# Write a stream to Delta
query = (
    streaming_df
    .writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "s3://checkpoints/orders-stream/")
    .start("s3://data-lake/delta/orders/")
)

# Read a Delta table as a stream
orders_stream = (
    spark.readStream
    .format("delta")
    .option("maxFilesPerTrigger", 100)  # micro-batch size
    .load("s3://data-lake/delta/orders/")
)

Change Data Feed (CDF)

Delta Lake CDF captures row-level changes, enabling downstream incremental processing without full table scans.

# Enable CDF on an existing table
spark.sql("ALTER TABLE orders SET TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true')")

# Read changes since version 10
changes = spark.read     .format("delta")     .option("readChangeFeed", "true")     .option("startingVersion", 10)     .table("orders")

changes.show()
# _change_type: insert | update_preimage | update_postimage | delete

Table Properties and Performance Config

spark.sql('''
    ALTER TABLE orders SET TBLPROPERTIES (
        'delta.autoOptimize.optimizeWrite' = 'true',
        'delta.autoOptimize.autoCompact'   = 'true',
        'delta.logRetentionDuration'       = 'interval 30 days',
        'delta.deletedFileRetentionDuration' = 'interval 7 days'
    )
''')

Conclusion

Delta Lake transforms object storage into a reliable, performant data lakehouse. ACID transactions eliminate data corruption from concurrent writes and failures. MERGE operations enable efficient CDC and upsert patterns. Time travel provides an audit trail and easy rollback capability. Schema evolution allows safe, additive changes without pipeline breakage. Combined with OPTIMIZE, VACUUM, and Structured Streaming integration, Delta Lake is the foundation for modern lakehouse architectures that scale from gigabytes to petabytes.