DuckLake: A New Lakehouse Format That Stores Metadata in SQL

DuckLake is a new open lakehouse format from DuckDB Labs that puts table metadata in a standard SQL database — Postgres, MySQL, SQLite, or DuckDB — instead of writing thousands of small Avro and JSON files like Iceberg and Delta Lake do. v1.0 shipped in April 2026 under the MIT license. For Spark Scala teams the immediate story is not "rip out Iceberg," but the metadata-in-SQL idea is interesting enough to be worth understanding now.

The Headline

DuckLake was introduced on May 27, 2025 by Mark Raasveldt and Hannes Mühleisen — the same DuckDB Labs team behind DuckDB. The pitch is short: keep the open Parquet files on object storage that every modern table format already uses, but replace the file-based metadata layer with a single transactional SQL database.

v1.0 landed in April 2026 with a stable on-disk format and backward-compatibility guarantees. The specification and the DuckDB extension are both MIT-licensed.

The way DuckDB Labs positions it matters: DuckLake is a catalog format, not a table format. It is not a direct replacement for Iceberg or Delta alone. It is a replacement for the whole top half of the lakehouse stack — "Iceberg + Polaris" or "Delta + Unity" collapsed into one tier.

The Architecture in One Diagram

Three layers, two of them familiar:

-- Layer 1: Object storage (same as Iceberg / Delta)
-- s3://warehouse/orders/
--   data/
--     part-00000-*.parquet
--     part-00001-*.parquet

-- Layer 2: Metadata catalog (this is the new idea)
-- Postgres / MySQL / SQLite / DuckDB
-- Tables:
--   ducklake_table          -- table definitions
--   ducklake_snapshot       -- snapshot history
--   ducklake_data_file      -- pointers to Parquet files
--   ducklake_delete_file    -- deletion vectors
--   ducklake_schema_version -- schema evolution
--   ... (about 25 tables total)

-- Layer 3: Engine (DuckDB today; others via interop)

Every operation that would write a manifest file in Iceberg writes a row in a SQL table instead. Every snapshot pointer that would be an atomic rename in Delta is a transactional UPDATE in the catalog database.

The trick is that ACID — the hardest part of a lakehouse format — is handled by the catalog database itself. The "C" in ACID (consistency) comes from foreign keys, the "I" (isolation) from MVCC, and the "A" (atomicity) from regular SQL transactions. The format doesn't have to reinvent any of it.

What This Actually Buys You

Three concrete wins, and they're real:

Metadata operations are fast. Cold metadata reads on Iceberg or Delta require listing and parsing files on S3 — frequently hundreds of milliseconds per query plan. A SQL lookup in Postgres against an indexed ducklake_snapshot table returns in a couple of milliseconds. For workloads that do a lot of small queries, planning time dominates and this matters.

No small-file metadata problem. Iceberg writes a new manifest list and at least one new manifest per commit. Delta writes a new JSON entry per commit and periodic Parquet checkpoints. High-frequency writers (streaming ingest, frequent merges) accumulate thousands of metadata files that have to be rewritten by maintenance jobs. DuckLake just commits rows.

Multi-table transactions. Because the catalog is a real database, you can wrap multi-table writes in a single transaction with the catalog's normal BEGIN/COMMIT semantics. Iceberg and Delta both lack first-class multi-table ACID; you build it yourself with two-phase commit patterns or accept eventual consistency between tables.

What It Costs You

The honest counter-list:

You now operate a database. A Postgres instance with backup, monitoring, and high availability is a different burden than an S3 bucket with a _delta_log/ prefix. For teams that already run Postgres for application data, this is mostly free. For teams that pride themselves on a stateless lakehouse, it's a real architectural addition.

Scale is unproven at the very top end. Iceberg powers tables with petabytes and millions of files at Netflix and Apple. DuckLake's design is sound but the production track record at that scale doesn't exist yet. The metadata database is not a free lunch — at extreme scale you eventually have to think about partitioning, vacuuming, and connection limits on it.

Engine support is DuckDB-first. DuckLake's reference and most mature implementation is the DuckDB ducklake extension. There is no native Spark reader as of June 2026. If your stack is Spark plus Trino plus Flink, Iceberg's ecosystem is doing more for you than DuckLake's will for the next year or two.

The Iceberg Interop Story

This is the part most relevant to Spark Scala teams today.

DuckLake 0.3, released September 2025, added bidirectional Iceberg interoperability. You can COPY data and metadata between DuckLake and Iceberg in either direction:

-- From DuckDB with the ducklake + iceberg extensions loaded

-- Copy an existing Iceberg table into DuckLake (metadata-only is supported too)
COPY FROM ICEBERG 's3://warehouse/iceberg/orders'
TO DUCKLAKE my_ducklake.orders;

-- Or push a DuckLake table out to Iceberg so Spark / Trino / Athena can read it
COPY FROM DUCKLAKE my_ducklake.orders
TO ICEBERG 's3://warehouse/iceberg/orders_export';

A few things worth knowing:

• Metadata-only copies from Iceberg to DuckLake skip the data and bring only the snapshot history. This is the migration path if you want to move from Iceberg to DuckLake without rewriting Parquet.
• Deletion vectors that DuckLake writes use an Iceberg-compatible binary format, so engines that read Iceberg v3 deletion vectors can read DuckLake's too.
• Live multi-engine reads (a Spark job and a DuckDB process querying the same DuckLake table simultaneously) are not the primary mode. The interop is migration-shaped, not query-shaped. If you need Spark to read it natively, copy it out to Iceberg first.

For Spark Scala teams running on Iceberg with a Polaris catalog, this means DuckLake can sit alongside the existing setup for DuckDB-backed BI or lightweight services, with a clear migration path to and from Iceberg if priorities change.

How It Compares to Iceberg and Delta

A quick side-by-side on the dimensions that matter most for Spark Scala teams:

                          DuckLake          Iceberg              Delta Lake
Metadata storage          SQL database      Avro + JSON files    JSON + Parquet checkpoints
Catalog requirement       Built in          External required    Optional
Metadata read latency     ~milliseconds     ~hundreds of ms      ~hundreds of ms (cold)
Small-file metadata       None              Compaction needed    Checkpoint compaction
Multi-table ACID          Yes               No                   No
Engine maturity           DuckDB-first      Spark/Trino/Flink/+  Spark-first, others via UniForm
Production track record   New (v1.0 2026)   Years at large scale Years at large scale
License                   MIT               Apache 2.0           Apache 2.0

The pattern is clear: DuckLake wins on the dimensions where a real database beats a pile of files, and loses on the dimensions where the ecosystem has had years to mature.

What This Means for Spark Scala Teams in 2026

Three honest takes depending on where you are.

If you're already on Iceberg or Delta and it's working: keep going. The cost of moving table formats — migration tooling, retraining pipelines, validating equivalence on production data — is higher than the metadata-latency wins you'd get from DuckLake today. The format isn't going anywhere, but it also isn't desperate for early adopters. Revisit in 12 months when Spark connectors and managed offerings exist.

If you're standing up a fresh lakehouse and your scale is moderate: DuckLake deserves a serious look, especially if you also run DuckDB for BI or local analytics, and especially if you already operate Postgres. The simpler operational story (one database instead of a catalog + a manifest layer) is real, and v1.0 means the format is stable.

If you're using Spark plus DuckDB side by side: DuckLake gives you a more natural shared substrate. The pattern in DuckDB for Spark Scala Developers — DuckDB for ad-hoc reads, Spark for distributed writes — works against Iceberg today, but the metadata-lookup cost on the DuckDB side is non-trivial. A DuckLake table that Spark writes via an export step and DuckDB reads natively could be measurably faster for that read path.

Where the Format Goes From Here

The two trends to watch over the rest of 2026:

1. Spark connector. A native Spark reader for DuckLake would change the calculation significantly. The Iceberg interop path works but adds a copy step. There is community interest but no official ETA from the Apache Spark side.
2. Managed catalog services. Today you bring your own Postgres or MySQL. A managed DuckLake catalog (the equivalent of Polaris for Iceberg) would lower the operational bar enough to matter for teams that don't already run a database.

The broader signal is consistent with what we wrote in The State of Spark Scala in 2026 — the lakehouse layer is consolidating, not fragmenting, and interop is becoming a first-class concern. DuckLake is interesting partly because it acknowledges that. It's not trying to win by replacing Iceberg or Delta everywhere; it's trying to win by being better at the metadata tier and friendly to the rest of the ecosystem.

Summary

• What it is: an open lakehouse format that stores metadata in a SQL database (Postgres / MySQL / SQLite / DuckDB) and data in Parquet on object storage.
• Why it exists: to fix the metadata-file proliferation and slow planning that Iceberg and Delta inherit from file-based catalogs.
• What you get: millisecond metadata reads, multi-table transactions, no small-file metadata problem, simpler ops if you already run a database.
• What you give up: a stateless lakehouse, mature Spark/Trino/Flink connectors, and large-scale production track record.
• Where Spark fits today: through Iceberg interop — copy in or out, no native Spark reader yet.
• Worth trying if: you're starting fresh, you already run DuckDB or Postgres, and your scale is moderate.
• Worth waiting on if: you have a working Iceberg or Delta stack and your pain points aren't planning latency or small-file metadata.

Read the original announcement at ducklake.select and the v0.3 Iceberg interop release at duckdb.org for the technical details. For broader context on how DuckLake fits next to existing formats, see Apache Iceberg vs Delta Lake.