Apache Iceberg

Apache Iceberg is an open table format: a specification that turns a pile of data files in object storage (like Parquet on S3) into a real table with ACID transactions, schema evolution, and time travel. It's the technology at the heart of the modern data lakehouse — the architecture that combines the cheap, open storage of a data lake with the reliability and management features of a data warehouse.

The problem Iceberg solves is old and painful. Data lakes stored raw files that any engine could read cheaply, but they had no notion of a "table": concurrent writes corrupted data, schema changes were dangerous, and "which files make up this table right now?" had no reliable answer. Iceberg adds a metadata layer on top of the files that answers those questions atomically — and does it in an open format that multiple query engines can share, so your data isn't locked into one vendor's warehouse.

TL;DR

Quick Example

Create an Iceberg table, insert, evolve the schema, and time-travel — all standard SQL, here via Spark:

The same table is readable by Trino, Flink, DuckDB, and others — because the format, not the engine, defines the table.

Core Concepts

Table Format vs File Format vs Engine

These three layers are easy to conflate:

Iceberg sits in the middle. It doesn't store your rows (Parquet does) and doesn't run your queries (the engine does) — it defines the table so engines and files can agree on what data exists.

The Metadata Tree

Iceberg's power comes from a layered metadata structure that lets an engine find exactly the right files without listing object storage:

Column statistics stored in the manifests let the engine prune files before reading — skipping everything that can't match a query's filter. This is why Iceberg queries are fast even over huge tables.

Snapshots: Atomicity and Time Travel

Every write creates a new snapshot — an immutable view of the table at that moment. A commit atomically swaps the catalog pointer to the new snapshot; readers always see a consistent version, never a half-written one. Because old snapshots persist (until expired), you can:

This is ACID for a data lake, achieved through immutable snapshots and an atomic pointer swap.

The Catalog

The catalog is the registry that maps table names to their current metadata location and handles the atomic commit. It's the component that makes concurrent writes safe and multi-engine access coherent — every engine consults the same catalog to find the current table state. Common implementations include the REST catalog (an open API spec), plus AWS Glue, Nessie, Hive Metastore, and vendor catalogs.

Why Iceberg Matters

Engine Independence (No Lock-In)

The defining benefit: one copy of your data, in open formats, readable and writable by many engines. Your batch pipeline (Spark), your interactive analytics (Trino/DuckDB), your streaming (Flink), and your warehouse (Snowflake) all operate on the same tables. You're not copying data between systems or locked into one vendor's proprietary storage. This is what turns a lake into a lakehouse.

Safe Evolution

Schema changes — add, drop, rename, reorder columns — are metadata-only operations that never rewrite data or break old files, because Iceberg tracks columns by stable IDs, not by position or name. Partition evolution is equally safe: you can change how a table is partitioned going forward without rewriting history, and Iceberg's hidden partitioning means queries don't need to know the partition columns to benefit from pruning.

Correctness on Object Storage

Object stores (S3, GCS, Azure Blob) are cheap and infinite but only eventually consistent and have no transactions. Iceberg's snapshot-and-pointer model provides transactional correctness on top of them — no more corrupted tables from concurrent writes or partial failures.

Best Practices

Maintain Your Tables

Iceberg tables accumulate small files (from frequent writes) and old snapshots. Run periodic maintenance: compaction (rewrite many small files into fewer large ones for read speed), snapshot expiration (drop old snapshots you no longer need to time-travel to), and orphan-file cleanup. Unmaintained tables get slow and expensive.

Choose a Catalog Deliberately

The catalog is central infrastructure. The REST catalog is the open, engine-neutral standard and a safe default; managed catalogs (Glue, Nessie, vendor offerings) trade some openness for convenience. Pick one that all your engines support and that fits your governance needs.

Let Iceberg Handle Partitioning

Use hidden partitioning and partition transforms (days(ts), bucket(16, id)) rather than manually deriving partition columns. Queries then get pruning automatically without users knowing the partition layout, and you can evolve the partitioning later.

Don't Treat It Like OLTP

Iceberg is built for analytical (OLAP) workloads — large scans, appends, and batch updates — not high-frequency single-row transactions. For that, use an operational database. Row-level updates and deletes are supported (via merge-on-read or copy-on-write) but are not a substitute for a transactional store.

Comparison

All three are open table formats solving the same core problem; Iceberg has gained the broadest cross-engine and cross-vendor adoption, making it the common default for a neutral lakehouse.

Common Mistakes

Skipping Table Maintenance

Using It as a Transactional Database

Manual Partition Columns

Deriving order_date yourself and partitioning on it (the old Hive way) locks the layout and forces queries to know it. Use Iceberg's partition transforms and hidden partitioning instead.

FAQ

Is Iceberg a database?

No — it's a table format, a specification for organizing data files and metadata. It has no query engine and no storage of its own; it defines how files in object storage (Parquet) combine into a table that engines like Spark, Trino, or DuckDB can query. Think of it as the layer that makes a data lake behave like a warehouse table.

What is a data lakehouse?

A lakehouse is an architecture that puts warehouse-grade features — ACID transactions, schema management, time travel, governance — directly on top of cheap, open object storage, instead of loading data into a proprietary warehouse. An open table format like Iceberg is what makes it possible: one copy of open data, managed reliably, queryable by many engines.

Iceberg vs Delta Lake vs Hudi — which do I pick?

All three are open table formats with similar core features. Iceberg leads on engine independence and open catalogs, making it the strongest choice when you want to avoid vendor lock-in and use multiple engines. Delta Lake is deeply integrated with Spark and Databricks; Hudi excels at high-frequency upserts and streaming ingestion. Pick based on your engine ecosystem and workload.

How does time travel work?

Every write creates an immutable snapshot, and old snapshots persist until you expire them. You can query the table AS OF a past snapshot ID or timestamp, and roll back to a prior snapshot to undo a bad write. It's the same mechanism that gives Iceberg atomic commits — the snapshot list plus an atomic pointer swap.

Does Iceberg replace my data warehouse?

It can, or it can complement one. Many teams use Iceberg as the open storage layer and point multiple engines at it — including cloud warehouses that now read and write Iceberg natively. Rather than replacing the warehouse's engine, Iceberg replaces its proprietary storage, so your data stays open and portable.

Related Topics

References