Data Lakehouse
A data lakehouse is a data-management architecture that combines the two things teams historically had to choose between: the cheap, open, flexible storage of a data lake and the reliability, performance, and management features of a data warehouse. Instead of dumping raw files in a lake for data scientists and copying curated data into a warehouse for analysts, the lakehouse serves both from one copy of data in open storage — with transactions, schemas, and governance layered on top.
The pattern emerged to end a costly, error-prone split. The classic setup ran a lake (object storage full of files — cheap and open, but no reliability guarantees) alongside a warehouse (fast and governed, but proprietary and expensive), with brittle pipelines shuttling data between them. The lakehouse collapses that into a single tier: open storage made trustworthy by an open table format. This page covers the architecture, its building blocks, and when it's the right choice.
TL;DR
- A lakehouse unifies the data lake (cheap, open, flexible file storage) and the data warehouse (reliable, fast, governed) into one architecture over one copy of data.
- It's built on cheap object storage plus an open table format (Apache Iceberg, Delta Lake, Hudi) that adds ACID transactions, schema management, and time travel.
- Decoupled storage and compute is a defining trait: many independent engines query the same data, and you scale (and pay for) compute separately from storage.
- It serves both analytics (BI, SQL) and ML/data science from the same data — no more lake-plus-warehouse duplication.
- The medallion architecture (bronze → silver → gold) is the common way to organize progressively refined data within a lakehouse.
- It shines when you have large, varied data and multiple workloads; a plain warehouse is simpler when your data is modest and purely relational.
Quick Example
The lakehouse's essence: warehouse-style SQL running directly on open table-format data in object storage — no proprietary warehouse required.
Analysts get warehouse SQL; data scientists point their ML tools at the identical files — no export, no second copy.
Core Concepts
The Lake vs Warehouse Tradeoff It Resolves
The lakehouse keeps the lake's openness and cost and adds the warehouse's reliability and speed — the two columns' good halves, without maintaining both systems.
The Open Table Format Is the Key Enabler
What turns a pile of files into a lakehouse is the open table format — Apache Iceberg, Delta Lake, or Apache Hudi. It adds the metadata layer that gives object-storage files ACID transactions, safe schema evolution, time travel, and the statistics that make queries fast. Without it you just have a data lake; with it, the lake behaves like a warehouse table that many engines can trust. The table format is to the lakehouse what the storage engine is to a database.
Decoupled Storage and Compute
A defining architectural property: storage and compute are separate and scale independently. Data sits once in object storage; any number of compute engines — Spark for pipelines, Trino/DuckDB for interactive queries, ML frameworks for training — read and write it directly. You spin compute up and down (and pay for it) per workload, without moving or copying the data. This is cheaper and more flexible than a warehouse that couples its storage and compute.
The Medallion Architecture
The common way to organize data within a lakehouse is progressive refinement in layers:
Each layer is a set of tables; pipelines promote data downstream. Bronze preserves raw history, silver makes it trustworthy, gold makes it consumable. This "medallion" pattern gives structure to what would otherwise be an unmanaged lake.
When to Use a Lakehouse
The lakehouse is powerful but not always the simplest choice.
A lakehouse fits when:
- You have large and varied data — structured, semi-structured, and unstructured together.
- You serve multiple workloads — BI/SQL analytics and ML/data science — and want to avoid duplicating data across a lake and a warehouse.
- Openness and avoiding lock-in matter — you want multiple engines and no proprietary storage.
- Cost at scale matters — object storage plus decoupled compute is economical for big data.
A plain warehouse is simpler when:
- Your data is modest in size and purely relational — a managed warehouse is less to operate.
- You have one workload (BI) and don't need ML on the same data.
- You value turnkey simplicity over openness and don't want to manage table formats, catalogs, and compaction.
💡 Tip: The lakehouse vs warehouse line is blurring — modern cloud warehouses increasingly read and write open table formats like Iceberg natively. Often the real question isn't "lake, warehouse, or lakehouse" but "which engine, over open storage, for which workload." Openness is becoming the default even inside warehouses.
Best Practices
Standardize on an Open Table Format and Catalog
The table format (Iceberg and its catalog) is the foundation — pick one your engines all support, and use an open catalog so multiple engines share consistent table state. This is what preserves the lakehouse's engine independence and no-lock-in promise.
Organize With the Medallion Pattern
Don't let the lakehouse become an unmanaged swamp. Structure data as bronze/silver/gold layers so raw history, trustworthy data, and consumable marts are clearly separated, and pipelines have a clear promotion path.
Maintain Your Tables
Open table formats accumulate small files and old snapshots. Schedule compaction, snapshot expiration, and cleanup (see Apache Iceberg) so query performance and storage cost stay in check. An unmaintained lakehouse degrades.
Govern From the Start
One copy of data serving many engines and teams needs central governance: access control, schema management, and data quality checks applied at the table/catalog layer. Build this in early rather than bolting it on once many consumers depend on the data.
Match Compute to Workload
Exploit decoupled compute: use the right engine per job — a heavy engine for large batch transforms, a lightweight one for interactive queries — all over the same tables. Don't force one engine to do everything.
Common Mistakes
Turning It Into a Data Swamp
Adopting a Lakehouse for Small, Simple Data
If your data is modest and purely relational with a single BI workload, a managed warehouse is simpler and cheaper to operate. The lakehouse's benefits — openness, multi-workload, scale — appear with large, varied data and multiple consumers. Don't take on table-format and catalog complexity you don't need.
Skipping Table Maintenance
A lakehouse with no compaction or snapshot expiration slowly fills with tiny files and stale metadata, and queries crawl. Treat maintenance as routine operations, not an afterthought.
FAQ
What's the difference between a data lake, a warehouse, and a lakehouse?
A data lake is cheap, open object storage for any data — flexible but with no transactions or guarantees (easy to turn into a "swamp"). A warehouse is fast, reliable, and governed but proprietary and pricier, and best for structured data. A lakehouse combines them: open, cheap storage made reliable and fast by an open table format, serving both BI and ML from one copy of data — the lake's openness with the warehouse's trustworthiness.
How is a lakehouse different from Apache Iceberg?
Apache Iceberg is a table format — a specific technology that adds ACID transactions, schema evolution, and time travel to files in object storage. A lakehouse is the broader architecture built using such a format: open storage, decoupled compute, medallion layers, governance, serving multiple workloads. Iceberg (or Delta/Hudi) is a key building block; the lakehouse is the whole system you build with it.
What does "decoupled storage and compute" mean?
It means the data lives once in object storage, independent of any compute engine, and multiple engines read and write it directly — scaling and billing compute separately from storage. You can run a big batch engine and a lightweight query engine against the same tables, spin compute up and down per workload, and never copy the data between systems. It's cheaper and more flexible than a warehouse that binds storage to its own compute.
What is the medallion architecture?
It's the common pattern for organizing data within a lakehouse into progressively refined layers: bronze (raw ingested data, kept as-is for history and replay), silver (cleaned, validated, conformed), and gold (business-ready aggregates and marts for BI and ML). Pipelines promote data downstream through the layers, giving structure and trust to what would otherwise be an unmanaged lake.
Do I need a lakehouse, or is a warehouse enough?
A warehouse is enough — and simpler — when your data is modest, mostly relational, and serves a single BI workload. A lakehouse pays off with large, varied data (structured and unstructured), multiple workloads (analytics and ML) you don't want to duplicate data for, and a desire for openness and cost efficiency at scale. Also note the lines are blurring: modern warehouses increasingly speak open table formats natively.
Related Topics
- Apache Iceberg — The open table format that powers lakehouses
- Data Warehousing — The warehouse half of the convergence
- Data Engineering — Building the pipelines that populate a lakehouse
- Apache Spark — A primary engine for lakehouse transforms
- DuckDB — Lightweight interactive querying over lakehouse tables
- Snowflake — A cloud warehouse now embracing open formats
- dbt — Transformation and modeling across medallion layers