Data Engineering
Data engineering is the discipline of reliably moving, transforming, and serving data so that analysts and ML systems can trust it. The deliverable isn't a pipeline for its own sake — it's trustworthy datasets with clear semantics, good performance, and predictable cost.
Most data-engineering pain comes not from the transformation logic but from everything around it: schema drift, late-arriving data, non-idempotent backfills, and teams disagreeing on what "active user" means. Good data engineering treats datasets like products with contracts.
TL;DR
- The product is trustworthy datasets, not just pipelines.
- Modern stacks favor ELT (load raw, transform in the warehouse).
- Treat datasets like APIs — schema contracts and compatibility rules.
- Add data-quality checks at boundaries; make backfills idempotent.
Quick Example
The shape of a modern pipeline — load raw, then transform in the warehouse (ELT):
Core Concepts
A typical system has these stages:
- Sources — application databases, event streams, SaaS tools.
- Ingestion — batch loads, change data capture (CDC), or streaming.
- Storage — data lake (object storage), warehouse, or lakehouse.
- Transformation — ETL/ELT jobs, often dbt-style modeling.
- Orchestration — schedulers and DAGs (Airflow, Dagster, Prefect).
- Serving — BI tools, reverse ETL, feature stores.
- Observability — data-quality checks, lineage, freshness SLAs.
ETL vs ELT
What matters isn't the acronym — it's whether transformations are versioned, tested, and auditable.
Best Practices
- Data contracts — define schemas and compatibility rules; treat a dataset like an API.
- Prefer immutable, append-only event logs where possible.
- Partition/cluster by access patterns (e.g. date, tenant); use columnar formats (Parquet).
- Maintain a golden path — templates, conventions, and tests for new pipelines.
Data quality
Add checks at boundaries: schema validation, null/uniqueness/range checks, freshness and volume anomaly detection, and reconciliation against source-of-truth metrics.
Common Mistakes
Non-idempotent backfills
Silent schema drift
FAQ
ETL or ELT?
ELT is the modern default: load raw data into a warehouse/lakehouse and transform it there, where compute is cheap and transformations can be versioned and tested (e.g. with dbt). ETL still fits when the destination is constrained or you must transform before landing for compliance reasons.
Data lake, warehouse, or lakehouse?
A warehouse stores structured, query-optimized data for analytics. A data lake stores raw files (any format) cheaply in object storage. A lakehouse combines both — lake storage with warehouse-like tables and transactions (e.g. Delta, Iceberg). Many stacks land raw in a lake and model into a warehouse/lakehouse.
How do I ensure data quality?
Validate at boundaries — schema, nulls, uniqueness, ranges — plus freshness and volume anomaly detection and reconciliation against trusted metrics. Make checks part of the pipeline so bad data fails fast instead of silently flowing downstream.
What's a data contract?
An agreement on a dataset's schema, semantics, and compatibility guarantees, treated like an API contract between producers and consumers. It prevents the silent breakage that happens when an upstream change ripples into every downstream job.
Related Topics
- Data Science — A primary consumer of engineered data
- Databases — Where data lands and is served
- ClickHouse — Analytical serving layer
- Message Queues — Streaming ingestion
- Database Migrations — Schema evolution