Apache Airflow
Apache Airflow is the most widely used orchestrator for data pipelines. You define workflows as DAGs (directed acyclic graphs) in Python — tasks and the dependencies between them — and Airflow schedules them, runs them across workers, retries failures, and gives you a UI showing exactly what ran, when, and why it failed.
Orchestration is the connective tissue of data engineering: extract from APIs, load to the warehouse, run Spark or dbt transformations, refresh ML features — each step depending on the last, every night, with alerting when something breaks. Airflow is the incumbent standard for that job.
TL;DR
- A DAG is a Python file declaring tasks and their dependency order; the scheduler creates a run per interval and executes tasks when their upstreams succeed.
- Airflow is an orchestrator, not a data processor — tasks should trigger work (a query, a Spark job, an API call), not crunch gigabytes in the worker.
- Design every task idempotent: re-running it for the same interval must produce the same result, because retries and backfills will re-run it.
- Each run has a logical date — parameterize queries by it (
{{ ds }}) instead ofnow(), or backfills produce garbage. - The modern TaskFlow API (
@dag/@taskdecorators) replaces most boilerplate operator wiring. - XComs pass metadata, not data — move real data through storage (S3, warehouse tables), not between tasks.
Quick Example
A daily ETL DAG with the TaskFlow API:
The graph (extract → load → transform) appears in the UI; failures retry twice, and any day can be re-run safely because every step is keyed to ds.
Core Concepts
DAGs, Tasks, and Runs
- DAG: the workflow definition — schedule, default retries, task graph.
- Task: one node — a Python function, SQL statement, Bash command, or a provider operator (S3, BigQuery, Databricks, Kubernetes pod...).
- DAG run: one dated instance of the workflow; task instance: one task within it, with states (
queued → running → success/failed/up_for_retry).
Dependencies read naturally:
The Logical Date
Each run processes a data interval and carries its logical date. This is Airflow's central idea and its most common stumbling block:
Done right, backfills become trivial: tell Airflow to run January 1–31, and it executes 31 parameterized runs.
Idempotency
Tasks re-run — on retry, on backfill, on a manual clear. Every task must tolerate that:
Non-idempotent pipelines are the ones that need a human every time something hiccups.
XComs
Tasks exchange small values (paths, row counts, IDs) via XCom — stored in Airflow's metadata database. Passing DataFrames through XCom bloats the DB and breaks at scale; write data to storage and pass the reference.
Architecture and Operations
Operationally, most teams run managed Airflow — AWS MWAA, Google Cloud Composer, or Astronomer — rather than self-hosting the scheduler/worker/DB stack. Either way:
- Alert on failures and SLA misses (
on_failure_callback→ Slack/PagerDuty); a silent orchestrator is worthless — see Alerting. - DAG files are code: version them, review them, deploy through CI/CD, and test DAG integrity (imports, no cycles) in the pipeline.
- Keep top-level DAG code cheap — the scheduler re-parses files constantly; API calls or DB queries at module level throttle the whole deployment.
Best Practices
Orchestrate, Don't Compute
Workers are for coordination. Heavy lifting belongs in the systems built for it:
A task that loads 20 GB into worker memory turns your orchestrator into a fragile, undersized compute engine.
Small, Observable Tasks
One task = one retryable unit. A monolithic run_everything() task means any failure re-runs everything and the UI tells you nothing. Split by failure domain: extract, load, transform, validate.
Validate Data, Not Just Exit Codes
A pipeline that "succeeded" while loading zero rows is a worse failure than a crash. Add check tasks (row counts, freshness, dbt tests / Great Expectations) after loads.
Set catchup Intentionally
catchup=True with start_date=2024-01-01 schedules two years of backfill runs the moment you deploy. Default to catchup=False; backfill explicitly when you mean to.
Airflow vs Alternatives
New greenfield teams increasingly evaluate Dagster; Airflow remains the safest hire-for, integrate-with choice, and Airflow 3 modernized the UI, versioned DAGs, and event-driven scheduling.
Common Mistakes
Using datetime.now() in Task Logic
Breaks backfills and retries — the same task run at different times produces different results. Use the templated {{ ds }} / data_interval_start.
Passing Data Through XCom
XCom lives in the metadata database. Row counts: fine. DataFrames: outage in waiting. Pass storage paths.
Heavy Code at DAG File Top Level
requests.get(...) at module level runs on every scheduler parse loop, not per DAG run. Keep module scope to definitions only.
No Data-Quality Gates
Exit code 0 ≠ correct data. Downstream consumers finding empty tables is how trust in the data platform dies.
FAQ
Is Airflow only for data engineering?
That's its home, but it orchestrates anything schedulable with dependencies: ML retraining (MLOps), report generation, infrastructure housekeeping. For sub-minute or event-driven work, use a message queue/background jobs instead — Airflow's granularity is batch.
Airflow or just cron?
Cron runs commands on a clock. The moment you need "B after A succeeds," retries, backfill, or a UI answering "did last night's load run?", you've outgrown it.
Airflow or Dagster?
Dagster's asset model (declare outputs, get lineage and freshness for free) and local dev story are genuinely better for warehouse-centric stacks; Airflow wins on maturity, integrations, managed offerings, and hiring pool. Both are solid — switching costs matter more than the delta.
How does dbt fit with Airflow?
Complementary: dbt owns SQL transformations inside the warehouse; Airflow orchestrates the whole pipeline around it (extract/load, dbt run, validations, downstream consumers). Cosmos renders dbt models as individual Airflow tasks.
How do I test DAGs?
Three layers: import tests (DAG file parses, no cycles), unit tests on task functions (plain Python — a TaskFlow benefit), and a staging deployment running against non-production data.
Related Topics
- Data Engineering — The discipline Airflow serves
- Apache Spark — The heavy compute Airflow triggers
- Apache Kafka — Streaming counterpart to batch orchestration
- Background Jobs — App-level async work (different tool)
- CI/CD — Deploying DAGs like any other code
- MLOps — Retraining pipelines on a schedule