Airflow DAGs, Scheduling, and Backfills
Design an Airflow task around the data interval it processes and the output it should publish, not the clock time when a worker happens to execute it. Retries and historical runs should reproduce the intended partition result without duplicating effects. Scheduling creates opportunities to run; application logic still determines whether the work is correct.
Before a backfill, define the historical range, code and input versions, publication policy, downstream effects and capacity limit. A graph with dependencies is useful, but it cannot infer which business operation is safe to repeat. That decision belongs in the task contract and its tests.
TL;DR
- Design each task around the data interval it processes and the output it publishes, not the wall-clock time a worker runs it.
- Retries and historical runs should reproduce the partition result without duplicating effects: replace a partition instead of appending to it.
- Scheduling creates opportunities to run; your task logic decides whether the work is correct.
- Before a backfill, define the range, code and input versions, publication policy, downstream effects, and capacity limits.
- Check commands against your Airflow major version; examples from older releases often differ.
Quick Example
This Python function models a partition output with a half-open interval. It is not an Airflow DAG, operator or scheduler test. Two rows are synthetic, and the row at the ending boundary belongs outside the demonstrated interval.
Both calls return three and leave one declared output entry. That demonstrates replacement in this in-memory model with unchanged inputs. It does not provide a durable transaction, concurrent writer control, immutable input snapshot, atomic object publication or actual task retry. Those properties need tests of the chosen storage and Airflow integration.
Core Concepts
Logical time identifies data, not necessarily execution
Airflow's DAG-run documentation distinguishes the data interval from execution time and describes logical date as the interval start for the relevant scheduled-run model. It also cautions that manually triggered intervals depend on the timetable and trigger path rather than automatically matching a supplied logical date. DAG runs and intervals
Use the actual timetable contract to determine which records belong to a run. Make timezone and boundary conventions explicit. A daily interval in a business timezone is not automatically a fixed twenty-four-hour UTC window, especially around clock changes. The example deliberately uses explicit UTC boundaries and makes no timetable claim.
Idempotency depends on identity and inputs
Airflow's best-practices guide recommends transaction-like tasks, consistent rerun outcomes, specific input partitions and avoiding duplicate-producing inserts. The underlying requirement is that the same intended work does not create a new unintended effect on every attempt. Task best practices
Choose a stable output identity and define what a repeated attempt does. It might replace an interval result, upsert by a meaningful key, or publish a new reviewed version. The storage operation must support the promised behavior under failures and concurrency. A dictionary assignment in the example does not establish those external guarantees.
Stable identity is insufficient when inputs move. Reading whatever is latest on each retry can produce different results for the same logical interval. Record source partitions or snapshots and the applicable code version when reproducibility matters. If intentional correction changes the output, make that revision policy visible rather than disguising it as the original result.
This is an editorial operating matrix. No scheduler, database, object store or remote side effect was exercised here.
Catchup and backfill need bounded operation
The documentation describes catchup as creating runs for eligible missing scheduled intervals and backfill as running a specified historical period. Review the actual Airflow version, timetable and configuration rather than assuming every deployment has the same default or command syntax.
Before creating historical work, enumerate the intended intervals and estimate input and downstream load. Limit concurrent work according to actual service capacity and coordinate any notifications or external updates. Reprocessing a year of data can be correct analytically while being disruptive operationally if it reissues every downstream effect.
Green orchestration does not prove correct data
Airflow derives DAG-run status from task states, including leaf tasks, and documents how trigger rules can produce surprising success status when earlier work failed. Review those rules and add checks for the data contract rather than relying on a final green box alone.
Verify output completeness, relevant counts, expected partition identity and consumer compatibility. A task that catches an exception and returns normally may look successful while publishing nothing useful. Keep failure information and data validation observable at the boundary the downstream consumer needs.
Best Practices
Define each task's input interval, source identity, output key, retry behavior and publication boundary. Keep the contract near the implementation and test it with boundary records. Use an authorized disposable environment to inject failures around publication and inspect the resulting external state.
Separate compute from irreversible effects where the design permits. A historical transformation can often be rerun safely while notifications or other actions require explicit deduplication and approval. Make those differences visible in the DAG and operating procedure instead of hiding them behind a generic retry count.
Test backfill behavior on a small representative range before increasing scope. Compare expected and actual intervals and outputs, inspect resource usage and downstream behavior, and define a stop condition. Retain the command/configuration and code version so another operator can reproduce the intended exercise.
Reverse the initial retry strategy when the external operation cannot safely repeat. A longer delay or more attempts does not repair an ambiguous side effect. Add durable identity, reconciliation or manual review as appropriate to the actual system. Likewise, reduce backfill scope when the chosen input version cannot support a consistent historical result.
Common Mistakes
Using the current time to select data
Appending output on every attempt
Treating backfills as harmless
Using commands from another major version
FAQ
How do I test a DAG's partition logic before running Airflow?
Keep interval selection and partition replacement in plain functions, like the example, and unit-test them with fixed intervals. Then test scheduling, task state, retries, executor behavior, and publication in a representative Airflow environment, because those depend on the deployment.
Does idempotent mean inputs can change freely?
No. Define the work identity and the inputs it represents. If a correction intentionally changes data, record that revision and how consumers should interpret it rather than assuming every repeated attempt means the same thing.
What should a backfill plan contain?
The interval range, timetable, code/input identities, reprocessing and publication policy, capacity limits, downstream effects, validation and stop conditions. Start with a small rehearsal before authorizing a larger run.