Dagster

Most orchestrators think in tasks: run this, then that, on a schedule. Dagster thinks in assets — the tables, files, models, and reports a pipeline produces. You declare each asset as a Python function, its upstream assets as inputs, and Dagster derives the dependency graph, tracks when each asset was last materialized and from which code, shows lineage across the whole platform, and knows exactly what to recompute when something upstream changes or a historical partition needs fixing.

That asset-centric model makes Dagster particularly strong for analytics engineering and machine learning platforms, where the question is usually "is this table fresh and correct?" rather than "did task 7 succeed?". It integrates closely with dbt, ingestion tools, and warehouses, and runs open source or as the managed Dagster+ service.

TL;DR

Quick Example

Two daily-partitioned assets, a quality check, and a schedule:

From the UI or API, you can launch a backfill for any range of partitions — for example, rebuilding March after fixing a bug in daily_revenue.

Core Concepts

Software-Defined Assets

An asset definition says what data exists, how to compute it, and what it depends on. Dagster records each materialization with metadata — row counts, schema, code version, run ID — so the catalog answers when an asset was last updated, by which code, and whether upstream data has changed since. Asset groups, owners, tags, and kinds organize large graphs. Lower-level ops and jobs remain available for task-style work.

Resources and I/O Managers

Resources are configured objects injected into assets — warehouse connections, API clients, cloud storage — defined with ConfigurableResource and replaced with fakes in tests. I/O managers decide where asset outputs are stored and how inputs are loaded (Parquet in object storage, warehouse tables, DataFrames), keeping asset code focused on transformation. Assets can also write data themselves and return only metadata.

Partitions and Backfills

Each partition materializes independently, and Dagster tracks which partitions are missing, stale, or failed. Backfills launch runs for chosen partition ranges, with concurrency limits, and can run as a single run over many partitions where the compute engine supports it. See Pipeline Backfills and Historical Corrections.

Automation

Asset Checks and Data Quality

Asset checks validate assets after or alongside materialization: null rates, uniqueness, freshness, row-count changes, or custom logic. Checks can be blocking (stopping downstream assets) or advisory, and their results appear next to each asset. See Data Quality Management.

Integrations

See dbt.

Deployment

Dagster Compared With Airflow

See Airflow.

Best Practices

Model Data, Not Steps

Define an asset for each meaningful table or file. Graphs of assets stay readable as pipelines grow, and lineage becomes accurate automatically.

Make Partitions Idempotent

Each partition's materialization should fully replace that partition's data, so reruns and backfills are safe.

Test Assets as Functions

Call asset functions directly with small inputs and fake resources in unit tests, and use dg.materialize for small end-to-end tests.

Attach Checks to Critical Assets

Add freshness and quality checks to assets that feed dashboards or models, and alert on failures.

Common Mistakes

Hidden Dependencies

Appending in Partitioned Assets

One Giant Asset

FAQ

What is Dagster?

An open-source data orchestrator built around software-defined assets. It schedules and runs data pipelines while tracking lineage, freshness, partitions, and data quality for every asset produced.

What is a software-defined asset?

A Python function decorated with @asset that describes a piece of data — a table, file, or model — how to compute it, and which other assets it depends on.

Is Dagster better than Airflow?

They take different approaches. Dagster emphasizes assets, lineage, partitions, and local development; Airflow has the largest ecosystem and task-based scheduling. Teams focused on analytics and data products often prefer Dagster; teams with existing Airflow investment often stay.

How do backfills work in Dagster?

You select an asset or job with partitions and a range of partition keys; Dagster launches runs for those partitions, respecting concurrency limits, and records results per partition.

Does Dagster work with dbt?

Yes. The dbt integration loads every dbt model as an asset, preserving dependencies, so dbt models and Python assets appear in one lineage graph and can be scheduled together.

Related Topics

References