Dagster

About Dagster

Open-source data orchestrator for machine learning, analytics, and ETL. Focuses on development, production, and observation of data assets with integrated pipeline views.

Key Features

• 1Asset-centric model: pipelines are defined as software-defined assets
• 2Integrated lineage graph showing dependencies between data assets
• 3Type system for inputs and outputs enables early error detection
• 4First-class partitioning support for incremental and backfill runs
• 5Built-in integration with dbt, Spark, Airflow, and major cloud services

How Python Data Engineers Use Dagster

Dagster's asset-based model lets Python engineers define their ETL outputs as `@asset` functions — Dagster infers the dependency graph automatically. Engineers materialize assets on demand or on schedule, observe run history and lineage in the Dagit UI, and use Dagster's type system to catch schema mismatches between pipeline steps at development time.