Luigi

About Luigi

Developed by Spotify, Luigi helps build complex pipelines of batch jobs, handling dependency resolution, workflow management, and task visualization.

Key Features

• 1Task-based DAG definition with explicit dependency declarations
• 2Built-in target system ensures idempotent, resumable pipelines
• 3Central scheduler with web UI for monitoring task status
• 4Rich library of built-in tasks for Hadoop, Hive, S3, and Postgres
• 5Easy to extend with custom Task and Target subclasses

How Python Data Engineers Use Luigi

Luigi is used at companies like Spotify to orchestrate multi-step batch jobs — engineers subclass `luigi.Task`, define `requires()` for dependencies and `output()` for targets, and Luigi's scheduler ensures each task runs only once its upstream tasks have produced their outputs, with automatic retry on failure.