Orchestration Tools

Platform to programmatically author, schedule, and monitor workflows. Allows for complex pipeline construction and efficient task management with robust dependency handling.

Developed by Spotify, Luigi helps build complex pipelines of batch jobs, handling dependency resolution, workflow management, and task visualization.

Easy-to-use, powerful, and reliable system to process and distribute data, offering a web-based user interface for data flow management.

Workflow management system designed for modern infrastructure, with a focus on simplicity, ease of use, and flexibility in defining and executing workflows.

Open-source data orchestrator for machine learning, analytics, and ETL. Focuses on development, production, and observation of data assets with integrated pipeline views.

Open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Designed for large-scale computational tasks with powerful workflow features.

An open-source Python framework for creating reproducible, maintainable, and modular data science code. Kedro applies software engineering best practices to data pipelines with built-in data catalog, pipeline visualization, and experiment tracking.

A lightweight Python library for defining data transformations as a directed acyclic graph (DAG). Hamilton uses Python function signatures to define dataflow, making pipelines self-documenting, testable, and easy to maintain.

A scalable, event-driven, language-agnostic orchestration and scheduling platform. Kestra provides a declarative YAML-based workflow definition with a rich UI, supporting hundreds of plugins for data engineering, DevOps, and microservice orchestration.

An open-source data transformation framework for managing, testing, and deploying SQL and Python-based data pipelines. SQLMesh provides virtual data environments, automatic change detection, and incremental processing for efficient data warehouse management.

An open-source framework and web-based IDE for managing datasets and their dependencies in data warehouses. Dataform uses SQL with added features like dependency management, testing, and documentation, now integrated into Google Cloud.

A batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban provides a web UI for managing workflows, handles job dependencies, and supports pluggable job types for running various data processing tasks.