Apache NiFi

Platform to programmatically author, schedule, and monitor workflows. Allows for complex pipeline construction and efficient task management with robust dependency handling.

Developed by Spotify, Luigi helps build complex pipelines of batch jobs, handling dependency resolution, workflow management, and task visualization.

Workflow management system designed for modern infrastructure, with a focus on simplicity, ease of use, and flexibility in defining and executing workflows.