Kestra

About Kestra

A scalable, event-driven, language-agnostic orchestration and scheduling platform. Kestra provides a declarative YAML-based workflow definition with a rich UI, supporting hundreds of plugins for data engineering, DevOps, and microservice orchestration.

Key Features

• 1API-first workflow orchestration platform with a YAML-based DAG definition
• 2500+ built-in tasks for Python, SQL, HTTP, cloud services, and dbt
• 3Real-time topology view in the UI with live task execution logs
• 4Event-driven triggers: webhooks, schedules, file detection, and message queues
• 5Namespace-based organization for multi-team workflow management

How Python Data Engineers Use Kestra

Python data engineers define workflows in Kestra's YAML syntax, calling Python scripts as task steps using the `io.kestra.core.tasks.scripts.Python` task type. Kestra passes variables between tasks via outputs, enabling data to flow from a Python ingestion script through a SQL transformation task to a Python validation step — all orchestrated without writing custom Airflow operators.