Loading...
Orchestration Tools
Kubernetes-Native Workflow Engine
★ 4.6
SQL-Based Data Transformation
★ 4.1
pip install argo-workflowsnpm install -g @dataform/clipip install argo-workflowsnpm install -g @dataform/cliPython data engineers use Argo Workflows on Kubernetes to orchestrate containerised data pipelines where each task runs as an isolated Docker container. Python scripts are packaged as container images and executed as workflow steps, enabling reproducible, scalable batch processing. Argo is particularly popular in ML engineering teams for model training pipelines and in cloud-native data platforms where Kubernetes is the deployment target.
Python data engineers use Dataform's API and CLI from Python automation scripts to trigger pipeline runs, validate assertion results, and manage compilation output programmatically. Dataform is particularly used in Google Cloud environments as a managed dbt-like alternative — engineers write SQLX files and Python orchestration scripts call the Dataform API to schedule and monitor runs.
Individual Tool Pages