Azkaban

About Azkaban

A batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban provides a web UI for managing workflows, handles job dependencies, and supports pluggable job types for running various data processing tasks.

Key Features

• 1Batch workflow scheduler originally developed at LinkedIn
• 2DAG-based job dependency management via properties files
• 3Web UI for workflow visualization, monitoring, and manual triggering
• 4HDFS and Hadoop ecosystem integration for data pipeline scheduling
• 5Email and SLA alerting on job failure and delay

How Python Data Engineers Use Azkaban

Python data engineers submit Azkaban workflow packages using the Azkaban REST API from Python scripts — uploading a ZIP of job definition files, triggering execution, and polling for completion status. Python jobs run as command-type Azkaban tasks that execute Python scripts on the cluster, with Azkaban managing the dependency ordering between pipeline steps.