Apache Gobblin

About Apache Gobblin

A universal data ingestion framework for Hadoop from LinkedIn. Gobblin handles the complete data ingestion lifecycle including extraction, transformation, quality checks, and publishing for both batch and streaming data sources.

Key Features

• 1Distributed data ingestion framework originally developed at LinkedIn
• 2Source and writer plugin model for custom connectors
• 3Compaction and deduplication of ingested data built in
• 4Throttling and rate limiting for polite API consumption
• 5Gobblin-as-a-Service for cloud-native execution on Kubernetes

How Python Data Engineers Use Apache Gobblin

Python data engineers interact with Gobblin by defining configuration files that specify source, extractor, converter, and writer plugins — executed as a Hadoop or standalone Java job. Python orchestration scripts manage Gobblin execution via REST API, monitor job completion, and process ingested output files with PySpark for downstream transformation and loading.