DataHub

About DataHub

Open-source metadata platform for the modern data stack. Provides powerful and flexible metadata search, discovery, and lineage capabilities. Features real-time metadata updates, data quality monitoring, and governance workflows. Extensive Python SDK for automation and integration.

Key Features

• 1Extensible metadata platform with a graph-based metadata model
• 2Automated ingestion connectors for 50+ sources via Python recipes
• 3Column-level lineage tracking across transformations and queries
• 4Data contracts for defining and enforcing schema and freshness expectations
• 5Browser-based search, governance workflows, and ownership management

How Python Data Engineers Use DataHub

Python data engineers use DataHub's Python SDK and ingestion framework to crawl metadata from databases, dbt projects, and Airflow — writing YAML recipe files that the `datahub` CLI ingests on a schedule. Custom Python emitters push metadata about internal pipeline assets that built-in connectors don't cover.