Marquez

About Marquez

Open-source metadata service for collection, aggregation, and visualization of data ecosystem metadata. Provides common interface to track data lineage across your entire data platform. Offers Python client for integration and supports OpenLineage standard for lineage collection.

Key Features

• 1OpenLineage-compliant metadata service for tracking dataset inputs and outputs
• 2Namespace and job model links pipeline runs to their data lineage
• 3REST API for emitting and querying lineage events
• 4Integrations with Airflow, Spark, dbt, and Great Expectations
• 5Visual lineage graph in the Marquez UI for impact analysis

How Python Data Engineers Use Marquez

Python data engineers integrate Marquez with Airflow using the `openlineage-airflow` package, which automatically emits lineage events for each task — capturing which datasets a task reads and writes without any code changes. Engineers query the Marquez API to build impact analysis tools that identify downstream jobs affected by an upstream schema change.