datacompy

About datacompy

A Python library by Capital One that facilitates the comparison of two DataFrames across Pandas, Polars, Spark, and more. datacompy provides detailed match reports with configurable tolerance levels, ideal for validating data pipeline outputs.

Key Features

• 1Compares two pandas DataFrames and reports row-level and column-level differences
• 2Supports Spark DataFrames via the SparkCompare class for distributed comparison
• 3Produces a detailed report: unmatched rows, column mismatches, and value diffs
• 4Handles column name mapping when schemas differ between DataFrames
• 5Used for ETL validation, migration testing, and data reconciliation

How Python Data Engineers Use datacompy

Python data engineers use DataComPy to validate ETL migrations and pipeline refactors — comparing the output of a new pipeline version against the old output to confirm they produce identical results. A common use case is a database migration test: load the same source data through both the old and new ingestion code, compare the resulting DataFrames with DataComPy, and block the migration until all differences are resolved.