Dedupe

About Dedupe

Python library using machine learning to perform deduplication and entity resolution on structured data. Particularly useful for identifying and merging duplicate records.

Key Features

• 1Active learning-based deduplication — learns from human-labeled examples
• 2Record linkage across two datasets (entity resolution / matching)
• 3Blocking rules reduce comparison space from O(n²) to manageable size
• 4Supports string, categorical, lat/lon, datetime, and custom field types
• 5Trained models serializable to disk for reuse in production

How Python Data Engineers Use Dedupe

Data engineers use Dedupe to clean messy customer or product data where the same entity appears with slightly different names or addresses. Engineers label a small training set via the interactive CLI, Dedupe learns a similarity model, then applies it at scale to cluster duplicate records — outputting canonical entity IDs for use in downstream analysis.