Apache Hudi

About Apache Hudi

An open-source framework for managing storage for real-time data processing on top of data lakes. Hudi provides record-level insert, update, and delete capabilities along with change streams, enabling incremental data pipelines on large-scale datasets.

Key Features

• 1Data lake table format enabling upserts and deletes on object storage
• 2Copy-on-Write and Merge-on-Read table types for read vs. write optimization
• 3Incremental pull queries retrieve only changed records since a timestamp
• 4Timeline-based metadata tracks all table operations for audit and rollback
• 5Native integration with Spark, Flink, Hive, and Presto

How Python Data Engineers Use Apache Hudi

Python data engineers use Hudi with PySpark to build CDC (Change Data Capture) pipelines on data lakes — ingesting database change events from Kafka and applying upserts to Hudi tables on S3 using `UPSERT` operation type. Hudi handles deduplication and merge semantics automatically, enabling mutable data lake tables without full partition rewrites.