Apache Samza

About Apache Samza

A distributed stream processing framework that uses Apache Kafka for messaging and Apache Hadoop YARN for fault tolerance, processor isolation, security, and resource management. Samza provides a simple API for building stateful stream processing applications.

Key Features

• 1Stateful stream processing framework built on Kafka and YARN
• 2Local state stores per partition for fault-tolerant stateful computations
• 3Exactly-once semantics with Kafka changelog-backed state recovery
• 4Flexible windowing: tumbling, sliding, and session windows
• 5High-level Streams API and low-level Task API for custom processing

How Python Data Engineers Use Apache Samza

Python data engineers interact with Apache Samza primarily through its REST API or by bridging Python logic into Samza jobs via subprocess calls. More commonly, Python pipelines produce events to Kafka topics that Samza jobs consume for stateful aggregation — Python handles data ingestion and enrichment while Samza manages low-latency stateful stream processing at scale.