Loading...
Stream Processing
Python Stream Processing
★ 4.5
Scalable Stream Processing
★ 4.6
pip install faust-streamingpip install pysparkpip install faust-streamingpip install pysparkFaust lets Python data engineers write Kafka stream processors entirely in async Python — defining an agent as a coroutine that processes messages from a topic and produces results to another. Engineers use Faust tables for stateful aggregations (e.g., running counts or session windows) that persist across restarts via RocksDB.
Python data engineers use Spark Structured Streaming via PySpark to process high-volume Kafka streams at scale. A streaming job reads a Kafka topic as a DataFrame, applies transformations (filtering, aggregations, joins with static data), and writes results continuously to Delta Lake or a database — using the same PySpark syntax as batch jobs.
Individual Tool Pages