Loading...
Stream Processing
Distributed Event Streaming Platform
★ 4.8
Python ETL for Real-Time Data
★ 4.3
pip install confluent-kafkapip install pathwaypip install confluent-kafkapip install pathwayPython data engineers use `confluent-kafka-python` or `kafka-python` to produce events to topics and consume them in real-time. A common pattern is a Faust or plain consumer loop that reads messages, transforms them with pandas or Pydantic, and writes results to a database or another topic. Kafka is the backbone of event-driven data architectures in Python shops.
Python data engineers use Pathway to write streaming pipelines in pure Python without learning a separate API — the DataFrame-like operators (filter, join, groupby, reduce) work identically on both batch files and live Kafka streams. Pathway is particularly suited for real-time feature engineering pipelines that need to update model inputs as new events arrive.
Individual Tool Pages