Loading...
Stream Processing
Python ETL for Real-Time Data
★ 4.3
Scalable Stream Processing
★ 4.6
pip install pathwaypip install pysparkpip install pathwaypip install pysparkPython data engineers use Pathway to write streaming pipelines in pure Python without learning a separate API — the DataFrame-like operators (filter, join, groupby, reduce) work identically on both batch files and live Kafka streams. Pathway is particularly suited for real-time feature engineering pipelines that need to update model inputs as new events arrive.
Python data engineers use Spark Structured Streaming via PySpark to process high-volume Kafka streams at scale. A streaming job reads a Kafka topic as a DataFrame, applies transformations (filtering, aggregations, joins with static data), and writes results continuously to Delta Lake or a database — using the same PySpark syntax as batch jobs.
Individual Tool Pages