Pathway

About Pathway

A performant open-source Python ETL framework with a Rust runtime, supporting over 300 data sources. Pathway enables building real-time data processing pipelines with a simple Python API, handling both batch and streaming workloads seamlessly.

Key Features

• 1Python-native stream processing framework with a DataFrame-like API
• 2Unified batch and streaming: the same code runs on historical and live data
• 3Incremental computation engine automatically handles late data corrections
• 4Connectors for Kafka, S3, PostgreSQL, and local files
• 5LLM App framework for building AI pipelines on live data streams

How Python Data Engineers Use Pathway

Python data engineers use Pathway to write streaming pipelines in pure Python without learning a separate API — the DataFrame-like operators (filter, join, groupby, reduce) work identically on both batch files and live Kafka streams. Pathway is particularly suited for real-time feature engineering pipelines that need to update model inputs as new events arrive.