Stream Processing Tools & Datasets for Python Data Engineering

Easy-to-use, powerful, and reliable system to process and distribute data, offering a web-based user interface for data flow management.

Distributed event streaming platform capable of handling trillions of events a day. Used for building real-time streaming data pipelines and applications with high-throughput, fault-tolerance, and scalability.

Framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Known for high performance in streaming data processing with exactly-once semantics.

Real-time computation system making it easy to process unbounded streams of data reliably. Fast and scalable distributed real-time computation framework for stream processing.

Stream processing library porting ideas from Kafka Streams to Python. Used for building high-performance and reliable real-time stream processing applications with Pythonic API.

Extension of Apache Spark API enabling scalable, high-throughput, fault-tolerant processing of live data streams. Integrated within Spark ecosystem for complex real-time data processing tasks.

Streaming data platform API-compatible with Apache Kafka but designed for better performance and easier operational management. Modern streaming platform for mission-critical workloads.

Advanced unified programming model for defining and executing data processing workflows that can run on any execution engine. Provides portability across multiple execution environments including Apache Flink, Apache Spark, and Google Cloud Dataflow. Ideal for building flexible, scalable data pipelines.

A distributed stream processing framework that uses Apache Kafka for messaging and Apache Hadoop YARN for fault tolerance, processor isolation, security, and resource management. Samza provides a simple API for building stateful stream processing applications.

An open-source framework for managing storage for real-time data processing on top of data lakes. Hudi provides record-level insert, update, and delete capabilities along with change streams, enabling incremental data pipelines on large-scale datasets.

A streaming SQL database that runs SQL queries continuously on incoming data streams. PipelineDB is built as a PostgreSQL extension, allowing you to use standard SQL to define continuous views over streaming data for real-time analytics.

A performant open-source Python ETL framework with a Rust runtime, supporting over 300 data sources. Pathway enables building real-time data processing pipelines with a simple Python API, handling both batch and streaming workloads seamlessly.

A streaming database built for IoT data storage and real-time processing. HStreamDB provides stream ingestion, storage, and processing in a unified platform, enabling end-to-end streaming data management for IoT and event-driven applications.

An API gateway built for event-driven architectures and streaming. Zilla natively supports Kafka, SSE, WebSocket, MQTT, and HTTP protocols, enabling seamless integration between REST/GraphQL clients and event-driven backends.

A framework for building real-time streaming data processing applications. SwimOS combines streaming data processing with a built-in state store and UI capabilities, enabling continuous intelligence applications that process and visualize data in real-time.

An open-source distributed pub-sub messaging system originally created by Yahoo. Pulsar provides multi-tenancy, geo-replication, and unified messaging and streaming with a serverless compute framework for lightweight processing.

An open-source storage system that provides a new abstraction — a stream — for continuous and unbounded data. Pravega offers auto-scaling, exactly-once semantics, and durable storage for building reliable streaming data ingestion pipelines.

A fully managed, cloud-based service from AWS for real-time data streaming and processing. Kinesis enables collecting, processing, and analyzing streaming data at any scale, with integrations across the AWS ecosystem.