Real-time Tools & Datasets for Python Data Engineering

An open-source document database designed for building real-time web applications. RethinkDB pushes query results to applications in real-time, eliminating the need for polling and making it ideal for collaborative apps and live dashboards.

A column-oriented, distributed data store designed for sub-second OLAP queries on event data. Druid is used for powering interactive analytical applications, real-time dashboards, and exploratory analytics on high-cardinality data.

An in-memory, ACID-compliant relational database that uses a shared-nothing architecture for horizontal scalability. VoltDB is designed for applications requiring high-velocity data ingestion with real-time decisioning and analytics.

A performant open-source Python ETL framework with a Rust runtime, supporting over 300 data sources. Pathway enables building real-time data processing pipelines with a simple Python API, handling both batch and streaming workloads seamlessly.

A streaming database built for IoT data storage and real-time processing. HStreamDB provides stream ingestion, storage, and processing in a unified platform, enabling end-to-end streaming data management for IoT and event-driven applications.

A framework for building real-time streaming data processing applications. SwimOS combines streaming data processing with a built-in state store and UI capabilities, enabling continuous intelligence applications that process and visualize data in real-time.

A no/low-code data pipeline platform that handles both batch and real-time data ingestion. Estuary Flow uses change data capture to stream data from databases and provides a visual interface for building and monitoring data pipelines.

A real-time data ingestion tool leveraging change data capture to stream database changes into data warehouses. Artie minimizes data latency by continuously syncing changes rather than running periodic batch extracts.

A pure-Python graphics and GUI library built on PyQt and NumPy. PyQtGraph provides fast, interactive scientific graphics for displaying real-time data, including 2D/3D plots, image display, and data analysis widgets.

A fully managed, cloud-based service from AWS for real-time data streaming and processing. Kinesis enables collecting, processing, and analyzing streaming data at any scale, with integrations across the AWS ecosystem.

An open-source CDC platform that monitors databases and streams every committed row-level change as a structured event. Debezium reads directly from database replication logs — capturing inserts, updates, and deletes in real time with no polling and no impact on query performance.