Loading...
Stream Processing
Streaming SQL Database
★ 3.8
Scalable Stream Processing
★ 4.6
N/A — PostgreSQL extensionpip install pysparkN/A — PostgreSQL extensionpip install pysparkPython data engineers use PipelineDB via standard `psycopg2` connections — defining continuous views in SQL that automatically aggregate incoming rows. A Python ingestion process writes events to PipelineDB streams using standard INSERT statements, and the continuous view query results update in real time, making precomputed aggregates instantly available for dashboards.
Python data engineers use Spark Structured Streaming via PySpark to process high-volume Kafka streams at scale. A streaming job reads a Kafka topic as a DataFrame, applies transformations (filtering, aggregations, joins with static data), and writes results continuously to Delta Lake or a database — using the same PySpark syntax as batch jobs.
Individual Tool Pages