Pravega

About Pravega

An open-source storage system that provides a new abstraction — a stream — for continuous and unbounded data. Pravega offers auto-scaling, exactly-once semantics, and durable storage for building reliable streaming data ingestion pipelines.

Key Features

• 1Durable, elastic stream storage built for long-term event retention
• 2Auto-scaling stream segments based on write throughput automatically
• 3Transactional writes for exactly-once event ingestion semantics
• 4Tiered storage moves cold segments to S3 or HDFS for cost efficiency
• 5Flink Pravega connector for native stream processing integration

How Python Data Engineers Use Pravega

Python data engineers use the `pravega-grpc-gateway` or Python client to produce durable event streams to Pravega. Unlike Kafka, Pravega retains streams indefinitely with cost-tiered storage — Python pipelines use this for event sourcing architectures where the full history of events must be replayable for reprocessing after logic changes.