Apache Pulsar

About Apache Pulsar

An open-source distributed pub-sub messaging system originally created by Yahoo. Pulsar provides multi-tenancy, geo-replication, and unified messaging and streaming with a serverless compute framework for lightweight processing.

Key Features

• 1Multi-tenant messaging platform with built-in geo-replication
• 2Tiered storage moves cold topic data to S3 or GCS automatically
• 3Pulsar Functions for lightweight stream processing co-located with brokers
• 4Persistent and non-persistent topic modes for different durability needs
• 5Compatible with Kafka consumers via Kafka-on-Pulsar (KoP) protocol

How Python Data Engineers Use Apache Pulsar

Python data engineers use the `pulsar-client` Python SDK to produce and consume messages from Pulsar topics. Pulsar Functions can be written in Python to perform lightweight transformations — filtering, enriching, or routing messages — without deploying a separate Faust or Spark Streaming cluster. Pulsar's topic compaction and retention policies simplify stateful event stream management.