Apache ORC

About Apache ORC

The smallest, fastest columnar storage format for Hadoop workloads. ORC provides highly efficient compression, predicate pushdown, and ACID transaction support, making it ideal for Hive-based data warehousing.

Key Features

• 1Columnar storage format optimized for Hive and Hadoop workloads
• 2Built-in lightweight indexes (min/max, bloom filter) for predicate pushdown
• 3ACID transaction support in Hive with ORC as the backing format
• 4Stripe-based file structure with built-in statistics for query planning
• 5Native support in Hive, Spark, and Presto for data lake analytics

How Python Data Engineers Use Apache ORC

Python data engineers use `pyorc` to read and write ORC files when working with Hive-based data lake environments where ORC is the standard format. In PySpark pipelines, ORC is specified as the write format for tables that will be queried via HiveQL with ACID upsert support — Spark handles ORC read/write transparently via the DataFrame API.