Apache Tez

About Apache Tez

An application framework for complex directed-acyclic-graph (DAG) based data processing tasks, built on top of Apache Hadoop YARN. Tez generalizes MapReduce to enable more efficient data processing pipelines with fewer read/write cycles.

Key Features

• 1DAG-based execution engine replacing MapReduce for Hive and Pig workloads
• 2In-memory data transfers between tasks eliminate intermediate disk writes
• 3Fine-grained resource allocation per task within a shared YARN container
• 4Supports both batch and interactive query patterns
• 5Tez UI for real-time DAG visualization and task-level performance profiling

How Python Data Engineers Use Apache Tez

Python data engineers interact with Apache Tez indirectly — Hive and Pig automatically use Tez as their execution engine when configured, delivering faster query results. Python pipelines that invoke Hive queries via `PyHive` or subprocess benefit from Tez's in-memory task chaining without any code changes to the Python layer.