Kryo

About Kryo

A fast and efficient object graph serialization framework for Java. Kryo is commonly used as the serialization backend for Apache Spark and other JVM-based data processing frameworks for high-performance data exchange.

Key Features

• 1High-performance Java/JVM object serialization library
• 2Significantly faster and more compact than Java native serialization
• 3Used as the default serializer in Apache Spark for RDD operations
• 4Supports registration of custom serializers for specific classes
• 5Works with Avro, Protobuf, and Thrift schemas

How Python Data Engineers Use Kryo

Python data engineers encounter Kryo when tuning PySpark job performance — enabling Kryo serialization in Spark config (`spark.serializer=org.apache.spark.serializer.KryoSerializer`) reduces shuffle data size and speeds up operations that cross network boundaries between Spark executors. PySpark's Python UDFs still use pickle for Python objects, but JVM-side data uses Kryo.