Loading...
Serialization Formats
Columnar Storage Format
★ 4.8
Fast JVM Serialization Framework
★ 4.1
pip install pyarrowN/A — Java librarypip install pyarrowN/A — Java libraryParquet is the standard output format for Python data pipelines writing to a data lake. Engineers use `pandas.to_parquet()` or `pyarrow.parquet.write_table()` to write DataFrames as efficiently compressed columnar files. Reading is equally simple — `pd.read_parquet('s3://bucket/prefix/')` reads an entire partitioned dataset, with DuckDB and Athena capable of querying Parquet files directly without loading.
Python data engineers encounter Kryo when tuning PySpark job performance — enabling Kryo serialization in Spark config (`spark.serializer=org.apache.spark.serializer.KryoSerializer`) reduces shuffle data size and speeds up operations that cross network boundaries between Spark executors. PySpark's Python UDFs still use pickle for Python objects, but JVM-side data uses Kryo.
Individual Tool Pages