Apache Giraph

About Apache Giraph

An iterative graph processing system built for high scalability, used at Facebook to analyze the social graph. Giraph processes billions of vertices and edges efficiently on Hadoop infrastructure using a vertex-centric programming model.

Key Features

• 1Iterative graph processing system modeled after Google's Pregel
• 2Bulk Synchronous Parallel (BSP) computation model for distributed graphs
• 3Runs on Hadoop YARN for cluster resource management
• 4Handles graphs with billions of vertices and edges
• 5Used by Facebook for social graph analysis at scale

How Python Data Engineers Use Apache Giraph

Python data engineers interact with Apache Giraph by preparing input graph data in the required format and submitting Giraph jobs via Hadoop command-line tools from Python subprocess calls. Graph computation results are written to HDFS and read back into Python pipelines via PySpark or pandas for further processing and feature engineering.