Ilum

About Ilum

A modular data lakehouse platform that simplifies the management and monitoring of Apache Spark clusters. Ilum provides a unified interface for running Spark jobs, managing data pipelines, and monitoring cluster health in lakehouse architectures.

Key Features

• 1Managed Apache Spark platform with a web UI for job submission and monitoring
• 2Multi-cluster management enabling different Spark versions and configs per team
• 3Built-in Jupyter notebook integration for interactive Spark development
• 4REST API for programmatic job submission and cluster management
• 5Role-based access control for managing multi-team Spark environments

How Python Data Engineers Use Ilum

Python data engineers use Ilum to submit and manage PySpark jobs without managing Spark cluster infrastructure directly. Ilum's REST API enables Python orchestration tools like Airflow to trigger Spark jobs programmatically as pipeline steps. It is used in organisations that need a self-hosted alternative to managed services like AWS EMR or Databricks, providing a control plane for Spark workloads running on Kubernetes or bare metal.