lakeFS

About lakeFS

An open-source platform that delivers resilience and manageability to object-storage-based data lakes. lakeFS provides git-like branching, merging, and versioning for data, enabling safe experimentation and CI/CD workflows for data pipelines.

Key Features

• 1Git-like branching and versioning for data lake objects on S3, GCS, or ADLS
• 2Branch data to experiment with transformations without affecting production
• 3Atomic commits group multiple object changes into a single consistent snapshot
• 4CI/CD hooks run data quality checks before merging data branches
• 5API and Python SDK for programmatic branch and merge operations

How Python Data Engineers Use lakeFS

Python data engineers use lakeFS to apply software engineering practices to data lake management. A pipeline writes to a lakeFS branch, data quality tests run against the branch, and the Python SDK merges the branch to main only on test success. This prevents bad pipeline outputs from reaching production consumers — the same guarantee that Git branches provide for code changes.