Loading...
File Systems & Storage
Unified Distributed Storage
★ 4.4
Fault-Tolerant Distributed File System
★ 3.7
pip install cephN/A — system package, install via package managerpip install cephN/A — system package, install via package managerPython data engineers in on-premise or private cloud environments use Ceph's S3-compatible RADOS Gateway as a drop-in replacement for AWS S3 — boto3 and awswrangler work unchanged by pointing them at the Ceph endpoint URL. CephFS is mounted as a shared file system that multiple Python pipeline worker nodes read from and write to simultaneously.
Python data engineers in on-premise environments use LizardFS as a shared POSIX file system mounted across pipeline worker nodes. Python scripts write output files to the LizardFS mount and those files are immediately visible to all other nodes in the cluster — enabling simple shared-nothing pipeline patterns where workers write outputs that other workers consume without message queue coordination.
Individual Tool Pages