Loading...
File Systems & Storage
Scalable Network File System
★ 4.0
Cloud-Backed File System
★ 3.8
N/A — system package, install via package managerpip install s3qlN/A — system package, install via package managerpip install s3qlPython data engineers in HPC and on-premise environments use GlusterFS as a shared storage layer accessible by multiple pipeline worker nodes simultaneously. Python jobs write output files to a GlusterFS mount point, and other nodes in the cluster can immediately read those files without data movement — simplifying distributed batch processing without object storage dependencies.
Python data engineers use S3QL to mount cloud object storage as an encrypted local file system — writing pipeline output files to a mounted S3QL volume using standard Python file I/O (`open()`, `write()`) without any cloud SDK code. S3QL's encryption-at-rest is useful for storing sensitive pipeline outputs in cloud storage with a stronger encryption posture than default S3 SSE.
Individual Tool Pages