Loading...
File Systems & Storage
Hadoop Distributed File System
★ 4.4
Simple Distributed File System
★ 4.2
pip install hdfsN/A — Go binary, see seaweedfs.compip install hdfsN/A — Go binary, see seaweedfs.comPython data engineers interact with HDFS using `pyarrow.fs.HadoopFileSystem` or the `hdfs` Python client. PySpark accesses HDFS transparently via `spark.read.parquet('hdfs:///path/')` — the cluster configuration points Spark to the NameNode. Python scripts that manage file operations (listing, deleting, moving files) use the `subprocess` module to call `hdfs dfs` commands or the WebHDFS REST API.
Python data engineers use SeaweedFS's S3-compatible API with boto3 to store and retrieve pipeline artifacts, model binaries, and intermediate data files. Its optimized handling of billions of small files makes it a good fit for storing ML training sample files or pipeline checkpoint files that would create excessive metadata overhead in traditional distributed file systems.
Individual Tool Pages