Data Ingestion
Database to Data Lake ETL
★ 3.5
Unified Logging Layer
★ 4.4
pip install db2lakeN/A — Ruby daemon, install via package managerpip install db2lakeN/A — Ruby daemon, install via package managerPython data engineers use db2lake to bootstrap data lake migration projects — extracting historical data from relational databases and writing it as partitioned Parquet files to S3 or HDFS. Once the initial migration is done, incremental extractions keep the lake in sync, and Python-based PySpark or DuckDB pipelines take over for ongoing processing.
Python data engineers use Fluentd to collect application logs from Python services and route them to Elasticsearch, BigQuery, or S3 for analysis. Python applications emit structured JSON logs which Fluentd's tail input plugin reads, applies filter plugins to parse and enrich, and forwards to the analytics destination — decoupling log production from storage decisions.
Individual Tool Pages