Self-Hosted Tools & Datasets for Python Data Engineering

The world's most popular open source relational database, widely used for web applications and data-driven projects. MySQL offers robust SQL support, replication, and a mature ecosystem of tools and connectors for Python integration.

An enhanced, drop-in replacement for MySQL created by the original MySQL developers. MariaDB offers improved performance, additional storage engines, and stronger commitment to open source development while maintaining full MySQL compatibility.

An in-memory computing platform combining a database with an application server. Tarantool delivers high performance for both OLTP and data-intensive applications with built-in Lua scripting and persistent storage capabilities.

A scalable, event-driven, language-agnostic orchestration and scheduling platform. Kestra provides a declarative YAML-based workflow definition with a rich UI, supporting hundreds of plugins for data engineering, DevOps, and microservice orchestration.

A batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban provides a web UI for managing workflows, handles job dependencies, and supports pluggable job types for running various data processing tasks.

An open-source data integration platform for modern data teams. Airbyte offers 300+ pre-built connectors to sync data from APIs, databases, and files into data warehouses and lakes, with support for custom connector development.

A CLI and code-first ELT platform built by GitLab. Meltano uses the Singer protocol for data extraction and loading, integrating with dbt for transformations. It provides a declarative, version-controlled approach to managing data pipelines.

An open-source data quality platform for the whole data platform lifecycle. DQOps provides over 150 built-in data quality checks, anomaly detection, and data quality dashboards for monitoring data pipelines across multiple data sources.

A modern, enterprise-ready business intelligence web application. Superset provides an intuitive interface for creating interactive dashboards, exploring data through SQL, and building rich visualizations without writing code.

An open-source tool for connecting to any data source, easily visualizing data, and sharing insights across your organization. Redash supports SQL queries, automated refreshes, and collaborative dashboard building.

The easy, open-source way for everyone in your company to ask questions and learn from data. Metabase provides a simple interface for querying databases, building dashboards, and sharing data insights without requiring SQL knowledge.

A robust, open-source message broker that supports multiple messaging protocols including AMQP, MQTT, and STOMP. RabbitMQ provides reliable message delivery with flexible routing, clustering, and federation for distributed data ingestion pipelines.

An open-source data collector for building a unified logging layer. FluentD structures data as JSON and provides 500+ community-contributed plugins for connecting various data sources and outputs, widely used for log aggregation and forwarding.

An open-source event messaging platform that provides a REST API on top of Kafka-like queues. Nakadi simplifies event streaming by offering schema registration, data governance, and subscription-based consumption without direct Kafka client management.

An open-source CDC platform that monitors databases and streams every committed row-level change as a structured event. Debezium reads directly from database replication logs — capturing inserts, updates, and deletes in real time with no polling and no impact on query performance.

A unified, distributed storage system providing object, block, and file storage in a single platform. CEPH is designed for excellent performance, reliability, and scalability, widely used in cloud infrastructure and data center environments.

A scalable, distributed network file system suitable for data-intensive tasks such as cloud storage and media streaming. GlusterFS aggregates disk storage from multiple servers into a single global namespace for large-scale data access.

A software-defined storage solution that is distributed, parallel, scalable, fault-tolerant, and geo-redundant. LizardFS provides a highly available file system with automatic data replication and self-healing capabilities.

An open-source platform that delivers resilience and manageability to object-storage-based data lakes. lakeFS provides git-like branching, merging, and versioning for data, enabling safe experimentation and CI/CD workflows for data pipelines.

An open-source systems monitoring and alerting toolkit with a powerful multi-dimensional data model and flexible query language (PromQL). Prometheus is the standard for monitoring cloud-native and Kubernetes-based data infrastructure.

An open-source analytics and interactive visualization platform. Grafana connects to dozens of data sources including Prometheus, InfluxDB, and Elasticsearch to create rich monitoring dashboards for data infrastructure and pipeline health.

uv is an extremely fast Python package and project manager written in Rust by Astral. It replaces pip, pip-tools, virtualenv, pyenv, pipx, and poetry in a single unified tool, delivering 10–100x faster dependency resolution and installation through intelligent global caching.

Poetry is a Python dependency management and packaging tool that consolidates setup.py, requirements.txt, and Pipfile into a single pyproject.toml. It provides deterministic builds via a lockfile, automatic virtual environment management, and one-command PyPI publishing.