Python Tools & Datasets for Python Data Engineering

Powerful Python library for data manipulation and analysis, offering DataFrame structures for efficient data cleaning, transformation, and analysis. Often used in the transform phase of ETL processes.

Python package specifically designed for ETL tasks, offering tools for data extraction, transformation, and loading. Suitable for simpler, script-based ETL processes.

Python API for Apache Spark, enabling scalable and efficient data processing. Particularly useful for ETL processes involving large datasets that need parallel processing across a cluster.

Python library that facilitates the loading phase in ETL processes. Designed to simplify loading data into various data stores or processing systems.

Lightweight Extract-Transform-Load (ETL) framework for Python 3.6+. Allows writing ETL scripts in pure Python, particularly suited for simple and straightforward ETL tasks.

Modern data pipeline tool focused on automating data preparation and feature engineering for machine learning. Streamlines the data transformation process in ETL workflows.

Platform to programmatically author, schedule, and monitor workflows. Allows for complex pipeline construction and efficient task management with robust dependency handling.

Developed by Spotify, Luigi helps build complex pipelines of batch jobs, handling dependency resolution, workflow management, and task visualization.

Workflow management system designed for modern infrastructure, with a focus on simplicity, ease of use, and flexibility in defining and executing workflows.

Open-source data orchestrator for machine learning, analytics, and ETL. Focuses on development, production, and observation of data assets with integrated pipeline views.

Parallel computing library that scales Pandas workflows to larger-than-memory datasets. Enables parallel processing while maintaining a familiar Pandas-like interface for big data.

Fundamental library for numerical computing in Python. Supports large multi-dimensional arrays and matrices with a vast collection of mathematical functions for array operations.

Library for web scraping and parsing HTML/XML documents. Extensively used in data wrangling to clean, parse, and extract data from web sources.

Powerful web crawling and scraping framework for extracting, cleaning, and processing large volumes of web data. Essential for data wrangling from web sources.

Simple library for processing textual data with APIs for common NLP tasks. Essential for data wrangling when dealing with text data and natural language processing.

Data validation and settings management library using Python type annotations. Ensures data conforms to defined schemas with Python's typing module, perfect for FastAPI and modern Python apps.

ORM/ODM/framework-agnostic library for object serialization and deserialization. Converts complex data types to and from native Python datatypes with robust validation.

Lightweight and extensible data validation library supporting complex data structures with customizable validation rules. Highly flexible for various validation needs.

Validates Python data structures with straightforward syntax and clear error messages. Ensures structure and content adhere to specified schemas.

Library for validating JSON data against JSON Schema standards. Essential when working with JSON data formats to ensure schema compliance.

Flexible API for data validation on dataframe structures. Validates dataframes in real-time, integrates with pydantic and fastapi. Essential for production data pipelines.

Fast, simple, and powerful validation library with declarative validation rules. Optimized for performance when validating data from various sources.

Widely used ORM library providing a full suite of enterprise-level persistence patterns. Designed for efficient, high-performing database access with flexible SQL abstraction.

Part of Django web framework, allows defining data models entirely in Python. Provides powerful abstraction layer to translate Python code to SQL seamlessly.

Small, expressive ORM with simple and intuitive interface. Lightweight and easy to use, perfect for small to medium-sized applications prioritizing simplicity.

Unique ORM using generator expressions for queries. Intuitive and user-friendly, allowing complex queries in pure Python that mirror human language.

Popular ORM providing object-oriented interface with tables as classes and rows as instances. Supports variety of database backends with simplicity focus.

Easy-to-use asyncio ORM inspired by Django. Designed for async/await syntax, making it perfect for asynchronous applications and modern Python development.

Async ORM built on SQLAlchemy core for asyncio programming. Provides simple and intuitive API for asynchronous database interactions with high performance.

Lightweight database migration tool for use with SQLAlchemy. Alembic allows you to create, manage, and invoke change management scripts for your database, facilitating schema migrations as your application evolves.

Django's powerful built-in migration framework that comes bundled with Django. Allows you to change your database schema without losing data using a simple and intuitive API.

Extension that handles SQLAlchemy database migrations for Flask applications using Alembic. Provides command-line tools to manage and automate database migrations in Flask projects.

Database schema migration tool that lets you manage your database schema by applying and rolling back migration scripts written in pure SQL or Python. Simple and flexible approach to database migrations.

Provides a way to deal with database schema changes in SQLAlchemy projects. Extends SQLAlchemy to have database schema versioning and migration capabilities for managing database evolution.

The original migration tool for Django before built-in migrations were added in Django 1.7. Still relevant for maintaining or upgrading legacy Django applications running older versions.

Stream processing library porting ideas from Kafka Streams to Python. Used for building high-performance and reliable real-time stream processing applications with Pythonic API.

Lightweight WSGI web application framework easy to get started with and versatile for complex applications. Popular for building web APIs thanks to simplicity and extensibility.

Powerful and flexible toolkit for building Web APIs in Django. Highly recommended for adding API capabilities to Django applications with comprehensive features and excellent documentation.

Modern, fast (high-performance) web framework for building APIs with Python 3.7+ based on standard type hints. Features automatic API documentation, easy to use, and blazing fast execution.

Python web framework and asynchronous networking library. Particularly useful for long-polling, WebSockets, and applications requiring long-lived connections to each user.

Reliable, high-performance Python framework for building large-scale app backends and microservices. Encourages REST architectural style while remaining highly effective and minimalist.

Comprehensive tool helping data teams validate, document, and profile their data. Define expectations for your data ensuring it meets quality standards before processing.

Generates profile reports from pandas DataFrames. Excellent tool for quickly understanding data with interactive HTML reports including statistics, distributions, and correlations.

Python API for Deequ, AWS library built on Apache Spark for defining and verifying data quality constraints. Useful for large-scale data processing and quality verification.

Python library using machine learning to perform deduplication and entity resolution on structured data. Particularly useful for identifying and merging duplicate records.

Open-source data quality tool with CLI for defining, running, and monitoring data quality checks. Write tests to verify data meets conditions like missing values, ranges, or uniqueness.

Automatic tool for cleaning and preprocessing data. Handles missing values, encodes categorical data, and scales features making data preparation efficient.

Python package for automated data validation within Data Engineering pipelines. Engineered to ingest and validate tabular data against predefined schemas.

Comprehensive library for creating static, animated, and interactive visualizations in Python. Matplotlib is versatile and widely used for plotting graphs and charts with extensive customization options.

Built on top of Matplotlib, Seaborn provides a high-level interface for drawing attractive and informative statistical graphics, simplifying the creation of complex visualizations with beautiful default themes.

Plotly offers a range of interactive plotting options and is known for its advanced graphics and interactivity, supporting complex visualizations with ease. Perfect for creating web-based dashboards.

Bokeh focuses on building interactive, web-ready plots, which can be a powerful tool for creating dynamic visualizations that can easily be embedded in web applications.

Altair is a declarative statistical visualization library for Python, offering a simple and concise way to create a wide range of statistical plots based on a logical data mapping.

Versatile library providing a range of supervised and unsupervised learning algorithms. Known for its ease of use and efficiency for data mining and data analysis with classical ML algorithms.

End-to-end open-source platform for machine learning enabling complex computations with data flow graphs. Widely used for deep learning applications with robust production support.

Open-source machine learning library known for its flexibility, ease of use, and as a preferred tool for research in deep learning and artificial intelligence. Dynamic computation graphs.

High-level neural networks API designed for fast experimentation with deep neural networks. Runs on top of TensorFlow offering a user-friendly interface for building models.

Highly efficient implementation of gradient boosting frameworks designed for speed and performance. Widely used in machine learning competitions and practical applications for structured data.

Gradient boosting framework using tree-based learning algorithms. Designed for speed and efficiency, supporting large datasets and distributed computing for various ML tasks.

Algorithm for gradient boosting on decision trees developed by Yandex. Particularly effective for datasets with categorical features, known for robustness and handling overfitting well.

The official Amazon Web Services (AWS) SDK for Python. Enables Python developers to write software that makes use of services like Amazon S3, EC2, Lambda, and more. Provides easy-to-use, object-oriented API as well as low-level access to AWS services, making it simple to integrate Python applications with AWS infrastructure.

Google Cloud Platform's official client library for Python, enabling seamless integration with GCP services like Compute Engine, Cloud Storage, BigQuery, and Pub/Sub. Designed for a Pythonic, intuitive experience when interacting with Google Cloud services, with idiomatic code patterns and comprehensive documentation.

Microsoft's comprehensive Azure SDK for Python offering a complete set of packages to interact with Azure resources and services. Supports wide range of Azure services including Virtual Machines, Storage, Databases, AI services, and more. Provides tools for effective resource management and service interaction within Azure ecosystem.

Official SDK for interacting with various IBM Cloud services programmatically. Provides comprehensive support for IBM Cloud services including CIS, DNS, IAM, VPC, Watson AI, and more. Enables management and automation of IBM Cloud resources with Python, compatible with Python 3.6 and above.

Official SDK for writing code to manage Oracle Cloud Infrastructure resources. Supports wide range of Oracle Cloud services with functionalities for compute, storage, networking, databases, and more. Available across multiple operating systems and Python versions, providing robust interface for OCI resource management.

Python library designed to create Entity Relationship diagrams by extracting data from databases or SQLAlchemy models. Particularly useful for database designers and developers who need to visualize and interpret complex relationships within database systems. Generates diagrams automatically from your Python code.

Data discovery and metadata engine for improving productivity of data analysts, scientists, and engineers when interacting with data. Provides powerful search, data previews, and column-level lineage. Integrates seamlessly with Python environments and modern data stacks for comprehensive metadata management.

Powerful data management system that makes data accessible by providing tools to streamline publishing, sharing, finding, and using data. Aimed at data publishers wanting to make their data open and available. Features data cataloging, API generation, and visualization capabilities.

Open-source metadata service for collection, aggregation, and visualization of data ecosystem metadata. Provides common interface to track data lineage across your entire data platform. Offers Python client for integration and supports OpenLineage standard for lineage collection.

Open-source metadata platform for the modern data stack. Provides powerful and flexible metadata search, discovery, and lineage capabilities. Features real-time metadata updates, data quality monitoring, and governance workflows. Extensive Python SDK for automation and integration.

Foundational library for data manipulation and analysis in Python. Provides fast, flexible, and expressive data structures (DataFrames) designed for working with structured, tabular, and time series data. Essential tool for data wrangling with comprehensive features for indexing, grouping, merging, and filtering.

Lightweight and async-ready ORM designed to work with FastAPI and Starlette. Particularly suited for applications requiring asynchronous database operations with minimal overhead and modern Python async/await patterns.

Python is a high-level, interpreted programming language that has become the dominant language for data engineering. Known for its clear syntax, extensive standard library, and rich ecosystem of data-focused packages. Essential foundation for all Python data engineering work.

The standard package installer for Python. Used to install and manage Python packages from the Python Package Index (PyPI) and other repositories. Essential tool for managing dependencies in any Python project, comes bundled with Python installations.

Tools for creating isolated Python environments, allowing you to manage project-specific dependencies without conflicts. venv comes built into Python 3, while virtualenv offers additional features. Critical for professional Python development and maintaining clean, reproducible environments.

A fast, in-process analytical database with zero external dependencies. DuckDB is designed for analytical query workloads and integrates seamlessly with Python and Pandas, making it ideal for local data analysis and embedded analytics.

A performant open-source Python ETL framework with a Rust runtime, supporting over 300 data sources. Pathway enables building real-time data processing pipelines with a simple Python API, handling both batch and streaming workloads seamlessly.

A fast, scalable, open-source machine learning and artificial intelligence platform. H2O supports widely used statistical and machine learning algorithms including gradient boosted machines, random forests, deep learning, and more with Python and R APIs.

Apache Spark's scalable machine learning library consisting of common learning algorithms and utilities including classification, regression, clustering, collaborative filtering, and dimensionality reduction. MLlib integrates seamlessly with Spark's data processing pipelines.

An open-source Python framework for creating reproducible, maintainable, and modular data science code. Kedro applies software engineering best practices to data pipelines with built-in data catalog, pipeline visualization, and experiment tracking.

A lightweight Python library for defining data transformations as a directed acyclic graph (DAG). Hamilton uses Python function signatures to define dataflow, making pipelines self-documenting, testable, and easy to maintain.

An open-source data transformation framework for managing, testing, and deploying SQL and Python-based data pipelines. SQLMesh provides virtual data environments, automatic change detection, and incremental processing for efficient data warehouse management.

An end-to-end data pipeline tool that combines ingestion, transformation using SQL and Python, and data quality checks in a single CLI. Bruin simplifies building and managing data pipelines with a unified developer experience.

An open-source data integration platform for modern data teams. Airbyte offers 300+ pre-built connectors to sync data from APIs, databases, and files into data warehouses and lakes, with support for custom connector development.

A CLI and code-first ELT platform built by GitLab. Meltano uses the Singer protocol for data extraction and loading, integrating with dbt for transformations. It provides a declarative, version-controlled approach to managing data pipelines.

A CLI tool to copy data between databases with a single command. ingestr supports 50+ sources and destinations, making it easy to move data between systems without writing custom integration code or managing complex configurations.

An open-source data quality platform for the whole data platform lifecycle. DQOps provides over 150 built-in data quality checks, anomaly detection, and data quality dashboards for monitoring data pipelines across multiple data sources.

A data catalog tool that integrates into your CI system to prevent data quality issues before they reach production. Grai maps data lineage across your stack and automatically tests the impact of schema changes on downstream consumers.

A decorator-first DataFrame contracts and validation library for Python. daffy lets you define data contracts as decorators on functions, supporting Pandas, Polars, PyArrow, and Modin DataFrames for lightweight pipeline validation.

A modern, enterprise-ready business intelligence web application. Superset provides an intuitive interface for creating interactive dashboards, exploring data through SQL, and building rich visualizations without writing code.

An open-source tool for connecting to any data source, easily visualizing data, and sharing insights across your organization. Redash supports SQL queries, automated refreshes, and collaborative dashboard building.

A pure-Python graphics and GUI library built on PyQt and NumPy. PyQtGraph provides fast, interactive scientific graphics for displaying real-time data, including 2D/3D plots, image display, and data analysis widgets.

A natural language database query interface with automatic chart generation. QueryGPT allows users to ask questions about their data in plain English and receive SQL queries, results, and visualizations automatically.

A utility belt for handling data on AWS using Python. AWS Data Wrangler extends Pandas with connectors to AWS services like S3, Glue, Athena, Redshift, and more, simplifying data ingestion and extraction in AWS-based pipelines.

A delimited data preboarding framework that fills the gap between managed file transfer and the data lake. CsvPath provides a domain-specific language for validating, transforming, and routing CSV and other delimited files before ingestion.

A polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Kreuzberg extracts text and structured data from documents like PDFs, images, and office files for data ingestion pipelines.

A columnar storage format available to any project in the Hadoop ecosystem. Parquet provides efficient compression and encoding schemes, making it the de facto standard for analytical workloads in data lakes and warehouses.

Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. Protocol Buffers provide a compact binary format with strong typing and schema evolution, widely used in gRPC and high-performance data systems.

A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL provides a standard POSIX file system interface with features like deduplication, compression, and encryption.

A Python library by Capital One designed to make data analysis, monitoring, and sensitive data detection easy. Data Profiler automatically identifies data types, statistical patterns, and PII across structured and unstructured datasets.

An open-source data profiler focused on discovery and validation of complex patterns in data. Desbordante finds functional dependencies, association rules, and other data constraints that go beyond basic statistical profiling.

A Python library by Capital One that facilitates the comparison of two DataFrames across Pandas, Polars, Spark, and more. datacompy provides detailed match reports with configurable tolerance levels, ideal for validating data pipeline outputs.