Lightweight Tools & Datasets for Python Data Engineering

Python package specifically designed for ETL tasks, offering tools for data extraction, transformation, and loading. Suitable for simpler, script-based ETL processes.

Lightweight Extract-Transform-Load (ETL) framework for Python 3.6+. Allows writing ETL scripts in pure Python, particularly suited for simple and straightforward ETL tasks.

Fundamental library for numerical computing in Python. Supports large multi-dimensional arrays and matrices with a vast collection of mathematical functions for array operations.

Library for web scraping and parsing HTML/XML documents. Extensively used in data wrangling to clean, parse, and extract data from web sources.

Simple library for processing textual data with APIs for common NLP tasks. Essential for data wrangling when dealing with text data and natural language processing.

Lightweight and extensible data validation library supporting complex data structures with customizable validation rules. Highly flexible for various validation needs.

Validates Python data structures with straightforward syntax and clear error messages. Ensures structure and content adhere to specified schemas.

Fast, simple, and powerful validation library with declarative validation rules. Optimized for performance when validating data from various sources.

Small, expressive ORM with simple and intuitive interface. Lightweight and easy to use, perfect for small to medium-sized applications prioritizing simplicity.

Lightweight and async-ready ORM designed to work with FastAPI and Starlette. Particularly suited for applications requiring asynchronous database operations with minimal overhead and modern Python async/await patterns.

A lightweight, distributed relational database built on SQLite and using the Raft consensus protocol. RQLite provides fault-tolerant, replicated SQLite with an easy-to-use HTTP API, ideal for edge computing and embedded applications.

A fast, in-process analytical database with zero external dependencies. DuckDB is designed for analytical query workloads and integrates seamlessly with Python and Pandas, making it ideal for local data analysis and embedded analytics.

A lightweight Python library for defining data transformations as a directed acyclic graph (DAG). Hamilton uses Python function signatures to define dataflow, making pipelines self-documenting, testable, and easy to maintain.

A CLI data integration tool specialized in moving data between databases and storage systems. Sling provides a simple command-line interface for extracting and loading data with support for incremental syncs, transformations, and multiple output formats.

A CLI tool to copy data between databases with a single command. ingestr supports 50+ sources and destinations, making it easy to move data between systems without writing custom integration code or managing complex configurations.

An open-source tool for live importing all your Google Sheets to your data warehouse. Google Sheets ETL automates the extraction of spreadsheet data into structured tables, bridging the gap between business users and data infrastructure.

A decorator-first DataFrame contracts and validation library for Python. daffy lets you define data contracts as decorators on functions, supporting Pandas, Polars, PyArrow, and Modin DataFrames for lightweight pipeline validation.

A lightweight Node.js ETL framework for moving data from databases to data lakes and data warehouses. db2lake provides simple configuration-driven extraction with support for incremental loads and multiple output formats.

A Python library by Capital One that facilitates the comparison of two DataFrames across Pandas, Polars, Spark, and more. datacompy provides detailed match reports with configurable tolerance levels, ideal for validating data pipeline outputs.

uv is an extremely fast Python package and project manager written in Rust by Astral. It replaces pip, pip-tools, virtualenv, pyenv, pipx, and poetry in a single unified tool, delivering 10–100x faster dependency resolution and installation through intelligent global caching.