// data sources

Free Datasets for Python Data Engineering

128 curated free datasets. APIs for real-time pipelines, downloadable files for batch processing and ML training.

128 datasets

idnamedatevalue

World Bank World Development Indicators (WDI)

Download

The World Bank World Development Indicators provides 1,600+ time-series indicators covering poverty,

#csv#batch-processing#health#education

idnamedatevalue

World Food Programme (WFP) Data

Download

WFP provides datasets on food security, hunger, malnutrition, food aid distribution, humanitarian as

#csv#batch-processing#machine-learning

idnamedatevalue

Global Financial Inclusion (Global Findex) Database

Download

The Global Findex Database offers datasets on financial inclusion indicators, access to banking, usa

#csv#batch-processing#finance#entertainment

idnamedatevalue

United Nations Conference on Trade and Development (UNCTAD) Data

Download

UNCTAD provides datasets on trade, investment, development, globalization, economic indicators and o

#csv#batch-processing#finance#entertainment

idnamedatevalue

Global Urban Observatory (GUO) Data

Download

The GUO offers datasets on urbanization, urban population growth, city demographics, slum population

#csv#batch-processing#entertainment#demographics

idnamedatevalue

Global Human Settlement Layer (GHSL)

Download

The Global Human Settlement Layer provides satellite-derived datasets on population density, built-u

#csv#batch-processing#demographics

idnamedatevalue

Global Entrepreneurship Monitor (GEM) Data

Download

The Global Entrepreneurship Monitor publishes annual datasets on startup activity, entrepreneurial a

#csv#batch-processing#environment

idnamedatevalue

NASA Earth Observing System Data and Information System (EOSDIS)

Download

EOSDIS provides access to a wide range of Earth observation datasets, including satellite imagery, c

#csv#batch-processing#weather#space

PreviousPage 6 of 6Next

// common questions

Frequently Asked Questions

What is the difference between API datasets and downloadable datasets?

API datasets give you a live HTTP endpoint — you query them in real time and always get fresh data, but you need an internet connection and often an API key. Downloadable datasets are static files (CSV, JSON, Parquet, etc.) you fetch once and store locally, which is better for batch processing, offline analysis, and ML training where you need a fixed, reproducible snapshot.

How do I load these datasets into Python with pandas?

For downloadable datasets, use pd.read_csv(url) or pd.read_parquet(url) directly with the file URL. For APIs, use the requests library to fetch JSON and pass it to pd.json_normalize(). Many API datasets also have official Python clients — check each dataset's detail page for code examples.

What data formats are available across these datasets?

The most common formats are CSV (universal, works everywhere), JSON (APIs and nested data), Parquet (columnar, best for large-scale analytics with Spark or DuckDB), and XML (older government datasets). Filter by the format tag in the Domain rail to find datasets in a specific format.

Are these datasets free to use in production?

Most are free for non-commercial and research use; some APIs have rate limits on their free tier and paid plans for higher volume. Government and public domain datasets (data.gov, World Bank, etc.) are generally unrestricted. Always check the individual dataset's license before using it in a commercial product.

How do I integrate an API dataset into an Airflow or Prefect pipeline?

Wrap the API call in a Python operator or task function, store credentials in your secret manager (Airflow Variables/Connections or Prefect Blocks), and write the response to your data lake or warehouse. For rate-limited APIs, add retry logic with exponential backoff. The dataset detail pages include Python usage examples you can adapt directly.