// data sources

Free Datasets for Python Data Engineering

128 curated free datasets. APIs for real-time pipelines, downloadable files for batch processing and ML training.

128 datasets

idnamedatevalue

Mapbox API

API

The Mapbox API provides mapping and location-based services for developers to integrate customizable

#rest-api#json#entertainment#maps

idnamedatevalue

The New York Times API

API

Access NYT articles, book reviews, bestseller lists, movie reviews, event listings, and archive cont

#rest-api#json#news

idnamedatevalue

Wolfram Alpha API

API

Access Wolfram Alpha's computational knowledge engine for structured answers to mathematical, scient

#rest-api#json#science

idnamedatevalue

United Nations API

API

Access statistical datasets from the UN Statistics Division covering population, national accounts,

#rest-api#json#public-domain#free

idnamedatevalue

National Weather Service API

API

Access free weather forecasts, alerts, observations, and radar data from the US National Weather Ser

#rest-api#json#weather#government

idnamedatevalue

TMDb (The Movie Database) API

API

Access a community-built database of movies and TV shows including titles, cast, crew, genres, ratin

#rest-api#json#entertainment

idnamedatevalue

US Geological Survey (USGS) API

API

Access real-time and historical earthquake data, seismic hazard information, and geological datasets

#rest-api#json#government#public-domain

idnamedatevalue

Twilio API

API

A communications API providing access to global phone numbers, SMS messaging, voice calls, and Whats

#rest-api#json#education

idnamedatevalue

Wolfram Alpha Short Answers API

API

A simplified Wolfram Alpha endpoint that returns concise plain-text answers to factual and computati

#rest-api#json#machine-learning

idnamedatevalue

Datamuse API

API

A word-finding query engine that returns words related by meaning, sound, spelling, and context. Use

#rest-api#json#science

idnamedatevalue

Oxford Dictionaries API

API

Access definitions, pronunciations, synonyms, antonyms, translations, and usage examples from Oxford

#rest-api#json#education

idnamedatevalue

CoinMarketCap API

API

The API provides access to cryptocurrency market data, including prices, market capitalization, trad

#rest-api#json#finance#cryptocurrency

idnamedatevalue

IBM API

API

A collection of IBM Cloud APIs covering weather forecasts, financial market data, natural language p

#rest-api#json#weather

idnamedatevalue

Random User Generator API

API

Generate realistic synthetic user profiles including names, addresses, photos, email addresses, and

#rest-api#json#machine-learning

idnamedatevalue

FoodData Central API

API

It provides REST access to FoodData Central (FDC). It is intended primarily to assist application de

#rest-api#json#entertainment#government

idnamedatevalue

Systembolaget

API

The Swedish government monopoly liquor store API providing product catalogues, store locations, inve

#rest-api#json#government#public-domain

idnamedatevalue

AmiiboAPI

API

A RESTful database API covering all Nintendo Amiibo figures with attributes including series, charac

#rest-api#json#entertainment#no-auth

idnamedatevalue

MarvelAPI

API

The Marvel Comics API allows developers everywhere to access information about Marvel's vast library

#rest-api#json#entertainment

idnamedatevalue

PolygonStockAPI

API

Provides real-time and historical data for stocks, options, indices, forex, and cryptocurrencies. Wi

#rest-api#json#finance

idnamedatevalue

MusicBrainzAPI

API

An open-source music encyclopaedia API providing structured data on artists, albums, recordings, lab

#rest-api#json#entertainment#machine-learning

idnamedatevalue

OpenBreweryAPI

API

A free, open-source database API of breweries worldwide with details on beer types, locations, addre

#rest-api#json#entertainment#machine-learning

idnamedatevalue

NASA Exoplanet API

API

Access confirmed exoplanet data collected by NASA's Kepler, K2, and TESS missions, including orbital

#rest-api#json#science#space

idnamedatevalue

OpenAQ API

API

Retrieve real-time and historical air quality measurements including PM2.5, PM10, ozone, NO2, and CO

#rest-api#json#entertainment#environment

idnamedatevalue

OpenStreetMap API

API

Access open-source global map data including roads, buildings, points of interest, land use, and adm

#rest-api#json#entertainment#maps

PreviousPage 2 of 6Next

// common questions

Frequently Asked Questions

What is the difference between API datasets and downloadable datasets?

API datasets give you a live HTTP endpoint — you query them in real time and always get fresh data, but you need an internet connection and often an API key. Downloadable datasets are static files (CSV, JSON, Parquet, etc.) you fetch once and store locally, which is better for batch processing, offline analysis, and ML training where you need a fixed, reproducible snapshot.

How do I load these datasets into Python with pandas?

For downloadable datasets, use pd.read_csv(url) or pd.read_parquet(url) directly with the file URL. For APIs, use the requests library to fetch JSON and pass it to pd.json_normalize(). Many API datasets also have official Python clients — check each dataset's detail page for code examples.

What data formats are available across these datasets?

The most common formats are CSV (universal, works everywhere), JSON (APIs and nested data), Parquet (columnar, best for large-scale analytics with Spark or DuckDB), and XML (older government datasets). Filter by the format tag in the Domain rail to find datasets in a specific format.

Are these datasets free to use in production?

Most are free for non-commercial and research use; some APIs have rate limits on their free tier and paid plans for higher volume. Government and public domain datasets (data.gov, World Bank, etc.) are generally unrestricted. Always check the individual dataset's license before using it in a commercial product.

How do I integrate an API dataset into an Airflow or Prefect pipeline?

Wrap the API call in a Python operator or task function, store credentials in your secret manager (Airflow Variables/Connections or Prefect Blocks), and write the response to your data lake or warehouse. For rate-limited APIs, add retry logic with exponential backoff. The dataset detail pages include Python usage examples you can adapt directly.