// data sources

Free Datasets for Python Data Engineering

128 curated free datasets. APIs for real-time pipelines, downloadable files for batch processing and ML training.

128 datasets

idnamedatevalue

OpenWeatherMap API

API

Current weather, forecasts, and historical data for any location. Free tier gets you 1,000 calls/day.

#rest-api#json#weather#entertainment

idnamedatevalue

NASA API

API

Space imagery, Mars rover photos, and asteroid data from NASA. One of the most fun free APIs out there.

#rest-api#json#space#government

idnamedatevalue

Twitter API

API

Tweets, profiles, trends, and engagement metrics. Good for NLP and social analytics projects.

#rest-api#json#social-media#oauth

idnamedatevalue

GitHub API

API

Repos, commits, PRs, and org data. Perfect for developer analytics and open source research.

#rest-api#json#oauth#api-key-required

idnamedatevalue

Reddit API

API

Posts, comments, and upvotes via PRAW. Great starting point for sentiment analysis and NLP.

#rest-api#json#social-media#machine-learning

idnamedatevalue

Wikipedia API

API

Article content, summaries, and page views. Massive corpus, completely free and open.

#rest-api#json#education#news

idnamedatevalue

OpenAI API

API

Access GPT language models, embeddings, and image generation tools from OpenAI. Commonly used in dat

#rest-api#json#machine-learning#paid

idnamedatevalue

WeatherAPI

API

Access current conditions, hourly forecasts, 14-day outlooks, historical weather, and astronomy data

#rest-api#json#weather#real-time

idnamedatevalue

Spotify API

API

Access music metadata, audio features (tempo, energy, danceability), playlist data, artist catalogue

#rest-api#json#entertainment#oauth

idnamedatevalue

Google Books API

API

Programmatically access metadata for millions of digitised books including titles, authors, publishe

#rest-api#json#education#entertainment

idnamedatevalue

Google Maps Platform APIs

API

A suite of APIs covering Maps, Places, Geocoding, Distance Matrix, and Directions from Google. Used

#rest-api#json#entertainment#maps

idnamedatevalue

YouTube Data API

API

Access YouTube video metadata, channel statistics, playlist data, comments, captions, and trending c

#rest-api#json#entertainment#oauth

idnamedatevalue

NewsAPI

API

Access news articles, headlines, and metadata from over 80,000 sources worldwide, searchable by keyw

#rest-api#json#news

idnamedatevalue

CoinGecko API

API

Access real-time and historical cryptocurrency prices, market capitalisation, trading volume, exchan

#rest-api#json#finance#cryptocurrency

idnamedatevalue

Alpha Vantage API

API

Access real-time and historical stock prices, forex exchange rates, cryptocurrency data, and 50+ tec

#rest-api#json#finance#cryptocurrency

idnamedatevalue

OpenCage Geocoding API

API

Convert addresses to geographic coordinates and reverse geocode coordinates back to human-readable a

#rest-api#json#maps

idnamedatevalue

PokeAPI

API

A comprehensive RESTful API covering all Pokémon species, abilities, moves, types, items, evolutions

#rest-api#json#no-auth

idnamedatevalue

OMDB API

API

The Open Movie Database API offers access to a vast collection of movie and television show data, in

#rest-api#json#entertainment

idnamedatevalue

Dog CEO's Dog API

API

Provides access to a database of dog breed images, breed lists, and random dog photos. A lightweight

#rest-api#json#entertainment

idnamedatevalue

Cat Facts API

API

A lightweight REST API that returns random facts and trivia about cats. Useful for learning API inte

#rest-api#json#education#entertainment

idnamedatevalue

JokeAPI

API

Provides a collection of jokes categorised by type (programming, puns, dark, misc) with filtering by

#rest-api#json#entertainment#no-auth

idnamedatevalue

Chuck Norris API

API

A RESTful API serving Chuck Norris jokes with category filtering and search capabilities. Useful for

#rest-api#json#entertainment#no-auth

idnamedatevalue

Numbers API

API

Provides mathematical trivia, historical facts, and interesting properties for any number via a simp

#rest-api#json#education#no-auth

idnamedatevalue

The Breaking Bad API

API

Provides structured data about Breaking Bad characters, episodes, quotes, and deaths. A clean, well-

#rest-api#json#education#entertainment

PreviousPage 1 of 6Next

// common questions

Frequently Asked Questions

What is the difference between API datasets and downloadable datasets?

API datasets give you a live HTTP endpoint — you query them in real time and always get fresh data, but you need an internet connection and often an API key. Downloadable datasets are static files (CSV, JSON, Parquet, etc.) you fetch once and store locally, which is better for batch processing, offline analysis, and ML training where you need a fixed, reproducible snapshot.

How do I load these datasets into Python with pandas?

For downloadable datasets, use pd.read_csv(url) or pd.read_parquet(url) directly with the file URL. For APIs, use the requests library to fetch JSON and pass it to pd.json_normalize(). Many API datasets also have official Python clients — check each dataset's detail page for code examples.

What data formats are available across these datasets?

The most common formats are CSV (universal, works everywhere), JSON (APIs and nested data), Parquet (columnar, best for large-scale analytics with Spark or DuckDB), and XML (older government datasets). Filter by the format tag in the Domain rail to find datasets in a specific format.

Are these datasets free to use in production?

Most are free for non-commercial and research use; some APIs have rate limits on their free tier and paid plans for higher volume. Government and public domain datasets (data.gov, World Bank, etc.) are generally unrestricted. Always check the individual dataset's license before using it in a commercial product.

How do I integrate an API dataset into an Airflow or Prefect pipeline?

Wrap the API call in a Python operator or task function, store credentials in your secret manager (Airflow Variables/Connections or Prefect Blocks), and write the response to your data lake or warehouse. For rate-limited APIs, add retry logic with exponential backoff. The dataset detail pages include Python usage examples you can adapt directly.