DuckDB

About DuckDB

A fast, in-process analytical database with zero external dependencies. DuckDB is designed for analytical query workloads and integrates seamlessly with Python and Pandas, making it ideal for local data analysis and embedded analytics.

Key Features

• 1Embedded OLAP database — runs in-process with no server to manage
• 2Reads Parquet, CSV, JSON, and Arrow files directly without loading
• 3Vectorized query execution for fast analytical queries on large datasets
• 4Python DataFrame integration: query pandas DataFrames and return results as DataFrames
• 5Can query S3, GCS, and HTTPS files directly via httpfs extension

How Python Data Engineers Use DuckDB

Python data engineers use DuckDB to run fast analytical SQL queries directly on Parquet files in a data lake without a database server. `duckdb.query('SELECT * FROM parquet_scan("s3://bucket/file.parquet")')` returns an Arrow table convertible to pandas — enabling complex aggregations on large files in seconds without loading them fully into memory.