Use DuckDB and Polars Together in Python Pipelines (2026) — stop picking a single winner. Scan and join with DuckDB, transform with Polars, and pass Arrow buffers between them with almost no copy cost.
You already have the face-off posts: DuckDB vs Polars benchmarks and Polars vs Pandas. This guide is the practical follow-up: how to use both in one pipeline.
When to reach for which
| Job | Prefer |
|---|---|
| SQL joins, CTEs, window functions on Parquet/CSV | DuckDB |
| Python expression chains, feature engineering, typed transforms | Polars |
| Larger-than-RAM scans with spill | DuckDB |
| Replacing a Pandas ETL script | Polars (then DuckDB for heavy SQL) |
| Production pipeline | Both over Apache Arrow |
Mental model: DuckDB is your in-process analytics SQL engine (see also DuckDB for Python developers). Polars is your fast DataFrame API. Arrow is the shared memory format.
Install
pip install duckdb polars pyarrow
# Python 3.10+ (3.13/3.14 fine for this pattern)
Pattern 1 — DuckDB scans, Polars transforms
import duckdb
import polars as pl
con = duckdb.connect()
# DuckDB: push filters + aggregate close to the files
agg = con.execute("""
SELECT
customer_id,
date_trunc('day', order_ts) AS day,
sum(amount) AS revenue,
count(*) AS orders
FROM read_parquet('data/orders/*.parquet')
WHERE order_ts >= DATE '2026-01-01'
GROUP BY 1, 2
""").pl() # -> Polars DataFrame (Arrow-backed)
# Polars: expressive column work
features = (
agg
.with_columns(
(pl.col("revenue") / pl.col("orders")).alias("aov"),
pl.col("day").dt.weekday().alias("weekday"),
)
.filter(pl.col("orders") >= 3)
.sort(["customer_id", "day"])
)
print(features.head())
Pattern 2 — Register a Polars frame, finish in SQL
import duckdb
import polars as pl
events = pl.DataFrame({
"user_id": [1, 1, 2, 2, 2],
"event": ["view", "buy", "view", "view", "buy"],
"ts": [
"2026-09-01 10:00:00",
"2026-09-01 10:05:00",
"2026-09-01 11:00:00",
"2026-09-01 11:02:00",
"2026-09-01 11:10:00",
],
}).with_columns(pl.col("ts").str.to_datetime())
con = duckdb.connect()
con.register("events", events)
funnel = con.execute("""
SELECT
user_id,
count(*) FILTER (WHERE event = 'view') AS views,
count(*) FILTER (WHERE event = 'buy') AS buys
FROM events
GROUP BY user_id
ORDER BY user_id
""").pl()
print(funnel)
Pattern 3 — One pipeline sketch
- DuckDB
read_parquet/read_csvwith predicates - SQL join + grain aggregation
.pl()into Polars for feature columns / ML-ready frame- Write Parquet from Polars (
sink_parquet/write_parquet)
import duckdb
import polars as pl
con = duckdb.connect()
daily = con.execute("""
SELECT
sku,
date_trunc('day', sold_at) AS day,
sum(qty) AS units,
sum(qty * price) AS gmv
FROM read_parquet('data/sales.parquet')
GROUP BY 1, 2
""").pl()
ranked = (
daily
.with_columns(
pl.col("gmv").rank(method="dense", descending=True).over("day").alias("rank_in_day")
)
.filter(pl.col("rank_in_day") <= 10)
)
ranked.write_parquet("data/top_skus_by_day.parquet")
Pitfalls to avoid
- Don’t bounce through Pandas or CSV “just to convert” — that reintroduces copies.
- Keep heavy multi-table joins in DuckDB; keep long expression trees in Polars.
- Be explicit about dtypes at the boundary (timestamps, decimals) so Arrow handoff stays clean.
- For notebooks, prefer lazy Polars (
scan_parquet) when the DuckDB step already reduced the grain.
Wrap-up
In 2026 the winning stack is usually DuckDB + Polars, not DuckDB or Polars. Use SQL where SQL is clearer, Polars where Python expressions are clearer, and Arrow so you don’t pay a tax to switch. Pair this with the vs-benchmarks post when you need numbers — use this post when you need a pipeline shape you can ship.