Parallel Programming with Dask in Python 2026: Intro Guide
Dask lets you use every CPU core on your laptop, and later a whole cluster, while writing code that looks like NumPy, pandas or plain Python. This guide is a hands-on introduction for 2026 using Dask 2026.8.0 on Python 3.13. Every example below was run on those versions.
Why Dask?
- Bigger than memory: data is split into chunks or partitions, so a 50 GB CSV folder can be processed on a 16 GB machine.
- Parallel by default: independent pieces of work run on several threads or processes at once.
- Familiar APIs:
dask.arraycopies NumPy,dask.dataframecopies pandas, anddask.delayedwraps any function. - Lazy: Dask first builds a task graph and only runs it when you call
.compute().
1. Install Dask
uv pip install "dask[complete]" # or: pip install "dask[complete]"The complete extra pulls in pandas, NumPy, PyArrow and the distributed scheduler. Check the version:
import dask
print(dask.__version__)2026.8.02. dask.delayed: parallelise ordinary functions
Decorate a function with @delayed and calling it returns a lazy placeholder instead of a result. Dask records the calls as a graph and runs independent calls in parallel.
import time
import dask
from dask import delayed
@delayed
def load(n):
time.sleep(0.5) # pretend this reads a file or calls an API
return list(range(n))
@delayed
def total(xs):
return sum(xs)
parts = [total(load(n)) for n in (10, 20, 30, 40)]
grand_total = delayed(sum)(parts)
start = time.perf_counter()
print(grand_total.compute()) # threads run the 4 loads in parallel
print(f"took {time.perf_counter() - start:.1f}s")1450
took 0.7sRun one after another, the four 0.5-second loads would take about 2 seconds. Dask runs them at the same time. For more patterns, see deferring computation with delayed.
3. dask.array: NumPy in chunks
import dask.array as da
x = da.random.default_rng(42).random((20_000, 20_000), chunks=(2_000, 2_000))
print(x) # lazy: nothing computed yet
print(x.npartitions, "chunks,", round(x.nbytes / 1e9, 1), "GB logical size")
col_means = x.mean(axis=0)
print(col_means[:3].compute().round(3))dask.array<random, shape=(20000, 20000), dtype=float64, chunksize=(2000, 2000), chunktype=numpy.ndarray>
100 chunks, 3.2 GB logical size
[0.497 0.5 0.5 ]The full array would need 3.2 GB of RAM, but Dask only keeps a few 32 MB chunks in memory at a time. Choosing a good chunk size matters a lot; read allocating memory for a Dask array for the details.
4. dask.dataframe: pandas across many files
from pathlib import Path
import numpy as np
import pandas as pd
import dask.dataframe as dd
# create 4 small CSV files so the example is self-contained
Path("sales").mkdir(exist_ok=True)
rng = np.random.default_rng(0)
for month in range(1, 5):
pd.DataFrame({
"region": rng.choice(["north", "south", "east", "west"], 50_000),
"amount": rng.integers(5, 500, 50_000),
}).to_csv(f"sales/2026-{month:02d}.csv", index=False)
df = dd.read_csv("sales/2026-*.csv")
print(df.npartitions, "partitions")
summary = df.groupby("region")["amount"].agg(["sum", "mean"]).compute()
print(summary.sort_index().round(1))4 partitions
sum mean
region
east 12623579 252.2
north 12513258 252.3
south 12636080 250.7
west 12597944 252.3A glob pattern turns a folder of files into one DataFrame with one partition per file. Since 2025 Dask DataFrame uses a query optimizer, so filters and column selections are pushed down before any data is read.
5. The distributed scheduler and dashboard
The default scheduler uses threads. For pure-Python code, or to watch progress live, start a local cluster:
from dask.distributed import Client, LocalCluster
import dask.array as da
if __name__ == "__main__":
with LocalCluster(n_workers=2, threads_per_worker=2, processes=True) as cluster, \
Client(cluster) as client:
print("dashboard:", client.dashboard_link)
x = da.ones((10_000, 10_000), chunks=1_000)
print((x + x.T).sum().compute())dashboard: http://127.0.0.1:8787/status
200000000.0Open the dashboard link while a job runs to see task progress, worker memory and the task stream. The same code connects to a real cluster by passing its address to Client("tcp://scheduler:8786").
6. persist() vs compute()
.compute()returns a normal NumPy or pandas object. Use it for small final results..persist()runs the graph but keeps the result as a Dask collection held in memory. Use it when you will reuse an intermediate result several times.
import dask.dataframe as dd
df = dd.read_csv("sales/2026-*.csv")
big = df[df.amount > 250].persist() # computed once, kept in memory
print(len(big)) # reuses the persisted partitions
print(big.amount.min().compute()) # no CSV re-read here either100430
251Best practices for 2026
- Aim for chunks or partitions of roughly 100 MB. Thousands of tiny tasks cause scheduler overhead.
- Prefer Parquet over CSV for large data: it is columnar, compressed and much faster to read.
- Call
.compute()once at the end. Usedask.compute(a, b)to compute several results in one pass. - Do not use Dask if the data fits comfortably in memory and pandas or Polars is already fast enough.
- Use the
if __name__ == "__main__":guard whenever you start aLocalClusterwith processes.
Conclusion
Dask gives you parallel, out-of-core computing with APIs you already know: delayed for custom functions, dask.array for NumPy-style data and dask.dataframe for pandas-style tables. Start on your laptop with the default scheduler, then move to dask.distributed when you need the dashboard or more machines. Continue with the full Parallel Programming With Dask in Python 2026 guide.