Loading Data in Chunks with Pandas – Memory-Efficient Processing 2026

Loading Data in Chunks with Pandas – Memory-Efficient Processing 2026

When dealing with very large datasets that don’t fit into memory, loading data in chunks using Pandas’ chunksize parameter is one of the most effective strategies. This approach processes data in manageable batches while keeping memory usage low.

TL;DR — How to Load Data in Chunks

• Use pd.read_csv(..., chunksize=N)
• Each chunk is a regular DataFrame
• Process each chunk independently and aggregate results
• Ideal for files > 1–2 GB

1. Basic Chunked Loading

import pandas as pd
# Create a small demo file so this example runs as-is
# (small stand-in for large_sales_data.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("large_sales_data.csv", index=False)

chunk_size = 100_000   # Adjust based on available memory

for chunk in pd.read_csv("large_sales_data.csv", 
                        chunksize=chunk_size,
                        parse_dates=["order_date"],
                        dtype={"customer_id": "int32", "amount": "float32"}):
    
    # Process each chunk here
    print(f"Processing chunk with {len(chunk)} rows")
    
    # Example: Calculate statistics for this chunk
    chunk_summary = chunk.groupby("region")["amount"].agg(["sum", "mean", "count"]).round(2)
    print(chunk_summary)

2. Aggregating Results Across All Chunks

total_sales = 0.0
region_totals = {}

for chunk in pd.read_csv("large_sales_data.csv", chunksize=100_000):
    # Update running totals
    total_sales += chunk["amount"].sum()
    
    # Accumulate by region
    for region, group in chunk.groupby("region"):
        if region not in region_totals:
            region_totals[region] = 0.0
        region_totals[region] += group["amount"].sum()

print(f"Grand Total Sales: ${total_sales:,.2f}")
for region, total in sorted(region_totals.items(), key=lambda x: x[1], reverse=True):
    print(f"{region:10} : ${total:,.2f}")

3. Best Practices in 2026

• Choose chunk size based on available RAM (typically 50,000 – 200,000 rows)
• Specify dtypes when reading to reduce memory usage
• Use parse_dates for date columns
• Accumulate results across chunks (running totals, group statistics, etc.)
• Consider writing processed chunks to Parquet for better performance
• Monitor memory usage during chunk processing

How chunksize Works

pd.read_csv(path, chunksize=n) does not return a DataFrame. It returns a reader object that yields DataFrames of up to n rows each, so only one chunk is in memory at a time.

Because each chunk is a normal DataFrame, you can filter, group and aggregate it with the usual tools. The key is to combine partial results correctly: sums and counts can be added, while means must be rebuilt from total sums and total counts.

Choose a chunk size that fits comfortably in memory, often tens or hundreds of thousands of rows, and pass dtype= and usecols= to reduce memory further.

Worked Example: A Correct Mean Across Chunks

We compute a per-region mean from chunks and compare it with the full-file result.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
pd.DataFrame({
    "region": rng.choice(["N", "S", "E"], 10_000),
    "amount": rng.uniform(10, 500, 10_000).round(2),
}).to_csv("orders.csv", index=False)

parts = []
for chunk in pd.read_csv("orders.csv", chunksize=2_500):
    parts.append(chunk.groupby("region")["amount"].agg(["sum", "count"]))

combined = pd.concat(parts).groupby(level=0).sum()
chunked_mean = combined["sum"] / combined["count"]

full_mean = pd.read_csv("orders.csv").groupby("region")["amount"].mean()
print(np.allclose(chunked_mean.sort_index(), full_mean.sort_index()))
print(chunked_mean.round(2))

Summing the per-chunk sums and counts and dividing at the end gives exactly the same means as reading the whole file. Averaging the four chunk means directly would be slightly wrong whenever chunks contain different numbers of rows per region.

Common Mistakes

• Averaging chunk averages instead of combining sums and counts.
• Expecting read_csv with chunksize to return a DataFrame.
• Storing every chunk in a list and concatenating them all.
• Choosing chunks too small, which slows processing.

Quick Summary

• chunksize returns an iterator of DataFrames.
• Combine sums and counts, then divide.
• Use dtype= and usecols= to save memory.
• Keep only partial results between chunks.

Conclusion

Loading data in chunks is a vital technique for handling large datasets that exceed available memory. In 2026, using Pandas chunksize combined with careful dtype specification and incremental aggregation allows you to process files of almost any size efficiently. This approach keeps memory usage predictable and enables you to work with massive datasets on standard hardware.

Next steps:

• Try processing one of your large CSV files using chunked loading and accumulate key statistics across all chunks

Related Tutorials

• Using Iterators to Load Large Files into Memory – Memory-Efficient Data Loading 2026
• Using pandas read_csv iterator for Streaming Large Data