Using pandas read_csv iterator for Streaming Large Data

Using pandas read_csv iterator for Streaming Large Data

The chunksize parameter in pd.read_csv() turns the reader into a powerful iterator. This is the most common and effective way to stream and process very large CSV files without loading the entire dataset into memory.

TL;DR — Core Pattern

• Use pd.read_csv(..., chunksize=N)
• Each iteration gives you a normal pandas DataFrame chunk
• Process, enrich, or save each chunk independently

1. Basic Streaming with chunksize

import pandas as pd
# Create a small demo file so this example runs as-is
# (small stand-in for large_sales_data.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("large_sales_data.csv", index=False)

chunk_size = 100_000

for chunk in pd.read_csv("large_sales_data.csv", 
                        chunksize=chunk_size,
                        parse_dates=["order_date"],
                        dtype={"customer_id": "int32", "amount": "float32"}):
    
    print(f"Processing chunk with {len(chunk):,} rows")
    
    # Perform any operations on this chunk
    chunk["profit"] = chunk["amount"] * 0.25
    chunk["log_amount"] = chunk["amount"].apply(lambda x: round(x**0.5, 2) if x > 0 else 0)
    
    # Save or aggregate results
    # chunk.to_parquet(f"processed/chunk_{i}.parquet")

2. Real-World Streaming Pipeline

total_sales = 0.0
region_totals = {}

# Create a small demo file so this example runs as-is
# (small stand-in for 10GB_sales.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("10GB_sales.csv", index=False)

for chunk in pd.read_csv("10GB_sales.csv", chunksize=50000):
    # Update running aggregates
    total_sales += chunk["amount"].sum()
    
    # Group statistics per chunk
    chunk_group = chunk.groupby("region")["amount"].sum()
    for region, sales in chunk_group.items():
        region_totals[region] = region_totals.get(region, 0) + sales

print(f"Grand Total Sales: ${total_sales:,.2f}")
for region, total in sorted(region_totals.items(), key=lambda x: x[1], reverse=True):
    print(f"{region:12} : ${total:,.2f}")

3. Best Practices in 2026

• Choose chunk size based on available RAM (typically 50k–200k rows)
• Always specify dtypes and parse_dates to reduce memory usage
• Perform filtering and enrichment inside the loop
• Write processed chunks to Parquet or a database instead of keeping everything in memory
• Use low_memory=False only when necessary
• Monitor memory with tracemalloc during development

Streaming with the read_csv Iterator

read_csv with chunksize, or with iterator=True, returns a TextFileReader. You can loop over it for fixed-size chunks or call get_chunk(n) to pull a specific number of rows when you need more control.

Using the reader in a with block ensures the file is closed even if processing fails. Combined with usecols= and dtype=, this keeps memory low and parsing fast.

A typical streaming pipeline filters each chunk, writes the useful rows to an output file in append mode, and updates small summary statistics.

Worked Example: Filter and Append to a New File

Large orders are streamed from one CSV into another without loading the whole file.

import numpy as np
import pandas as pd

rng = np.random.default_rng(5)
pd.DataFrame({
    "order_id": range(20_000),
    "amount": rng.uniform(1, 1_000, 20_000).round(2),
}).to_csv("orders_big.csv", index=False)

kept = 0
with pd.read_csv("orders_big.csv", chunksize=5_000, usecols=["order_id", "amount"]) as reader:
    for n, chunk in enumerate(reader):
        big = chunk[chunk["amount"] > 900]
        big.to_csv("big_orders.csv", mode="w" if n == 0 else "a", header=(n == 0), index=False)
        kept += len(big)

check = pd.read_csv("big_orders.csv")
print(kept == len(check), check["amount"].min() > 900)

The first chunk writes the header and later chunks append without it, so the output is a single valid CSV. The final check confirms the row count and that only orders above 900 were kept.

Common Mistakes

• Writing the header for every chunk.
• Forgetting mode="a" and overwriting earlier chunks.
• Not closing the reader when errors occur.
• Reading all columns when only a few are needed.

Quick Summary

• chunksize or iterator=True returns a reader.
• Use a with block to close the file.
• Append filtered chunks to an output file.
• Use usecols= and dtype= to save memory.

Conclusion

Using pd.read_csv() with chunksize is the standard way to stream large CSV files in data science in 2026. It turns a potentially crashing script into a stable, memory-efficient pipeline. Combine it with proper dtype specification, chunked processing, and incremental aggregation to handle files of almost any size on standard hardware.

Next steps:

• Take one of your large CSV files and rewrite the loading script using chunksize iterator

Related Tutorials

• Loading Data in Chunks with Pandas – Memory-Efficient Processing 2026
• Generators for Handling Large Data Limits – Memory-Efficient Processing in Python 2026