Using Iterators to Load Large Files into Memory – Memory-Efficient Data Loading 2026

Using Iterators to Load Large Files into Memory – Memory-Efficient Data Loading 2026

When working with large files (several GB or more), loading the entire file into memory at once can cause out-of-memory errors. In 2026, using iterators is the standard approach for processing large files efficiently in data science pipelines.

TL;DR — Recommended Iterator-Based Loading

• Use open() with a for loop for line-by-line processing
• Use csv.DictReader for structured CSV files
• Use Pandas chunksize for large structured data
• Use generators (yield) for custom processing logic

1. Basic Line-by-Line Iteration (Most Memory Efficient)

# Create a small demo log file so this example runs as-is
with open("large_log.txt", "w", encoding="utf-8") as f:
    for n in range(1, 301):
        level = "ERROR" if n % 50 == 0 else "INFO"
        f.write(f"2026-01-01 12:{n % 60:02d}:00 {level} request {n} handled\n")

# For very large text or log files
with open("large_log.txt", "r", encoding="utf-8") as f:
    for line in f:                    # This is an iterator - loads one line at a time
        line = line.strip()
        if line and "ERROR" in line:
            print(line[:200])         # Process only what you need

2. Using csv Module with Iterator

import csv
import pandas as pd
# Create a small demo file so this example runs as-is
# (small stand-in for huge_dataset.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("huge_dataset.csv", index=False)

with open("huge_dataset.csv", "r", encoding="utf-8") as f:
    reader = csv.DictReader(f)        # This returns an iterator
    
    for row in reader:                # Processes row by row
        amount = float(row["amount"])
        if amount > 10000:
            print(f"High value transaction: {row['customer_id']} - ${amount:,.2f}")

3. Pandas chunksize – Best for Large Structured Files

import pandas as pd

# Create a small demo file so this example runs as-is
# (small stand-in for large_sales.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("large_sales.csv", index=False)

# Process large CSV in chunks (each chunk is a DataFrame)
chunk_size = 100_000

for chunk in pd.read_csv("large_sales.csv", chunksize=chunk_size, parse_dates=["order_date"]):
    # Process each chunk independently
    chunk_summary = chunk.groupby("region")["amount"].agg(["sum", "mean", "count"]).round(2)
    print(f"Processed chunk with {len(chunk)} rows")
    # Save or aggregate results from this chunk

4. Custom Generator for Maximum Control

def read_large_file(file_path, batch_size=10000):
    """Generator that yields batches of processed rows."""
    with open(file_path, "r", encoding="utf-8") as f:
        batch = []
        for line in f:
            batch.append(line.strip())
            if len(batch) >= batch_size:
                yield batch          # Yield the batch and pause
                batch = []
        if batch:
            yield batch

# Usage (demo file so this runs as-is)
with open("huge_file.txt", "w", encoding="utf-8") as f:
    f.write("first line\nsecond line\n" * 15000)

for batch in read_large_file("huge_file.txt"):
    # Process batch
    processed = [line.upper() for line in batch if line]
    print(f"Processed batch of {len(processed)} lines")

Best Practices in 2026

• Use iterators (line-by-line or chunked) for any file larger than available RAM
• Prefer csv.DictReader over manual splitting for CSV files
• Use Pandas chunksize when you need DataFrame operations on large files
• Write custom generators when you need full control over processing logic
• Always use the with statement to ensure files are properly closed

Why Iterators Save Memory

Reading a whole file with read() or readlines() loads every byte into memory at once. Iterating over the file object instead reads one line at a time, so memory use stays flat no matter how large the file is.

The same idea applies to csv.reader, pandas read_csv with chunksize, and custom generators.

Worked Example: Counting Errors in a Log

A small log file is created and scanned line by line, keeping only counts.

from collections import Counter

with open("app.log", "w", encoding="utf-8") as f:
    for n in range(1, 1001):
        level = "ERROR" if n % 100 == 0 else ("WARN" if n % 10 == 0 else "INFO")
        f.write(f"{level} event {n}\n")

counts = Counter()
with open("app.log", encoding="utf-8") as f:
    for line in f:
        counts[line.split(" ", 1)[0]] += 1

print(counts)

Only the Counter stays in memory, never the full file. The result shows 900 INFO, 90 WARN and 10 ERROR lines.

Common Mistakes

• Calling readlines() on very large files.
• Appending every line to a list during processing.
• Forgetting encoding= when opening text files.
• Trying to loop over the same file object twice.

Quick Summary

• Iterate over file objects line by line.
• Keep aggregates, not raw data.
• Use chunksize for large CSV files in pandas.
• Iterators are single-pass.

Conclusion

Using iterators to load large files is a critical skill for data scientists working with real-world datasets. In 2026, the best practice is to never load massive files entirely into memory. Instead, process them iteratively using file iterators, csv.DictReader, Pandas chunks, or custom generators. This approach keeps memory usage low and allows you to handle files of almost any size.

Next steps:

• Try processing a large CSV or log file using one of the iterator patterns shown above

Related Tutorials

• Loading Data in Chunks with Pandas – Memory-Efficient Processing 2026
• Using pandas read_csv iterator for Streaming Large Data