PyInns - Home

Filtering & Summing with Generators and Dask in Python 2026 – Best Practices

Combining Python generators with Dask is one of the most memory-efficient ways to filter and aggregate large or streaming datasets. In 2026, this pattern is widely used for processing logs, real-time data streams, and any dataset that cannot fit entirely in memory.

TL;DR — The Winning Pattern

Use a generator to yield filtered items lazily
Feed the generator into Dask using dd.from_delayed() or db.from_sequence()
Perform aggregation with Dask for parallelism
Compute only the final result

1. Pure Generator Approach (Memory Efficient)


def filtered_sales_generator(file_path):
    """Generator that filters and yields only relevant rows."""
    with open(file_path, 'r') as f:
        next(f)  # skip header
        for line in f:
            row = line.strip().split(',')
            try:
                amount = float(row[3])
                if amount > 1000 and row[5] == "completed":
                    yield {
                        "customer_id": int(row[0]),
                        "amount": amount,
                        "region": row[4]
                    }
            except:
                continue

# Convert generator to Dask Bag
import dask.bag as db
bag = db.from_sequence(filtered_sales_generator("large_sales.log"), npartitions=50)

# Perform parallel aggregation
result = (
    bag.groupby(lambda x: x["region"])
       .map(lambda group: {
           "region": group[0],
           "total_sales": sum(item["amount"] for item in group[1]),
           "transaction_count": len(group[1])
       })
       .compute()
)

print(result)

2. Hybrid Generator + Dask DataFrame (Recommended)


import dask.dataframe as dd
from dask import delayed
import pandas as pd

def chunk_generator(file_path, chunk_size=100_000):
    """Yield filtered pandas DataFrames as chunks."""
    import pandas as pd
    for chunk in pd.read_csv(file_path, chunksize=chunk_size):
        filtered = chunk[
            (chunk["amount"] > 1000) & 
            (chunk["status"] == "completed")
        ]
        if not filtered.empty:
            yield filtered

# Create delayed objects from generator
delayed_chunks = [delayed(chunk) for chunk in chunk_generator("sales_data.csv")]

# Build Dask DataFrame
ddf = dd.from_delayed(delayed_chunks)

# Now use full Dask power for aggregation
summary = ddf.groupby("region").amount.agg(["sum", "count"]).compute()

print(summary)

3. Best Practices in 2026

Use generators to filter data **before** it enters Dask to reduce memory pressure
Prefer Dask Bag for unstructured or line-based data
Use Dask DataFrame for tabular data with clear schema
Always provide proper `meta` when using `from_delayed()`
Rebalance partitions after heavy filtering with `.repartition(partition_size="256MB")`
Monitor memory usage in the Dask Dashboard

Conclusion

Filtering and summing with generators combined with Dask is one of the most powerful memory-efficient patterns in Python 2026. It allows you to process terabyte-scale or streaming data while keeping memory usage low and maintaining clean, readable code.

Next steps:

Replace your current list-based filtering loops with generator + Dask patterns
Related articles: Parallel Programming with Dask in Python 2026 • Filtering a Chunk in Dask – Best Practices in Python 2026 • Managing Data with Generators and Dask in Python 2026

TL;DR — The Winning Pattern

1. Pure Generator Approach (Memory Efficient)

2. Hybrid Generator + Dask DataFrame (Recommended)

3. Best Practices in 2026

Conclusion

Generating content...