Using Generator Functions in Python – Practical Patterns for Data Science 2026

Using Generator Functions in Python – Practical Patterns for Data Science 2026

Generator functions (using yield) are one of the most powerful tools for writing memory-efficient and clean data processing code. Once you learn how to use them effectively, they become essential for handling large datasets, streaming data, and building reusable pipelines.

TL;DR — How to Use a Generator Function

• Call the function like a normal function
• It returns a generator object (lazy iterator)
• Consume it with for loop, sum(), next(), or list()

1. Basic Usage

def high_value_sales(records):
    for row in records:
        if row["amount"] > 1500:
            yield row

# Usage
sales_data = [{"customer_id": 101, "amount": 2300}, {"customer_id": 102, "amount": 900}]

for sale in high_value_sales(sales_data):
    print(f"High value sale by customer {sale['customer_id']}: ${sale['amount']}")

2. Real-World Data Science Patterns

import pandas as pd

# Create a small demo file so this example runs as-is
# (small stand-in for sales_data.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("sales_data.csv", index=False)
df = pd.read_csv("sales_data.csv")

# Pattern 1: Chunked processing
def process_in_chunks(df, chunk_size=50000):
    for start in range(0, len(df), chunk_size):
        chunk = df.iloc[start:start+chunk_size]
        yield chunk

for chunk in process_in_chunks(df):
    chunk["profit"] = chunk["amount"] * 0.25
    print(f"Processed {len(chunk)} rows")

# Pattern 2: Feature engineering pipeline
def enrich_sales_data(df):
    for row in df.itertuples():
        profit = row.amount * 0.25
        yield {
            "customer_id": row.customer_id,
            "amount": row.amount,
            "profit": round(profit, 2),
            "category": "Premium" if profit > 500 else "Standard",
            "log_amount": round(row.amount ** 0.5, 2) if row.amount > 0 else 0
        }

# Consume the generator
enriched = list(enrich_sales_data(df))   # Only if you need the full list
enriched_df = pd.DataFrame(enriched)

3. Advanced Consumption Patterns

# Sum without loading everything
total_profit = sum(row["profit"] for row in enrich_sales_data(df))

# Get first N items
import itertools
top_10 = list(itertools.islice(enrich_sales_data(df), 10))

# Manual control with next()
gen = enrich_sales_data(df)
first_record = next(gen)

4. Best Practices in 2026

• Use generator functions for any processing that can be done row-by-row or chunk-by-chunk
• Keep generators stateless and pure when possible
• Use them inside other functions or pipelines
• Document what the generator yields
• Convert to list only when you really need random access or multiple passes

Consuming Generators

Generators can be consumed in several ways: a for loop, built-ins like sum and max, list() to materialise everything, next() for single items, and itertools tools such as islice to take only part.

Choose the consumer that matches your need. If you only need the first ten results, islice stops the generator early and saves all the work for the rest. If you need everything twice, materialise once into a list.

Generators can also feed each other, forming pipelines where each stage does one small job.

Worked Example: Taking Only What You Need

An infinite generator is consumed safely with islice and filtered in a pipeline.

from itertools import islice

def order_ids(start=1000):
    n = start
    while True:
        yield n
        n += 1

def even(ids):
    for i in ids:
        if i % 2 == 0:
            yield i

first_five = list(islice(order_ids(), 5))
print(first_five)

first_even = list(islice(even(order_ids(1001)), 3))
print(first_even)

print(sum(islice(order_ids(1), 10)))

The generator never ends, but islice takes only what is requested, so the program finishes instantly. The pipeline yields 1002, 1004 and 1006, and the sum of the first ten ids from 1 is 55.

Common Mistakes

• Calling list() on an infinite generator.
• Materialising large generators when a sum is all you need.
• Forgetting that each consumer uses up the generator.
• Building deep pipelines without testing each stage.

Quick Summary

• Consume with for, sum, list, next or islice.
• islice limits how much work is done.
• Chain generators into pipelines.
• Materialise once if you need reuse.

Conclusion

Generator functions are the professional way to handle large-scale data processing in Python. In 2026 data science projects, they allow you to write clean, reusable, and extremely memory-efficient code. Master the pattern of building and consuming generator functions, and you will be able to process datasets of almost any size without running out of memory.

Next steps:

• Convert one of your current data processing loops into a generator function and integrate it into your workflow

Related Tutorials

• Building Custom Generator Functions in Python – Advanced Memory-Efficient Patterns 2026
• Generators for Handling Large Data Limits – Memory-Efficient Processing in Python 2026