Building Custom Generator Functions in Python – Advanced Memory-Efficient Patterns 2026

Building Custom Generator Functions in Python – Advanced Memory-Efficient Patterns 2026

When generator expressions are not enough, you can create powerful custom generator functions using the yield keyword. These functions are the most flexible way to implement memory-efficient data processing pipelines in data science.

TL;DR — How to Build a Generator Function

• Use def and yield instead of return
• The function becomes a generator when it contains yield
• Perfect for streaming, chunked, or complex multi-step processing

1. Basic Custom Generator Function

def high_value_generator(data):
    for row in data:
        if row["amount"] > 1500:
            yield {
                "customer_id": row["customer_id"],
                "amount": row["amount"],
                "region": row["region"]
            }

# Usage (demo records)
sales_records = [
    {"customer_id": 101, "amount": 2300.0, "region": "North"},
    {"customer_id": 102, "amount": 800.0, "region": "South"},
    {"customer_id": 103, "amount": 1750.5, "region": "East"},
]
for record in high_value_generator(sales_records):
    print(f"High value: {record['customer_id']} - ${record['amount']:.2f}")

2. Real-World Data Science Examples

import pandas as pd
import csv

# Example 1: Chunked CSV reader generator
def read_csv_in_chunks(file_path, chunk_size=10000):
    with open(file_path, "r", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        chunk = []
        for row in reader:
            chunk.append(row)
            if len(chunk) >= chunk_size:
                yield chunk
                chunk = []
        if chunk:
            yield chunk

# Example 2: Advanced feature engineering generator
def process_sales_with_features(df):
    for row in df.itertuples():
        profit = row.amount * 0.25
        category = "Premium" if profit > 500 else "Standard"
        yield {
            "customer_id": row.customer_id,
            "amount": row.amount,
            "profit": round(profit, 2),
            "category": category,
            "log_amount": round(row.amount ** 0.5, 2) if row.amount > 0 else 0
        }

# Create a small demo file so this example runs as-is
# (small stand-in for sales_data.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("sales_data.csv", index=False)
df = pd.read_csv("sales_data.csv")

# Usage
for processed_row in process_sales_with_features(df):
    print(processed_row)

3. Best Practices in 2026

• Use generator functions when you need complex logic, multiple yields, or stateful processing
• Keep generators pure and stateless when possible
• Combine with itertuples() for fast DataFrame iteration
• Use yield from to delegate to another generator
• Always document what the generator yields

How a Generator Function Runs

A function that contains yield becomes a generator function. Calling it does not run the body; it returns a generator object. Each next() call runs the body until the next yield, hands back the value and pauses with all local variables intact.

When the function returns or reaches the end, the generator raises StopIteration and the loop ends. Code after the last yield, such as closing a file, runs when the consumer asks for one more item.

Generator functions are ideal when producing items needs several statements, state between items, or resources such as open files.

Worked Example: Watching the Pauses

Print statements show exactly when the body of a generator runs.

def numbered(items):
    print("start")
    for n, item in enumerate(items, start=1):
        print(f"  about to yield {n}")
        yield n, item
    print("finished")

gen = numbered(["a", "b"])
print("generator created")
print(next(gen))
print(next(gen))
print(next(gen, "done"))

Nothing is printed when the generator is created. Each next() runs the body up to the next yield, and the final call prints finished and returns the default because the generator is exhausted.

Common Mistakes

• Expecting the body to run when the function is called.
• Using return value instead of yield inside the loop.
• Forgetting that the generator is single-use.
• Leaving resources open by not consuming generators fully.

Quick Summary

• yield turns a function into a generator.
• Calling it returns a generator object.
• Each next() resumes until the next yield.
• Local state survives between items.

Conclusion

Custom generator functions using yield give you maximum control while maintaining excellent memory efficiency. In 2026 data science projects, they are the go-to solution for building reusable, streaming data pipelines, chunked readers, and complex transformation workflows that would otherwise consume too much memory.

Next steps:

• Convert one of your complex data processing loops into a custom generator function and experience the memory savings

Related Tutorials

• Using Generator Functions in Python – Practical Patterns for Data Science 2026
• How to Build a Generator Function in Python – Step-by-Step Guide for Data Science 2026