Conditionals in Generator Expressions – Memory-Efficient Filtering 2026

Conditionals in Generator Expressions – Memory-Efficient Filtering 2026

Adding conditionals to generator expressions combines the power of filtering with lazy evaluation. This pattern is extremely useful in data science when you need to process large datasets while keeping memory usage minimal.

TL;DR — Two Ways to Add Conditions

• Filtering: (expr for item in iterable if condition)
• Conditional Value: (expr_if_true if condition else expr_if_false for item in iterable)

1. Basic Filtering with if

scores = [85, 92, 78, 95, 88, 76, 91, 65]

# Generator that yields only high scores
high_scores = (score for score in scores if score >= 90)

for score in high_scores:
    print(score)

2. Conditional Expressions (if-else) in Generators

scores = [85, 92, 78, 95, 65, 45]

# Categorize on-the-fly without creating a full list
categories = (
    "High" if s >= 90 
    else "Medium" if s >= 75 
    else "Low" 
    for s in scores
)

for cat in categories:
    print(cat)

3. Real-World Data Science Examples

import pandas as pd
# Create a small demo file so this example runs as-is
# (small stand-in for large_sales_data.csv; in your project this is your real large file)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("large_sales_data.csv", index=False)

df = pd.read_csv("large_sales_data.csv")

# Example 1: Memory-efficient high-value transaction filter
high_value_revenue = sum(
    row.amount * 1.1 
    for row in df.itertuples() 
    if row.amount > 1500 and row.region == "North"
)

# Example 2: Lazy feature engineering with conditions
processed_amounts = (
    0 if amount <= 0 
    else round(amount ** 0.5, 2) 
    for amount in df["amount"] 
    if pd.notna(amount)
)

# Example 3: Streaming customer categorization
premium_customers = (
    row.customer_id 
    for row in df.itertuples() 
    if row.amount > 2000 
    and row.region in ("North", "West")
)

for cust_id in premium_customers:
    print(f"Premium customer: {cust_id}")

4. Best Practices in 2026

• Use filtering if at the end for selecting rows
• Use if-else inside for value transformation
• Combine with sum(), max(), or a single for loop to keep memory low
• Avoid deeply nested conditionals — they reduce readability
• For very complex logic, use a traditional generator function with yield

Conditions Inside Generators

Generator expressions accept the same two kinds of conditions as list comprehensions: a trailing if filters items, and an if-else expression chooses a value for each item. The only difference is that the work happens lazily.

Filtering lazily means that rejected items are never stored. Combined with next() or any(), it lets you stop as soon as the first match is found, without scanning the rest of the data.

For clarity, give complex filters a name with a small function, and keep each generator focused on one step.

Worked Example: First Match and Running Counts

We find the first large order lazily and count categories without building lists.

orders = [120, 45, 980, 0, 310, 1500, 75]

first_large = next((o for o in orders if o >= 900), None)
print(first_large)

large_count = sum(1 for o in orders if o >= 300)
print(large_count)

labels = ("large" if o >= 300 else "small" for o in orders if o > 0)
print(list(labels))

checked = 0
def is_huge(o):
    global checked
    checked += 1
    return o >= 1000
print(any(is_huge(o) for o in orders), "checked:", checked)

The first large order is 980, and three orders reach 300 or more. The counter shows that any() checked six values and stopped at 1500 without looking at the last one.

Common Mistakes

• Building a list just to count matching items.
• Putting if-else after the for clause.
• Expecting a filtered generator to keep its length.
• Writing filters with side effects in real code.

Quick Summary

• Trailing if filters lazily.
• if-else chooses values lazily.
• next() and any() stop early.
• sum(1 for ...) counts without lists.

Conclusion

Conditionals in generator expressions give you the best of both worlds: clean, readable filtering/transformation combined with excellent memory efficiency. In 2026 data science workflows, this pattern is preferred when processing large files, streaming data, or building memory-conscious pipelines.

Next steps:

• Replace list comprehensions that only get iterated once with conditional generator expressions to reduce memory footprint in your data processing code

Related Tutorials

• Generator Expressions in Python – Memory-Efficient Data Processing 2026
• Conditionals in List Comprehensions