Iterating Over Data in Python

Iterating Over Data in Python

Iteration is at the heart of data science workflows — from processing rows in a DataFrame to training models and generating reports. In 2026, writing efficient and Pythonic iteration code is essential for performance, readability, and scalability.

TL;DR — Recommended Iteration Patterns

• Use direct iteration (for item in data) instead of manual indexing
• Use enumerate() when you need the index/rank
• Use zip() when iterating over multiple sequences together
• For large DataFrames, prefer vectorized operations or itertuples()

1. Pythonic Iteration Patterns

scores = [85, 92, 78, 95, 88]

# Good - direct iteration
for score in scores:
    print(score)

# Good - with index using enumerate
for rank, score in enumerate(scores, start=1):
    print(f"Rank {rank}: {score}")

# Good - multiple sequences with zip
names = ["Alice", "Bob", "Charlie"]
for name, score in zip(names, scores):
    print(f"{name} scored {score}")

2. Iterating Over Pandas DataFrames

import pandas as pd

# Create a small demo file so this example runs as-is
# (in your project, sales_data.csv is your real export)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_id": range(1001, 1001 + n),
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("sales_data.csv", index=False)

df = pd.read_csv("sales_data.csv", parse_dates=["order_date"])

# Best for performance: itertuples()
for row in df.itertuples():
    if row.amount > 1000:
        print(f"High value order: {row.customer_id} - ${row.amount:.2f}")

# Acceptable for small datasets
for idx, row in df.iterrows():
    if row["region"] == "North":
        print(f"North sale: ${row['amount']:.2f}")

3. Best Practices for Iteration in Data Science 2026

• Prefer vectorized operations (df["amount"] * 1.1) over explicit loops when possible
• Use itertuples() instead of iterrows() for better performance
• Use enumerate() and zip() to make loops cleaner
• Avoid modifying the collection you are iterating over
• Use generators (yield) when processing very large or streaming data

Choosing an Iteration Style

Python offers several ways to go through data: for loops, comprehensions, generator expressions, built-ins like sum and any, and in pandas, vectorised operations. They differ in readability, speed and memory use.

A practical order of preference for data work is: vectorised pandas or NumPy first, then built-ins and comprehensions for plain Python data, then explicit loops when the logic is complex or has side effects.

When you must loop over DataFrame rows, itertuples() is much faster than iterrows(), and it keeps column types intact.

Worked Example: Three Ways to the Same Total

We compute a discounted total with a loop, a generator expression and a vectorised expression, and time them.

import time
import numpy as np
import pandas as pd

df = pd.DataFrame({"amount": np.random.default_rng(1).uniform(10, 100, 200_000)})

t0 = time.perf_counter()
loop_total = 0.0
for row in df.itertuples():
    loop_total += row.amount * 0.9
t1 = time.perf_counter()
gen_total = sum(a * 0.9 for a in df["amount"])
t2 = time.perf_counter()
vec_total = (df["amount"] * 0.9).sum()
t3 = time.perf_counter()

print(np.isclose(loop_total, vec_total), np.isclose(gen_total, vec_total))
print(f"loop {t1 - t0:.3f}s, generator {t2 - t1:.3f}s, vectorised {t3 - t2:.4f}s")

All three totals agree. On typical hardware the vectorised version is by far the fastest, often by one or two orders of magnitude, which is why it should be the default for column arithmetic.

Common Mistakes

• Looping over rows for simple column arithmetic.
• Using iterrows() on large frames.
• Writing complex comprehensions that hide logic.
• Measuring speed once instead of repeating timings.

Quick Summary

• Prefer vectorised operations for columns.
• Use comprehensions and built-ins for plain Python data.
• Use itertuples() when a row loop is needed.
• Keep explicit loops for complex logic.

Conclusion

Iteration is everywhere in data science. In 2026, the best practice is to favor Pythonic constructs — direct iteration, enumerate(), zip(), and itertuples() — while preferring vectorized Pandas operations whenever possible for performance. Writing clean iteration code improves readability, reduces bugs, and makes your data science workflows more maintainable.

Next steps:

• Review your current loops and refactor them using more Pythonic patterns with enumerate(), zip(), and itertuples()

Related Tutorials

• Iterating with a for Loop in Python
• List Comprehensions in Python