PyInns - Home

Working with CSV Files in Python: Simplify Data Processing and Analysis – Data Science 2026

CSV (Comma-Separated Values) files remain the most common format for sharing and storing tabular data in data science. Python offers two primary ways to work with them — the built-in csv module for low-level control and pandas.read_csv() for high-level efficiency. Mastering both lets you load, clean, and analyze data quickly while respecting memory limits and data types.

TL;DR — Recommended Approaches

Use pd.read_csv() for most data science tasks
Use csv.DictReader for memory-efficient streaming
Always specify dtypes and chunksize for large files
Combine with list/dict comprehensions for fast post-processing

1. Quick Start with pandas (Most Common)

import pandas as pd

# Basic read with type optimization
df = pd.read_csv("sales_data.csv",
                 dtype={"customer_id": "int32", "amount": "float32"},
                 parse_dates=["order_date"])

print(df.dtypes)
print(f"Memory usage: {df.memory_usage(deep=True).sum() / (1024**2):.2f} MB")

2. Memory-Efficient Streaming with csv Module

import csv

with open("large_sales.csv", "r", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:                    # processes row by row
        amount = float(row["amount"])
        if amount > 1000:
            print(f"High value: {row['customer_id']}")

3. Real-World Data Science Examples

# Example 1: Chunked processing for huge files
chunk_size = 100_000
for chunk in pd.read_csv("10GB_sales.csv", chunksize=chunk_size):
    chunk["profit"] = chunk["amount"] * 0.25
    print(f"Processed {len(chunk):,} rows")

# Example 2: Convert CSV rows to list of dicts (clean & fast)
with open("sales_data.csv", "r", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    records = [row for row in reader]        # list of dicts

# Example 3: Selective column loading
df = pd.read_csv("sales_data.csv", usecols=["customer_id", "amount", "region"])

4. Best Practices in 2026

Always specify dtypes and parse_dates when using pandas
Use chunksize for any file larger than a few GB
Prefer csv.DictReader for pure streaming or very low-memory scenarios
Use usecols to load only needed columns
Save processed results to Parquet for faster future reads

Conclusion

Working with CSV files is a foundational skill in data science. In 2026, combine pandas.read_csv() with smart dtype specification and chunking for most tasks, and fall back to the csv module when you need maximum memory efficiency. These techniques turn raw CSV files into clean, analysis-ready data structures (DataFrames, lists of dicts, or generators) while keeping your pipelines fast and scalable.

Next steps: