Handling Missing Values in Pandas

Handling Missing Values in Pandas

Missing values (NaN) are very common in real-world datasets. Knowing how to detect, analyze, and handle them properly is a fundamental skill in data manipulation. In 2026, Pandas provides several powerful methods to deal with missing data effectively.

TL;DR — Key Strategies

• df.isna().sum() – Count missing values
• df.dropna() – Remove rows/columns with missing values
• df.fillna() – Fill missing values with specific values
• Smart imputation based on context

1. Detecting Missing Values

import pandas as pd

# Create a small demo file so this example runs as-is
# (in your project, sales_data.csv is your real export)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("sales_data.csv", index=False)

df = pd.read_csv("sales_data.csv", parse_dates=["order_date"])

# Count missing values per column
missing = df.isna().sum()
print(missing)

# Percentage of missing values
missing_pct = df.isna().mean() * 100
print(missing_pct.round(2))

2. Removing Missing Values

# Drop rows with any missing values
df_clean = df.dropna()

# Drop rows only if all values are missing
df_clean = df.dropna(how="all")

# Drop columns with more than 30% missing values
threshold = len(df) * 0.7
df_clean = df.dropna(thresh=threshold, axis=1)

3. Filling Missing Values (Imputation)

# Fill with constant
df["amount"] = df["amount"].fillna(0)

# Fill with mean/median (column-wise)
df["amount"] = df["amount"].fillna(df["amount"].mean())

# Forward fill (useful for time series)
df["amount"] = df["amount"].ffill()

# Fill with group mean (smart imputation)
df["amount"] = df.groupby("region")["amount"].transform(lambda x: x.fillna(x.mean()))

4. Best Practices in 2026

• First, understand why the data is missing (MCAR, MAR, MNAR)
• Document your imputation strategy
• Use fillna(0) for counts and amounts when missing means "zero"
• Use group-wise imputation (groupby().transform()) when missingness depends on categories
• Consider using advanced imputation libraries like sklearn.impute for complex cases
• Always check the impact of your handling strategy on downstream analysis

Strategies for Missing Values

Missing values appear as NaN, None or NaT in pandas. Before handling them, find out how many there are and why they are missing, because the reason decides the right fix.

The main options are to drop rows or columns, to fill with a constant or statistic, or to fill from neighbouring values in ordered data. Whatever you choose, record it, because it changes the results of every later step.

Worked Example: Drop, Fill and Forward-Fill

One small table is handled in three different ways for comparison.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "day": pd.date_range("2026-01-01", periods=5, freq="D"),
    "temp": [5.0, np.nan, 7.0, np.nan, 6.0],
    "city": ["Oslo", "Oslo", None, "Oslo", "Oslo"],
})
print(df.isna().sum())

print(len(df.dropna()))
filled = df.fillna({"temp": df["temp"].mean(), "city": "Unknown"})
print(filled["temp"].tolist())
print(df["temp"].ffill().tolist())

Dropping keeps only two complete rows, which loses most of the data. Mean filling keeps all rows but flattens variation, while forward filling suits ordered readings.

Practical Tips

A practical order of work is: measure missingness, look for patterns, decide per column, apply the fix, and then measure again to confirm that no gaps remain where you did not expect them. Keep the original data untouched so you can try a different strategy later.

Common Mistakes

• Dropping rows without checking how much data is lost.
• Filling every column with the same value.
• Using fillna(method=...) which pandas 3 removed.
• Hiding the share of missing data from readers.

Quick Summary

• Measure missingness before acting.
• Choose drop, fill or ffill based on the cause.
• Use a dict to fill columns differently.
• Use ffill() and bfill() for ordered data.

Conclusion

Handling missing values correctly is crucial for accurate data analysis. In 2026, the best approach is to first explore the pattern of missingness, then choose the most appropriate method — whether removing, filling with constants, or using smart group-based imputation. Always document your decisions so others can understand and reproduce your work.

Next steps:

• Analyze the missing values in one of your datasets using isna().sum() and decide on the best handling strategy for each column

Related Tutorials

• Detecting Missing Values in Pandas – Best Techniques 2026
• Removing Missing Values in Pandas – When and How to Use dropna() 2026