Detecting Missing Values in Pandas – Best Techniques 2026

Detecting Missing Values in Pandas – Best Techniques 2026

Before you can handle missing values, you must first detect and understand them properly. In 2026, Pandas offers several powerful and efficient methods to identify, quantify, and visualize missing data in your datasets.

TL;DR — Essential Detection Commands

• df.isna().sum() – Count missing values per column
• df.isna().mean() * 100 – Percentage of missing values
• df.isnull().any() – Which columns have missing values
• df[df.isna().any(axis=1)] – Show rows with any missing values

1. Basic Missing Value Detection

import pandas as pd

# Create a small demo file so this example runs as-is
# (in your project, sales_data.csv is your real export)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("sales_data.csv", index=False)

df = pd.read_csv("sales_data.csv", parse_dates=["order_date"])

# Total missing values per column
missing_count = df.isna().sum()
print("Missing values per column:")
print(missing_count)

# Percentage of missing values
missing_pct = df.isna().mean() * 100
print("\nPercentage missing:")
print(missing_pct.round(2))

2. Advanced Detection Techniques

# Columns that have at least one missing value
cols_with_missing = df.columns[df.isna().any()]
print("Columns with missing values:", cols_with_missing.tolist())

# Rows that contain missing values
rows_with_missing = df[df.isna().any(axis=1)]
print(f"\nNumber of rows with missing values: {len(rows_with_missing)}")

# Detailed view of missingness pattern
missing_pattern = df.isna().sum(axis=1).value_counts().sort_index()
print("\nNumber of missing values per row:")
print(missing_pattern)

3. Visualizing Missing Values (Very Useful)

import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(12, 8))

# Heatmap of missing values
sns.heatmap(df.isna(), cbar=False, cmap="viridis", yticklabels=False)
plt.title("Missing Values Heatmap")
plt.show()

# Bar plot of missing values percentage
missing_pct.sort_values(ascending=False).plot(
    kind="bar",
    figsize=(12, 6),
    color="tomato"
)
plt.title("Percentage of Missing Values by Column")
plt.ylabel("Percentage (%)")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()

4. Best Practices for Detecting Missing Values in 2026

• Always start with df.isna().sum() and df.isna().mean()*100
• Check both column-wise and row-wise missingness patterns
• Use visualization (heatmap or bar plot) to understand the pattern of missing data
• Investigate why the data is missing — this guides your handling strategy
• Document the percentage and pattern of missing values in your analysis notes

How pandas Detects Missing Data

isna() returns a boolean DataFrame of the same shape, with True wherever a value is NaN, None or NaT. notna() is its opposite. Both treat empty strings and placeholder text such as "N/A" as real values, so those need converting first.

Placeholders can be turned into missing values while reading with na_values=, or afterwards with replace(). Once they are real NaN values, all of pandas' missing-data tools work on them.

Worked Example: Hidden Missing Values

A table with placeholder strings is checked before and after conversion.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name": ["Ana", "", "Cara", "Dev"],
    "age": ["34", "N/A", "29", "-"],
})
print(df.isna().sum().tolist())

clean = df.replace({"": np.nan, "N/A": np.nan, "-": np.nan})
clean["age"] = pd.to_numeric(clean["age"])
print(clean.isna().sum().tolist())
print(clean[clean["age"].notna()])

Before conversion pandas reports no missing values at all, which is misleading. After replacing the placeholders, one name and two ages are correctly counted as missing.

Practical Tips

A good habit is to define a single list of placeholder markers for your project and pass it to every read_csv call with na_values=. That keeps missing-data handling consistent across files and team members.

Common Mistakes

• Assuming empty strings are counted as missing.
• Comparing with == np.nan, which is always False.
• Converting to numbers before cleaning placeholders.
• Forgetting na_values= when reading files.

Quick Summary

• isna() and notna() return boolean masks.
• Placeholders must be converted to NaN first.
• Use na_values= in read_csv for common markers.
• Never test missingness with ==.

Conclusion

Properly detecting missing values is the first and most important step in data cleaning. In 2026, combining isna().sum(), percentage calculations, and visualizations gives you a complete picture of missingness in your dataset. Never jump straight to filling or dropping missing values without first understanding their pattern and potential causes.

Next steps:

• Run a full missing value analysis on one of your current datasets using the techniques shown above

Related Tutorials

• Detecting Any Missing Values with .isna().any() in Pandas
• Counting Missing Values in Pandas – Best Techniques 2026