Removing missing values

Removing missing values (using dropna() in pandas or drop_nulls() in Polars) is the quickest and cleanest way to prepare data for modeling — but it comes at a cost: every dropped row or column means permanently lost information. In 2026, you only drop when missingness is low, random, and non-critical. Otherwise, you risk introducing bias, reducing statistical power, or throwing away valuable signal.

Here’s a complete, practical guide: when to drop (and when not to), how to do it safely and precisely, what to check before and after, and modern alternatives for speed on large data.

1. Basic & Targeted Dropping in Pandas


import pandas as pd

# Realistic example: customer data with common missing patterns
data = {
    'customer_id': [1001, 1002, 1003, 1004, 1005, 1006],
    'age': [34, None, 45, 28, None, 31],
    'purchase_amount': [120.50, 89.99, None, 210.00, 45.75, None],
    'city': ['New York', 'Chicago', 'Los Angeles', None, 'Seattle', 'Boston'],
    'rating': [8, 7, None, 9, 6, 8]
}
df = pd.DataFrame(data)

print("Original shape:", df.shape)
print("Missing per column:\n", df.isna().sum())

# Option 1: Drop rows with ANY missing values
df_clean_any = df.dropna()
print("\nAfter dropping rows with ANY missing:", df_clean_any.shape)

# Option 2: Drop only if critical columns are missing
critical_cols = ['age', 'purchase_amount']
df_clean_critical = df.dropna(subset=critical_cols)
print("After dropping rows missing in critical columns:", df_clean_critical.shape)

# Option 3: Drop columns with ANY missing values
df_clean_cols = df.dropna(axis=1)
print("After dropping columns with ANY missing:", df_clean_cols.shape)

# Option 4: Drop rows only if ALL values are missing
df_clean_all = df.dropna(how='all')
print("After dropping completely empty rows:", df_clean_all.shape)

**Typical output:**

Original shape: (6, 5)
Missing per column:

customer_id| 0
age| 2
purchase_amount| 2
city| 1
rating| 1

dtype: int64 After dropping rows with ANY missing: (3, 5) After dropping rows missing in critical columns: (4, 5) After dropping columns with ANY missing: (6, 1) After dropping completely empty rows: (6, 5)

2. Fast & Modern: Dropping in Polars (2026 Large-Data Choice)


import polars as pl

df_pl = pl.from_pandas(df)

# Drop rows with any null
df_pl_clean_any = df_pl.drop_nulls()
print("After dropping rows with any null (Polars):", df_pl_clean_any.shape)

# Drop rows only if specific columns are null
df_pl_critical = df_pl.drop_nulls(subset=['age', 'purchase_amount'])
print("After dropping rows missing in critical columns (Polars):", df_pl_critical.shape)

# Drop columns with any null
null_cols = [col for col in df_pl.columns if df_pl[col].null_count() > 0]
df_pl_no_null_cols = df_pl.drop(null_cols)
print("Columns dropped due to nulls:", null_cols)

3. Before & After Visual Check (Always Do This)


import missingno as msno

# Before dropping
plt.figure(figsize=(10, 4))
msno.bar(df, color='teal')
plt.title('Missing Values BEFORE Dropping', fontsize=14)
plt.show()

# After dropping critical columns
plt.figure(figsize=(10, 4))
msno.bar(df_critical_clean, color='darkorange')
plt.title('Missing Values AFTER Dropping Critical Columns', fontsize=14)
plt.show()

When to Drop vs. When to Keep & Impute (2026 Decision Framework)

Scenario	Recommended Action	Why / Risk
<5% total missing, random	Drop rows	Minimal loss, safe
>20–30% missing in a column	Drop column (unless domain-critical)	Unreliable signal
Missingness clustered by group/date	Do NOT drop ? impute or add missing flag	Dropping causes bias
Time-series / sequential data	Prefer forward-fill / interpolation	Dropping breaks continuity
Modeling performance is key	Compare drop vs. impute (KNN/MICE) via CV	Dropping can hurt if missings carry signal

Best Practices & Common Pitfalls

Always print shape before/after — surprise data loss is the #1 gotcha
Use subset= to target only important columns — don’t destroy good data
Visualize with missingno.bar() before and after — confirms what you removed
Pitfall: dropna() without subset ? drops far too much when only one column is bad
Pitfall: dropping rows with missing target ? always clean target first: df.dropna(subset=['target'])
Large data? Use Polars drop_nulls() — 5–20× faster than pandas
Production: log rows/columns dropped — alert if loss > 5–10%

Conclusion

Dropping missing values is fast and simple — but only correct when missingness is low and ignorable. In 2026, always check shape before/after, visualize with missingno, use subset= for precision, and compare drop vs. impute on model performance. Done right, dropping gives you clean, unbiased data without sacrificing too much. Done wrong, it silently destroys your dataset’s power.