Dropping Duplicate Names & Rows in Pandas

Dropping Duplicate Names & Rows in Pandas

Handling duplicate data is a critical step in any data manipulation pipeline. In 2026, Pandas provides powerful and flexible tools to detect and remove duplicates based on single columns, multiple columns, or entire rows — especially useful when working with customer names, product names, or any identifier data.

TL;DR — Key Methods

• df.drop_duplicates() – Remove duplicate rows
• subset parameter – Drop duplicates based on specific columns (e.g., names)
• keep='first' / keep='last' / keep=False – Control which duplicate to keep
• duplicated() – Identify duplicates without removing them

1. Basic Duplicate Removal

import pandas as pd

# Small demo file so this example runs as-is
pd.DataFrame({
    "name": ["Ana", "Ben", "Ana", "Chen", "Ben", "Dana"],
    "email": ["ana@x.com", "ben@x.com", "ana@x.com", "chen@x.com", "ben@y.com", "dana@x.com"],
    "phone_number": ["111", "222", "111", "333", "222", "444"],
}).to_csv("customers.csv", index=False)

df = pd.read_csv("customers.csv")

# Remove exact duplicate rows
df_clean = df.drop_duplicates()

# Remove duplicates based on name only
df_clean = df.drop_duplicates(subset=["name"], keep="first")

print(f"Original rows: {len(df)}")
print(f"After removing duplicate names: {len(df_clean)}")

2. Dropping Duplicates on Multiple Columns (Most Common Real-World Use)

# Remove duplicates based on name + email combination
df_clean = df.drop_duplicates(
    subset=["name", "email"], 
    keep="last"                  # keep the most recent record
)

# Remove duplicates based on name + phone
df_clean = df.drop_duplicates(
    subset=["name", "phone_number"],
    keep=False                   # remove all duplicates
)

3. Identifying Duplicates First (Recommended for Safety)

# Find duplicate names
duplicates = df[df.duplicated(subset=["name"], keep=False)]

print("Duplicate names found:")
print(duplicates[["name", "email", "phone_number"]].sort_values("name"))

# Then safely remove them
df_clean = df.drop_duplicates(subset=["name"], keep="first")

4. Best Practices in 2026

• Always use duplicated() first to inspect duplicates before dropping them
• Use subset when duplicates are based on specific columns like names, emails, or IDs
• Choose keep="first" or keep="last" based on your business logic (e.g., keep latest record)
• Combine with sort_values() before dropping to keep the most recent data
• After dropping duplicates, reset the index with .reset_index(drop=True)

What Counts as a Duplicate

A duplicate is not always a fully identical row. In a customer list the same person can appear twice with different signup dates or slightly different spelling, so you must decide which columns define identity. drop_duplicates(subset=[...]) lets you choose those columns, and keep decides which copy survives: "first", "last" or False to remove every copy.

Text values often need cleaning first. "Ali Khan" and "ali khan " are different strings, so normalise case and whitespace before checking. Use df.duplicated(subset=...).sum() to count repeated rows before removing them; it is a cheap way to confirm that your rule matches what you expect.

Worked Example: Keep the Latest Record per Customer

Sorting before dropping lets you choose which version of a record to keep.

import pandas as pd

people = pd.DataFrame({
    "name": ["Ali Khan", "ali khan ", "Sara Ahmed", "Sara Ahmed"],
    "city": ["Lahore", "Karachi", "Multan", "Multan"],
    "updated": pd.to_datetime(["2026-01-05", "2026-03-01", "2026-02-10", "2026-02-10"]),
})

people["name"] = people["name"].str.strip().str.title()
people["name_key"] = people["name"].str.lower()
latest = (people.sort_values("updated")
                .drop_duplicates(subset="name_key", keep="last")
                .drop(columns="name_key"))
print(latest)

Ali Khan is kept once, with his newer Karachi address and a cleaned spelling, and the exact duplicate of Sara Ahmed is removed.

Common Mistakes

• Dropping before cleaning text: case and spaces hide real duplicates
• Keeping an arbitrary copy: sort first so keep="last" means "newest"
• Matching on name alone: two different people can share a name; add another column such as email
• Not checking the count: compare len() before and after to see what was removed
• Forgetting the result is a copy: assign it back or chain the next step; inplace=True is discouraged in modern pandas

Quick Summary

• Count with duplicated().sum() first
• Decide which columns define identity and pass them as subset
• Normalise text before comparing
• Sort, then choose keep deliberately

Conclusion

Properly handling duplicate names and rows is essential for data quality and accuracy. In 2026, using drop_duplicates() with the subset parameter and proper keep strategy allows you to clean your data efficiently while maintaining important business context. Always inspect duplicates first before removing them.

Next steps:

• Run df.duplicated(subset=["name"]) on your next dataset to see how many duplicate names exist, then clean them using the best strategy for your use case

Related Tutorials

• Dropping Duplicate Pairs in Pandas – Handling Duplicate Combinations 2026
• Creating DataFrames with Pandas in Python 2026
• Summaries by Group in Pandas – GroupBy & Aggregation Best Practices 2026