DataFrame Manipulation in Pandas – Essential Techniques 2026

DataFrame Manipulation in Pandas – Essential Techniques 2026

DataFrame manipulation is at the core of data analysis in Python. In 2026, mastering key Pandas operations such as filtering, selecting, transforming, and reshaping data allows you to work efficiently and write clean, professional code.

TL;DR — Core DataFrame Manipulation Techniques

• Filtering rows with conditions
• Selecting and reordering columns
• Creating and transforming columns
• Sorting and grouping data
• Handling missing values and duplicates

1. Filtering Rows

import pandas as pd

# Create a small demo file so this example runs as-is
# (in your project, sales_data.csv is your real export)
import numpy as np
rng = np.random.default_rng(42)
n = 60
pd.DataFrame({
    "order_date": pd.date_range("2026-01-01", periods=n, freq="3D"),
    "customer_id": rng.integers(101, 121, n),
    "customer_name": rng.choice([" ana", "ben ", "chen", "dana"], n),
    "product_id": rng.integers(1, 9, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "category": rng.choice(["Electronics", "Computers", "Books", "Home"], n),
    "payment_method": rng.choice(["card", "cash", "wallet"], n),
    "segment": rng.choice(["consumer", "business"], n),
    "quantity": rng.integers(1, 6, n),
    "amount": rng.uniform(50, 2500, n).round(2),
    "discount": rng.choice([0.0, 0.05, 0.1, 0.2], n),
}).assign(
    price=lambda d: (d["amount"] / d["quantity"]).round(2),
    profit=lambda d: (d["amount"] * rng.uniform(0.05, 0.3, n)).round(2),
).assign(
    ship_date=lambda d: d["order_date"] + pd.Timedelta(days=2),
    status=lambda d: np.where(d["amount"] > 2000, "priority", "standard"),
).to_csv("sales_data.csv", index=False)

df = pd.read_csv("sales_data.csv", parse_dates=["order_date"])

# Filter using boolean conditions
high_value = df[df["amount"] > 1000]

# Multiple conditions with & (and), | (or)
north_high = df[(df["region"] == "North") & (df["amount"] > 1500)]

# Using query() - often more readable
premium_sales = df.query("amount > 2000 and region == 'North'")

2. Selecting and Reordering Columns

# Select specific columns
df_subset = df[["order_date", "customer_id", "amount", "region", "category"]]

# Reorder columns
df_reordered = df[["customer_id", "order_date", "amount", "region", "category", "quantity"]]

# Select columns by pattern
amount_cols = df.filter(like="amount")

3. Creating and Transforming Columns

# Using assign() - recommended for chaining
df = df.assign(
    year = df["order_date"].dt.year,
    month = df["order_date"].dt.month_name(),
    profit = df["amount"] * 0.25,
    order_value_category = pd.cut(df["amount"], bins=[0, 500, 1500, float("inf")], 
                                  labels=["Small", "Medium", "Large"])
)

# Using lambda with apply (for complex logic)
df["is_high_value"] = df["amount"].apply(lambda x: "Yes" if x > 1000 else "No")

4. Best Practices in 2026

• Use method chaining with parentheses for readable pipelines
• Prefer .assign() over direct assignment when creating new columns
• Use .query() for complex filtering conditions
• Optimize dtypes early (int32, float32, category) to save memory
• Document your data manipulation steps clearly

Core Manipulation Steps

Most DataFrame work combines a few operations: selecting columns, filtering rows, creating new columns, sorting, and grouping. Chaining them with methods such as assign, query and sort_values keeps the steps in a readable pipeline.

Method chains avoid temporary variables and make each step easy to comment out while testing. Wrap long chains in parentheses so each method sits on its own line.

Worked Example: A Readable Pipeline

Orders are enriched, filtered, grouped and sorted in one chain.

import pandas as pd

orders = pd.DataFrame({
    "region": ["North", "South", "North", "East", "South"],
    "units": [3, 5, 2, 7, 1],
    "price": [10.0, 8.0, 12.0, 9.0, 15.0],
})

summary = (
    orders
    .assign(revenue=lambda d: d["units"] * d["price"])
    .query("units >= 2")
    .groupby("region", as_index=False)["revenue"].sum()
    .sort_values("revenue", ascending=False)
)
print(summary)

The South order with one unit is filtered out, leaving East with the highest revenue at 63.0. Each line performs one clear step, which makes the logic easy to review.

Practical Tips

When a chain gets hard to debug, insert a temporary .pipe(lambda d: print(d.shape) or d) step to print the shape at that point, then remove it once everything works as expected.

Common Mistakes

• Modifying data in place in the middle of a chain.
• Writing very long chains with no intermediate checks.
• Forgetting as_index=False when a flat table is wanted.
• Using query strings with column names that contain spaces.

Quick Summary

• Select, filter, create, sort and group are the core steps.
• assign and query keep chains readable.
• Wrap chains in parentheses.
• Check intermediate results while developing.

Conclusion

Effective DataFrame manipulation is the foundation of successful data analysis. In 2026, the most productive Pandas users combine filtering, column selection, transformation, and grouping using clean, readable patterns — especially method chaining and .assign(). Mastering these techniques allows you to transform raw data into valuable insights efficiently and professionally.

Next steps:

• Take one of your raw datasets and build a clean manipulation pipeline using filtering, column creation with .assign(), and grouping

Related Tutorials

• Creating DataFrames from List of Dictionaries (Row-oriented) in Pandas 2026
• Handling Missing Values in Pandas