Substitution in Regular Expressions

Substitution in Regular Expressions

Substitution is the most powerful part of regular expressions in Python. It lets you find patterns with re.search() or re.findall() and then automatically replace them with new text — or even with dynamically computed values. In data science, substitution is used daily for cleaning messy logs, anonymizing data, standardizing formats, fixing typos, and transforming raw text into structured features.

TL;DR — Core Substitution Tools

• re.sub(pattern, repl, text) → replace all matches
• re.subn(pattern, repl, text) → replace + return count
• Backreferences: \1, \g<1>
• Callable replacement function (dynamic logic)
• pandas .str.replace(..., regex=True) for vectorized substitution

1. Basic Substitution

import re

text = "Order ORD-12345 placed for $1,250.75 on 2026-03-19"

# Simple string replacement
clean = re.sub(r"ORD-\d+", "ORDER_ID", text)
print(clean)

# Backreference substitution
clean = re.sub(r"(\d{4}-\d{2}-\d{2})", r"DATE:\1", text)
print(clean)

2. Real-World Data Science Examples with Pandas

import pandas as pd

# Demo file so this example runs as-is (in your project this is your real log export)
pd.DataFrame({"log": [
    "2026-03-19 ERROR: Payment failed for ORD-12345 amount $1,250.75 Qty: 3 user alice@example.com!!! https://shop.example.com/o/12345 call +1 555-123-4567",
    "2026-03-19 WARNING: Slow response for ORD-98765 amount $87.50 Qty: 1 bob@company.com on 03/19/2026 aaaa",
    "INFO: user logged in ORD-555 amount $12.00 Qty: 2 carol@data.org <b>ok</b>",
]}).to_csv("logs.csv", index=False)

# Demo file so this example runs as-is (in your project this is your real log export)
pd.DataFrame({"log": [
    "2026-03-19 ERROR: Payment failed for ORD-12345 amount $1,250.75 Qty: 3 user alice@example.com!!! https://shop.example.com/o/12345 call +1 555-123-4567",
    "2026-03-19 WARNING: Slow response for ORD-98765 amount $87.50 Qty: 1 bob@company.com on 03/19/2026 aaaa",
    "INFO: user logged in ORD-555 amount $12.00 Qty: 2 carol@data.org <b>ok</b>",
]}).to_csv("logs.csv", index=False)

df = pd.read_csv("logs.csv")

# Example 1: Standardize currency format
df["amount"] = df["log"].str.extract(r"(\$[\d,]+\.\d{2})")[0].str.replace(
    r"\$([\d,]+\.\d{2})", r"USD \1", regex=True)

# Example 2: Anonymize emails
df["log"] = df["log"].str.replace(r"(\S+@\S+)", "[EMAIL]", regex=True)

# Example 3: Convert MM/DD/YYYY dates to ISO format
df["log"] = df["log"].str.replace(r"(\d{2})/(\d{2})/(\d{4})", r"\3-\1-\2", regex=True)
print(df["amount"].tolist())
print(df["log"][1])

3. Advanced Substitution with Backreferences & Functions

# Backreferences with groups
text = "Contact: Alice (alice@example.com)"
print(re.sub(r"(\w+) \((\S+@\S+)\)", r"\1 <\2>", text))

# Callable function (dynamic substitution)
def mask_number(match):
    return "*" * len(match.group(0))

print(re.sub(r"\d+", mask_number, "Order ID 98765, amount 1250.75"))

4. Best Practices in 2026

• Use raw strings r"..." for every regex pattern
• Prefer backreferences \1 or \g<1> for simple reordering
• Use a callable function when logic is complex
• Use re.subn() when you need to know how many changes were made
• Always apply substitution with pandas .str.replace(regex=True) on DataFrames for speed

Backreferences in the Replacement String

In re.sub, the replacement can refer to captured groups: \1 or \g<1> inserts group 1, and \g inserts a named group. \g<1> is needed when a digit follows the reference, because \10 would mean group 10.

Replacement strings should be raw strings too, for the same reason as patterns. To insert a literal backslash, write two of them in a raw string.

Named groups make substitutions easier to read, especially when reordering several parts such as date components.

Worked Example: Reformatting Names and Dates

Groups are reordered and reused in the replacement.

import re

names = "Khan, Bilal; Silva, Ana"
print(re.sub(r"(\w+), (\w+)", r"\2 \1", names))

dates = "Due 19/03/2026 and 05/04/2026"
iso = re.sub(r"(?P<d>\d{2})/(?P<m>\d{2})/(?P<y>\d{4})", r"\g<y>-\g<m>-\g<d>", dates)
print(iso)

print(re.sub(r"(\d)", r"\g<1>0", "Room 5"))

Surnames and first names are swapped, and named groups turn day-first dates into ISO dates. The last line uses \g<1> followed by a zero, which \10 could not express.

Practical Tips

Test substitutions on a small sample and print before and after values side by side to catch wrong group numbers.

Common Mistakes

• Using \10 when group 1 followed by 0 is meant.
• Writing replacements without a raw string.
• Referring to a group that does not exist.
• Reordering dates with regex when parsing would be safer.

Quick Summary

• \1 and \g insert groups.
• \g<1> avoids ambiguity with digits.
• Use raw strings for replacements.
• Named groups make swaps readable.

Conclusion

Substitution turns regular expressions from simple search tools into powerful data transformation engines. In 2026 data science projects, mastering re.sub(), backreferences, callable replacements, and pandas vectorized substitution is essential for cleaning, anonymizing, standardizing, and feature-engineering text at scale. These techniques complete the regex workflow and prepare your data for analysis and modeling.

Next steps:

• Take one of your current text-cleaning scripts and replace manual string operations with re.sub() for cleaner, more powerful results

Related Tutorials

• Calling Functions in Regular Expressions
• Grouping and Capturing in re Module