Calling Functions in Regular Expressions

Calling Functions in Regular Expressions

One of the most powerful features of Python’s re module is the ability to pass a callable function (instead of a static string) as the replacement argument to re.sub(). The function is automatically called for every match, receives the full Match object, and can return any dynamically computed replacement string. This technique is invaluable in data science for complex text cleaning, conditional transformations, data anonymization, feature engineering, and intelligent log parsing.

TL;DR — Calling Functions in Regex

• re.sub(pattern, repl_function, text)
• repl_function(match) receives a Match object
• Use match.group(0), match.group(1), etc. inside the function
• Perfect for conditional or computed replacements
• Works seamlessly with pandas .str.replace() (via lambda)

1. Basic Function Call in re.sub()

import re

def upper(match):
    return match.group(0).upper()

text = "hello world! python is awesome."
print(re.sub(r"\b\w+\b", upper, text))
# Output: HELLO WORLD! PYTHON IS AWESOME.

2. Real-World Data Science Examples

import pandas as pd

# Example 1: Anonymize emails
def anonymize_email(match):
    return "user@" + match.group(1).split("@")[1]

text = "Contact: alice@example.com or bob@company.com"
print(re.sub(r"(\S+@\S+)", anonymize_email, text))

# Example 2: Convert currency to numeric with a function in .str.replace()
def currency_to_float(match):
    return match.group(1).replace(",", "")

df = pd.DataFrame({"amount_text": ["$1,250.75", "$87.50", "$12.00"]})   # demo data
df["amount"] = df["amount_text"].str.replace(r"\$([\d,]+\.\d{2})",
                                             currency_to_float,
                                             regex=True).astype("float64")
print(df)

3. Advanced Conditional Replacement

def smart_replace(match):
    word = match.group(0)
    if word.isupper():
        return word.lower()
    elif word.islower():
        return word.upper()
    return word

text = "Python is GREAT for DATA Science"
print(re.sub(r"\b\w+\b", smart_replace, text))

4. Best Practices in 2026

• Use a dedicated function or lambda for clarity
• Keep the replacement function pure and fast
• Combine with re.finditer() when you also need match positions
• Use pandas .str.replace(..., regex=True) with lambdas for vectorized calls
• Always test on sample data first — function calls are powerful but can be slower on huge datasets

When to Pass a Function to re.sub

A string replacement is the same for every match. A function replacement is called once per match with the match object, and its return value becomes the replacement. This lets you convert, look up or compute something for each match.

Typical uses include converting units, masking personal data while keeping the last digits, and normalising number formats. The function must return a string; returning a number raises TypeError.

For large DataFrames, pandas .str.replace also accepts a callable when regex=True, so the same logic can run on a column.

Worked Example: Masking Card Numbers

Card numbers are masked except for the last four digits.

import re

text = "Paid with 4111111111111111, refund to 5500000000000004."

def mask(m):
    digits = m.group()
    return "*" * (len(digits) - 4) + digits[-4:]

print(re.sub(r"\b\d{13,16}\b", mask, text))

doubled = re.sub(r"\d+", lambda m: str(int(m.group()) * 2), "3 apples, 10 pears")
print(doubled)

Each card number is replaced by stars and its last four digits. The lambda doubles every number, and str() is needed because re.sub requires a string result.

Practical Tips

Keep replacement functions small and pure: they should depend only on the match object and return a string. This makes them easy to test on their own with re.search results before they are used inside re.sub.

If the function needs extra data, such as an exchange rate or a lookup table, pass it in with a closure or functools.partial rather than using global variables.

For very large texts, measure the speed of a function replacement against a fixed replacement. The function call adds overhead per match, which is usually small but can matter on millions of rows.

Log or count how many replacements were made, using re.subn, which returns both the new text and the number of substitutions.

Common Mistakes

• Returning a number instead of a string.
• Using a function when a fixed string would do.
• Ignoring the match object and recomputing positions.
• Writing heavy logic that slows down large columns.

Quick Summary

• A function computes each replacement.
• It receives the match object.
• It must return a string.
• pandas accepts callables with regex=True.

Conclusion

Calling functions inside re.sub() transforms regular expressions from simple find-and-replace tools into intelligent, programmable text processors. In 2026 data science projects, this pattern is essential for dynamic cleaning, anonymization, conditional formatting, and building advanced feature-extraction pipelines. Combine it with pandas vectorized methods to keep your workflows both powerful and scalable.

Next steps:

• Replace one of your static re.sub() calls with a custom function and see how much more flexible your text processing becomes

Related Tutorials

• Substitution in Regular Expressions
• Replacing Substrings in Python