Should you strip personal data before it reaches the model?
Selectively, per field, not as a blanket policy. Redacting everything is easy to sell and breaks any task where the personal detail is the point. Decide by whether the model needs to understand the value or merely carry it, and measure both what leaks and what you destroyed.
Engineering only. What follows is about what a system can technically do and what it costs in accuracy, not about what any organisation is required to do. Obligations are a question for your counsel, and they should see the data flow described below rather than a summary of it.
The instinct when sending customer data to a model API is to strip everything personal first, and there is a product category ready to sell you exactly that. It is easy to buy, easy to put in a security review, and it quietly breaks a class of tasks in a way nobody notices until accuracy is already unexplainable.
The problem is simple to state and easy to miss. Sometimes the personal data is the task.
Where blanket redaction destroys the work
Consider a few real requests. Check whether this email address is valid and matches the account. Personalise this reply using the customer's name. Decide whether these two records refer to the same person. Extract the beneficiary from this instruction.
In each of those, the value you want to remove is the input the task operates on. Replace it with a placeholder and the model produces confident nonsense, because you have asked it to reason about something it can no longer see. Worse, it fails silently: the response is still fluent, and the failure only surfaces downstream.
The useful distinction is whether the model needs to understand the value or merely carry it through.
| The model needs to | Treatment | Example |
|---|---|---|
| Carry it through untouched | Tokenise. Substitute a stable placeholder, restore on the way out. | A name that appears in a summary you are generating |
| Reason about its content | Send it, and control the boundary instead. | Validating a format, matching two records, extracting a specific party |
| Never see it at all | Drop it before the call. | A national identifier in a document you are summarising for topic |
Most fields fall in the first and third rows, which is why redaction is usually right. The second row is small and is where blanket policies do their damage, because it is invisible in a policy document and obvious only in the accuracy numbers.
Decide per field, in writing, once
The work is a table, not a tool. Go through the data classes your feature actually touches and record three things per class: does the model need to understand it, what happens to it before the call, and who decided. Do it once, review it when the feature changes, and hold it somewhere a security review can read.
Two practical notes from doing this repeatedly. Teams consistently over-scope what the model needs, because sending everything is easier than deciding, so the first pass usually removes a surprising amount for free. And the field people argue hardest about is usually free text, where the answer is that it contains all the classes at once and needs detection rather than a rule.
Free text is the hard case
Structured fields are easy: you know a column holds an account number and can decide once. The difficulty is free text, where a support message or a case note contains every class at once, in no predictable position, mixed into the content the task actually needs.
Rules do not work here, so this needs detection, and detection has a recall problem worth being honest about. A detector tuned to catch everything will also flag ordinary words that happen to look like names, and one tuned to avoid that will miss the unusual name, the international format, the identifier written with spaces in it. There is no setting that avoids both, which is the argument for measuring rather than trusting a vendor default.
Measuring it, which almost nobody does
Redaction has two error rates and they pull against each other, so a single accuracy number for a redaction system tells you nothing.
- What it missed. A personal detail that reached the provider. Each miss is the thing the control exists to prevent, so this is measured as a count of incidents rather than a percentage.
- What it destroyed. Text removed that was not personal at all, or was personal and needed. This is invisible in security terms and shows up as the feature getting worse.
Build a small labelled set the same way you would for extraction accuracy: a few hundred real examples with the personal spans marked by a person. Then measure both rates, and tune knowing what each step costs the other. Detectors are aggressive by default, and the over-redaction rate is generally worse than teams assume because nobody is looking for it.
Where this belongs in the system
Redaction has to happen at a boundary you control, before anything leaves, which in practice means the same gateway that handles spend attribution and provider routing. Doing it in application code means it is done in some paths and not others, and the paths that skipped it are the ones you will be asked about.
Two adjacent things worth doing at the same boundary. Redact before logging as well as before sending, because your observability platform sees everything the model saw, a point we make in what to log. And record what was redacted, as counts by class rather than values, so you can answer what the system removed last month without holding what it removed.
The short version
- Do not adopt a blanket policy. It is the expensive option disguised as the safe one.
- Classify per field by whether the model must understand the value or only carry it.
- Tokenise what passes through, drop what is never needed, send what the task genuinely operates on.
- Measure misses and over-redaction separately, on a labelled set.
- Enforce it at one gateway, and redact before logging too.
If you want the per field decision made against your actual data flow, our free AI audit maps what currently leaves your systems, and how we handle data governance covers building the boundary.
Related reading
Get this applied to your business.
The free AI audit measures your live setup and shows where AI would actually pay off.