Guides › Redaction, masking, tokenization, substitution

PII Redaction vs Masking vs Tokenization vs Substitution for LLMs

Q: Does masking make data anonymous?

No. Masking hides part of a value for display. Partial values combine with other fields, and a model performs that combination readily.

Q: Can we reuse a payments tokenization system for AI context?

Partly, for structured identifiers. Payment tokenization is built for field-level references and all-or-nothing vault access. AI context needs consistent stand-ins across free text and restoration scoped to the reader.

Hunter Moore, Founder and CTO · Updated October 15, 2026

Four techniques get used, and confused, when the goal is to keep personal and sensitive identifiers out of a model's context. Redaction removes them. Masking hides most of each one. Tokenization swaps each for a reference held in a vault. Substitution replaces each with a consistent, realistic stand-in that can be restored for an entitled reader. They differ in what the model can still reason about, whether anyone can get the real value back, how they fail under re-identification, and which regulatory definitions they satisfy. This page compares them on those four axes and explains why, for retrieval over sensitive data sent to a third-party model, substitution is the one that keeps the answer useful.

At a glance

Redaction removes the value. Irreversible, strongest, and it degrades every answer that needed to tell records apart.
Masking hides part of the value. Cosmetic; partial values combine, and a model is good at combining them.
Tokenization swaps the value for a vault reference. Reversible by whoever has vault access; built for fields, not free text; tokens read as holes to a model.
Substitution swaps the value for a consistent, realistic stand-in and restores it only for an entitled reader. The model reasons normally, the provider never holds the identifier, and restoration is scoped and logged. GDPR pseudonymization, designed for AI context.
All four share two failures. Detection recall on free text is never complete, and quasi-identifiers re-identify people regardless of technique, which is why substitution is paired with retrieval scoped to the asker.

The four techniques

Redaction

Redaction deletes the identifier from the text, usually leaving a placeholder such as [NAME] or a black bar. It is irreversible by design, which is its virtue for publication and its problem for a model: an assistant asked to summarize a case file in which every name is [NAME] cannot tell three people apart, cannot answer which account the question is about, and cannot follow a thread across documents. Redaction is the right tool when the information is never needed again. In a retrieval system it is a decision to degrade every answer that touches the redacted field.

Masking

Masking hides part of the value and shows the rest: a card number as four digits, a name as initials, a date as a year. It preserves format and a little utility, and it is usually irreversible. Its weakness for AI use is that partial values combine: initials plus a department plus a start date identify a person, and a model is good at exactly that kind of join. Masking is a display convention more than a protection, and the regulations treat it that way.

Tokenization

Tokenization replaces each value with a token and keeps the mapping in a vault; the token is meaningless outside the vault and the value is recoverable by anyone with vault access. It is the standard technique in payments, where it lets systems handle card references without handling card numbers, and it is often format-preserving so that downstream systems keep working. For free text and model context the limitations are that tokens generated per field are not consistent across documents unless the vault is designed for it, that restoration is typically all-or-nothing rather than scoped to who is asking, and that a token such as TOK_84A1 in a sentence reads as a hole to a model, which handles it less well than a plausible value.

Substitution

Substitution replaces each real identifier with a consistent, realistic stand-in: the same person gets the same stand-in name across every document and every query, an account number becomes a well-formed account number that maps back only inside your boundary, a date shifts by a fixed offset so intervals survive. The model reasons over a coherent record and answers naturally. The mapping is held inside your environment, and the real values are restored in the application only for a reader who holds the entitlement, with every restoration logged. In the vocabulary of GDPR this is pseudonymization with the additional information held separately; what makes it fit for AI use is the consistency and the scoped restoration, which neither redaction nor conventional tokenization provides.

Technique	What the model sees	Reversible?	Answer quality	Where it fails	Regulatory fit
Redaction	A placeholder or gap where the identifier was.	No.	Degraded: records become indistinguishable; references break.	Quasi-identifiers remain; detection misses leave real values in place.	Supports HIPAA Safe Harbor when all 18 types are removed; rarely GDPR anonymization on its own.
Masking	A partial value: last four digits, initials, a year.	Usually no.	Partial; enough for display, not for reasoning across records.	Partial values combine into identity; a model performs the join.	Display control; not de-identification under HIPAA or anonymization under GDPR.
Tokenization	An opaque token such as TOK_84A1.	Yes, for anyone with vault access.	Reduced: tokens are holes in prose; consistency across documents depends on vault design.	All-or-nothing restoration; vault is the crown jewel; per-field design doesn't cover free text.	PCI DSS pattern; GDPR pseudonymization when the vault is held separately.
Substitution	A realistic, consistent stand-in: same person, same stand-in, everywhere; well-formed numbers; shifted dates.	Yes, only for entitled readers, logged.	Preserved: coherent record, natural answer.	Quasi-identifiers in retained context; needs scoped retrieval alongside it; mapping must stay inside the boundary.	GDPR Art. 4(5) pseudonymization; can support HIPAA Safe Harbor or Expert Determination; NIST SP 800-188 practices for quasi-identifiers apply.

Where each one fails

All four depend on finding the identifiers first, and detection is the step most deployments underestimate. Structured fields are easy. Free text, which is most of what a knowledge base holds, requires entity recognition, and recall on names, addresses and identifiers embedded in prose is never complete. A technique applied to 94 percent of identifiers protects none of the people in the other six. The detection layer has to be measured, and the measurement has to be on your corpus rather than a benchmark.

The second failure is shared by every technique and is the one models make worse: quasi-identifiers. Removing or replacing the name does nothing if the paragraph still says a 54-year-old cardiologist in a town of 3,000 who joined in March. A model asked who the record is about will answer from the context, and often correctly. This is the re-identification problem that NIST SP 800-188 and the GDPR's definition of anonymization are built around, and it is why the reference architecture pairs substitution with scoped retrieval rather than relying on either alone: substitution limits what the provider holds, and entitlement filtering limits which records a given user can assemble.

The third failure is specific to the reversible techniques. A vault or a mapping is the most sensitive store in the system, because it turns every stand-in back into a person. Tokenization's conventional model gives vault access by system role; a user of the assistant who can see restored values can see all of them. Substitution designed for AI use scopes restoration to the asking user's entitlement and logs the attempts, so that repeated denied restorations become a detection signal rather than a silent failure.

What the regulations call these things

The GDPR defines pseudonymization in Article 4(5) as processing such that the data can no longer be attributed to a specific person without additional information kept separately under technical and organizational measures. Substitution and tokenization meet that definition when the mapping is held apart; pseudonymized data remains personal data, with reduced risk, which Recital 28 and the Article 32 security obligations recognize. Anonymization, under Recital 26, is the stronger state in which re-identification is not reasonably likely by any means; redaction and masking of direct identifiers alone rarely reach it because of quasi-identifiers. HIPAA offers two de-identification routes under 45 CFR 164.514: Safe Harbor, which removes eighteen listed identifier types, and Expert Determination, which certifies a low re-identification risk; a substitution scheme that shifts dates and replaces the eighteen types can support either, and the minimum necessary standard in 164.502(b) applies to what is sent regardless. NIST SP 800-188, De-Identifying Government Datasets, is the practical reference on quasi-identifiers and re-identification testing. PCI DSS is where tokenization's vault model comes from, and its requirements for the vault are a reasonable floor for a substitution mapping.

The test that matters

After applying any of the four techniques to a sample of real records, hand the result to the model you will use and ask it who each record is about and which records are about the same person. A model that answers correctly from quasi-identifiers has told you the technique is not enough on its own. Run the test per data category, on your corpus, and keep the result with the leakage number from the self-test.

Choosing by use case

The choice follows from two questions: does anyone need the real value later, and does the model need to tell records apart. If the answer to both is no, redaction is simplest and strongest. If the output is for display and the real value is unneeded, masking is adequate for low-sensitivity fields and should be treated as cosmetic for anything regulated. If a payment or identifier reference must flow through systems and be recovered by a service, tokenization with a vault is the established pattern. If a model is going to retrieve, reason and answer over records that identify people or accounts, and some readers of those answers are entitled to the real values while others are not, substitution with scoped restoration is the only one of the four that serves the use case without degrading it. The table puts the common cases against the techniques.

Use case	Real value needed later?	Model must tell records apart?	Technique
Publishing a dataset or report	No	No	Redaction, with quasi-identifier review
Showing a reference on a screen or receipt	No	No	Masking
Passing a payment or account reference between systems	Yes, by a service	No	Tokenization with a vault
Assistant answering questions over customer, patient or employee records via a third-party model	Yes, by entitled readers	Yes	Substitution with scoped restoration, plus entitlement-filtered retrieval
Fine-tuning or training on records	No	Sometimes	Substitution at dataset level, then the controls in the training data leakage guide
Self-hosted model over internal records	Yes	Yes	Optional on the provider question; still useful so logs and caches hold stand-ins

Where substitution sits in the retrieval path

Substitution belongs on the boundary between your environment and the model provider, after retrieval and entitlement filtering and before the context is sent. Applied at ingestion instead, it protects the index copy too, at the cost of running the mapping over every chunk; applied only at the prompt, it misses retrieved context and connector results, which are the high-volume route. Restoration belongs in the application, gated on entitlement, and every retrieval and restoration is a telemetry event under opaque IDs so the log itself holds nothing sensitive. The third-party platforms guide walks the four-step design, and the build procedure places it as step seven.

Where Hardshell sits

Hardshell performs substitution on the provider boundary as part of a data layer that also limits each retrieval to what the requesting user is entitled to see and records every retrieval and restoration under opaque IDs. Stand-ins are consistent across documents and queries so ChatGPT, Claude, Copilot or any other model reasons over a coherent record, the mapping never leaves the organization's environment, and real values are restored only for entitled readers. It deploys as a cloud service, in a private cloud or as an offline bundle.

Frequently asked questions

Is redacting names enough for GDPR?

Removing direct identifiers reduces risk and does not by itself make data anonymous, because quasi-identifiers in the remaining text can re-identify the person. Under the GDPR, data that can still be attributed to a person with additional information or by combination remains personal data. Redaction of direct identifiers plus retained context is usually pseudonymization at best.

Does masking make data anonymous?

No. Masking hides part of a value for display. Initials, partial numbers and years combine with other fields, and a model performs that combination readily. Treat masked data as personal data with a cosmetic layer.

Can we reuse a payments tokenization system for AI context?

For structured identifiers, partly. Payment tokenization is built for field-level references and all-or-nothing vault access. AI context needs consistent stand-ins across free text and restoration scoped to the reader. A vault can supply the mapping; the consistency and the scoping have to be designed in.

Does substitution degrade answer quality?

Not materially, when the stand-ins are consistent and well-formed. The model sees a coherent record with plausible names, numbers and shifted dates and answers as it would in the clear. Redaction degrades answers because it removes the distinctions the model needs; substitution preserves them.

What about identifiers a model already memorized?

These techniques govern what reaches a model as context. They do nothing about what a model learned in training, which is the subject of the training data leakage guide, and they do not prevent a model from inferring identity from quasi-identifiers in the context it is given, which is why they are paired with scoped retrieval.

Sources

Regulation (EU) 2016/679 (GDPR), Article 4(5) (pseudonymisation), Recitals 26 and 28, Article 32. 45 CFR 164.514(a)-(b) (HIPAA de-identification: Safe Harbor and Expert Determination) and 45 CFR 164.502(b) (minimum necessary).

NIST, SP 800-188, De-Identifying Government Datasets: Techniques and Governance, September 2023. PCI Security Standards Council, tokenization guidance for PCI DSS.

OWASP, Top 10 for LLM Applications 2026, sensitive information disclosure entry.