Also known as: PII detection, data redaction, anonymization, de-identification
Detecting and removing personally-identifiable information from LLM inputs and outputs — names, emails, phone numbers, addresses, IDs. A classic small-model task: high-volume, narrow, latency-sensitive, with structured target output.
PII redaction — finding and removing personally identifiable information from LLM inputs and outputs — is one of the most underrated production tasks in modern AI systems. It’s not glamorous; it’s also non-negotiable in any pipeline that touches healthcare, finance, legal, customer support, or HR data.
Different jurisdictions (GDPR, HIPAA, CCPA) have different definitions. Production systems usually adopt the strictest superset.
The Sweeney 2000 result: 87% of the US population is uniquely identifiable from the triple (zip code, date of birth, sex) alone. Quasi-identifiers compose multiplicatively — each one cuts the candidate population by an order of magnitude, and three rough features can re-identify a specific person from anonymized data. HIPAA’s Safe Harbor rule explicitly removes 18 such fields including dates more granular than year and zip codes more granular than the first three digits, precisely because composition attacks defeat per-field redaction. The practical implication: a PII model that treats fields independently is broken; you need a model that recognizes when a combination of features is identifying even when each in isolation isn’t.
PII redaction has the classic “specialize and ship” shape:
Every property that argues for a specialized fine-tuned small model over a frontier LLM call is present. A 0.5-3B model trained specifically on PII detection runs at single-digit milliseconds per request, costs ~10⁻⁵ of a frontier-LLM call, and hits 99%+ recall.
The training pipeline is the standard specialize-and-distill recipe:
Two places it usually sits:
For observability specifically, the input filter is what makes it safe to log full traces — you can capture rich data without violating compliance.
Production PII redaction is asymmetric: missing PII is a regulatory and reputational disaster; redacting a non-PII word that looks like PII just makes one response slightly worse. So PII models are tuned to high recall (99.5%+) at the cost of some precision . Threshold tuning is one of the few cases where “just lower the threshold” is the right answer — pre-compliance reviewers will tell you the same.
The right Pareto frontier for PII redaction is wildly asymmetric: a missed name is a lawsuit; a redacted non-name is a typo.
Regex misses context-dependent PII (Bob with last name Apple is different from buying an Apple). Frontier LLMs work but cost $0.01-1 per call and add 200-2000ms latency on every request — unworkable in the input pipeline. A 0.5-3B fine-tuned model hits 99%+ recall at <50ms latency on a single GPU.
PII redaction biases hard toward recall — missing PII is a compliance breach; redacting too aggressively just makes the response a bit less natural. Production systems target 99.5%+ recall; precision around 90-95% is fine. The Pareto frontier shifts dramatically vs balanced classification tasks.
Synthetic generation, mostly. A frontier LLM produces (text with PII, redacted text, span annotations) triples at scale; the small model trains on that. Real customer data is too risky to use directly for training PII detectors — synthetic data sidesteps the meta-compliance problem.
