Spider-Man Is Not a Person
This summer I put a redaction gateway in front of a small app my family uses. The gateway sits between the app and the model vendor, reads every request on the way out, and masks anything that looks like personal information before it leaves the building. It's the kind of thing a regulated shop needs and almost nobody builds correctly, so I built it to find out where "correctly" is.
The app went down for days, and its health check stayed green the entire time. I found out the way you find out about most things a health check missed: by using the app.
Five False Positives in a Static Prompt
The app's system prompt has a list of things a child is interested in: Spider-Man, Dogman, Pokemon, Mario. It also has the instruction "Output ONLY the JSON object."
Under the healthcare baseline, the gateway's name detector read the first four as PERSON. It read the literal token JSON as a LOCATION, which the address recognizer then turned into an ADDRESS. That's five hits inside a static system prompt that never changes, so every single request tripped the same five and every single request got blocked. One hundred percent of turns.
The app caught the failure and failed softly, which is the behavior you want and the reason nobody noticed. It returned HTTP 200 the whole time, and nothing outside the gateway's own traffic feed showed an outage. The health check was checking whether the app could answer, and it could. It just couldn't do the thing it existed to do. I found it by reading the gateway's feed, not from any alert.
The Score Is Not a Confidence
The obvious fix is a threshold: let names below some confidence through. I went to tune it and found that the detector's name layer returns 0.85 for every PERSON and every LOCATION it emits, real or not. Alice Whitfield and Margaret Chen score 0.85. So do Portugal, Spider-Man, and JSON.
A gate that removes one removes the other. The number isn't a confidence in the everyday sense, and you can't tune it like one. I had to write that down as a decision so nobody, including me, tries it again.
What the gateway needed was a rule that says "this exact term is not PII," instead of one that lowers the bar for a whole type. That's an allow list, scoped by type and matched on the span's own text. It suppresses a term, never a region, so an allow-listed word sitting next to a real name doesn't shield the name. Approval is the trust boundary, so that rule kind gets the strictest validation of anything in the pack: bounded count, no empty terms, no line spans, and a shorter maximum length than a deny term. The longer the string, the more likely someone is trying to switch off a whole type rather than name one thing.
There's a second limit that's easy to miss. The gateway has a flywheel that turns misses into proposed rules, and it can do that because in the synthetic harness the right answer is known up front. It has no such answer key for real application traffic, so it can't work out on its own that a detection was wrong. Only a human can say that Spider-Man in a system prompt isn't a person. That judgment goes through the same propose, approve, hot-reload path as everything else, with an audit event and a version bump. A proposal with no reason attached is rejected the moment it's written, because an unexplained suppression is unreviewable.
The pattern pack is append-only, so the allow list I wrote for that app can't be retired and stays in the pack. I recorded that as a consequence rather than fixing it. The app is no longer routed through the gateway, but the evidence it produced is why the rule exists, and that holds no matter which app turned it up.
The Only Honest Recall
The gateway reports its own recall. Of course it does. Every detector does, and every detector is grading its own homework when it does.
So the harness asks a different question. It plants known values into synthetic traffic, sends the traffic through the real gateway to a mock vendor, and then checks one thing: did the planted value physically arrive at the far side? If it did, that's a leak, whatever the detector claimed it caught.
That number is behavioral recall, measured on what got through rather than on what was reported. The detector's own number is reported recall. The gap between the two is the self-deception metric, and it's printed on the score page on purpose. On one 200-request run, behavioral recall was 100 percent and reported recall was 99 percent, so the detector was slightly wrong about itself in the safe direction. It could just as easily have been wrong the other way. The only reason I'd know is that I read what the far side received instead of trusting the proxy's report on itself.
The harness also plants traps: order IDs, version numbers, card numbers that fail the Luhn check. They look sensitive and aren't. Flagging them costs precision, and the run above flagged zero. Every miss goes into a permanent set that replays in future runs, so once the gateway has missed something it gets re-tested on it forever. Quality can climb but can't slip back. When the score dips it's usually because the corpus got harder, which is the point.
Nothing Auto-Applies
The flywheel proposes and a human approves, by design.
When the flywheel looks at a miss, it can propose a curated regex from a hand-vetted library, a score-gate adjustment (only if a probe proves the detector actually saw the span at some lower confidence), or a deny-list entry as the honest fallback when the engine never saw the span at all. If none of those honestly applies, the proposal is marked manual: a human needs to write a real recognizer. Nothing gets papered over.
I've watched enough vendor dashboards to know that the failure mode of a self-improving safety system is that it improves the number. The harness is built to catch its own optimism. Proposals ship with the count of misses they explain, approvals bump a version, and the ledger keeps every run, seed, and pack version, so the sparkline is a history and not a claim.
Twenty Years of This
I don't think any of this is specific to PII detection. A green health check on an app that can't do its job. A score that looks like a probability and isn't one. A system reporting its own accuracy. A dashboard that can only go up. I've seen every one of those inside a large regulated company, usually in a vendor product, usually discovered by an engineer reading raw logs at the wrong hour.
What I'd want a reader to take is the habit, not the gateway. Verify the artifact, not the process that produced it. Read what the far side received. And when a number tells you something flattering about a system, ask whether the number came from the system.
-- Justin Higgins. Software Engineer, Midwest. Built a redaction gateway and learned it could not tell a superhero from a person.
Companion pieces: Everything Looks Like Everything - the instrument that measured nothing. Twelve Days of Green Tests - what is the anchor I did not author?
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com