Spider-Man Is Not a Person
This summer I put a redaction gateway in front of a small app my family uses. The gateway sits between the app and the model vendor, reads every request on the way out, and masks anything that looks like personal information before it leaves the building. It is the kind of thing a regulated shop needs and almost nobody builds correctly, so I built it to find out where "correctly" is.
The app went down for days. Its health check stayed green the entire time. I found out the way you find out about most things a health check missed, by using the app.
Five False Positives in a Static Prompt
The app's system prompt has a list of things a child is interested in. Spider-Man. Dogman. Pokemon. Mario. It also has the instruction "Output ONLY the JSON object."
Under the healthcare baseline, the gateway's name detector read the first four as PERSON. It read the literal token JSON as a LOCATION, which the address recognizer then turned into an ADDRESS. Five hits inside a static system prompt that never changes, so every single request tripped the same five, and every single request got blocked. One hundred percent of turns.
The app caught the failure and failed softly, which is the behavior you want and the reason nobody noticed. It returned HTTP 200 the whole time. Nothing outside the gateway's own traffic feed showed an outage. The health check was checking whether the app could answer, and it could. It just could not do the thing it existed to do.
I found it by reading the gateway's feed, not by any alert.
The Score Is Not a Confidence
The obvious fix is a threshold. Names below some confidence, let them through. I went to tune it and found that the detector's name layer returns 0.85 for every PERSON and every LOCATION it emits. Real or not. Alice Whitfield and Margaret Chen score 0.85. So do Portugal, Spider-Man, and JSON.
A gate that removes one removes the other. The number is not a confidence in the everyday sense, and you cannot tune it as if it were. I had to write that down as a decision so nobody, including me, tries it again.
What the gateway needed was a different kind of rule. Not "lower the bar for this type" but "this exact term is not PII." An allow list, scoped by type, matched on the span's own text. It suppresses a term, never a region, so an allow-listed word sitting next to a real name does not shield the name. Approval is the trust boundary, so that rule kind gets the strictest validation of anything in the pack. Bounded count, no empty terms, no line spans, and a shorter maximum length than a deny term. The longer the string, the more likely someone is trying to switch off a whole type rather than name one thing.
There is a second limit that is easy to miss. The gateway has a flywheel that turns misses into proposed rules. It can do that because in the synthetic harness the right answer is known up front. It has no such answer key for real application traffic. It cannot work out on its own that a detection was wrong. Only a human can say that Spider-Man in a system prompt is not a person. So that judgment goes through the same propose, approve, hot-reload path as everything else, with an audit event and a version bump. A proposal with no reason attached is rejected the moment it is written. An unexplained suppression is unreviewable.
The pattern pack is append-only, so the allow list I authored for that app cannot be retired and stays in the pack. I recorded that as a consequence rather than fixing it. The app is no longer routed through the gateway. The evidence it produced is why the rule exists, and that holds no matter which app turned it up.
The Only Honest Recall
The gateway reports its own recall. Of course it does. Every detector reports its own recall, and every detector is grading its own homework when it does.
So the harness asks a different question. It plants known values into synthetic traffic, sends the traffic through the real gateway to a mock vendor, and then checks one thing: did the planted value physically arrive at the far side? If yes, that is a leak, no matter what the detector claimed it caught.
That number is behavioral recall. It is measured on what got through, not on what was reported. The detector's own number is reported recall. The gap between the two is the self-deception metric, and it is printed on the score page on purpose. On one 200-request run, behavioral recall was 100 percent and reported recall was 99 percent. The detector was slightly wrong about itself in the safe direction. It could just as easily have been wrong the other way. The only reason I would know is that I read what the far side received instead of trusting the proxy's report on itself.
The harness also plants traps: order IDs, version numbers, card numbers that fail the Luhn check. Things that look sensitive and are not. Flagging them costs precision, and the run above flagged zero. Every miss goes into a permanent set that replays in future runs, so once the gateway has missed something it gets re-tested on it forever. The quality can climb. It cannot slip back. When the score dips it is usually because the corpus got harder, which is the point.
Nothing Auto-Applies
The flywheel proposes. A human approves. That is not a limitation I accepted. It is the design.
When the flywheel looks at a miss, it can propose a curated regex from a hand-vetted library, a score-gate adjustment only if a probe proves the detector actually saw the span at some lower confidence, or a deny-list entry as the honest fallback when the engine never saw the span at all. If none of those honestly applies, the proposal is marked manual: a human needs to author a real recognizer. Nothing is papered over.
I have watched enough vendor dashboards to know that the failure mode of a self-improving safety system is that it improves the number. The harness is built to catch its own optimism. The proposals ship with the count of misses they explain. The approvals bump a version. The ledger keeps every run, seed, and pack version, so the sparkline is not a claim, it is a history.
Twenty Years of This
I do not think any of this is specific to PII detection. A green health check on an app that cannot do its job. A score that looks like a probability and is not one. A system reporting its own accuracy. A dashboard that can only go up. I have seen every one of those inside a large regulated company, usually in a vendor product, usually discovered by an engineer reading raw logs at the wrong hour.
The thing I would want a reader to take is not the gateway. It is the habit. Verify the artifact, not the process that produced it. Read what the far side received. And when a number tells you something flattering about a system, ask whether the number came from the system.
-- Justin Higgins. Software Engineer, Midwest. Built a redaction gateway and learned it could not tell a superhero from a person.
Companion pieces: Everything Looks Like Everything - the instrument that measured nothing. Twelve Days of Green Tests - what is the anchor I did not author?
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com