Showing Includes Saying
This summer I built a small communication tool for a child in my family who does not talk the way the world expects. A word becomes a picture the child can change. Underneath there is a governor that decides when to draw and when to wait. And there is a guard whose only job is to make sure the picture never shows a human child in distress, no matter what the words said.
The guard's test suite was green on the day the app drew exactly the thing the guard exists to prevent. I would like to say I caught it in review. I caught it by looking at the screen.
Words and Pixels
The guard worked on the prompt. It took the text that was about to go to the image model and rewrote it to strip the child-referring words. Then it checked that those words were gone from the new string. Every test it had was a test on that string.
The requirement was about pixels. Every check the guard had was about words. So the suite could not have caught the failure and would never have caught it, because it was measuring the wrong thing. The guard protected a picture and had never once looked at a picture.
I keep finding this rule in different costumes: verify the artifact, not the process. A guard on an artifact it never looks at has the same hole every time, and the hole is invisible to any test written against the process.
The Measurement
So I measured the pictures. 504 renders, three repeats per cell, scored for whether the output showed a human child in visible distress.
No guard at all: 52 percent.
The version-one guard, which rewrote the child word to "character": 50 percent.
A style suffix telling the model to draw no humans, with no rewrite: 33 percent.
Binding the figure to a specific non-human noun, an animal: 0 percent, zero out of 48.
Version one was no different from having no guard, statistically, and on two prompts it scored worse than none. The reason is obvious once you have the number and invisible before. "Character" is an unbound noun. An image model asked for a character draws the cartoon subject it draws most often, and that is a human child. Only naming a figure the model must actually draw moves pixels.
Version two of the guard does not look for child-words. It asks whether the figure in the prompt is unbound: unnamed, or named only by a word that commits to nothing. Version one fired only when a child-word and a bad-situation word showed up together, and it caught 34 percent of the harmful prompts. Version two catches 91 percent.
Showing Includes Saying
Then the second half, which I had not thought about at all.
The app narrates the picture back to the child. If the guard rewrote the image prompt and the narration used the original, unguarded text, the app would say the thing it had just refused to draw. Saying it is the blunter of the two. Guarding the image and then narrating the unguarded version is worse than not guarding.
So the rewrite applies to both. Showing includes saying. That is now a sentence in the decision log. There is also a test that writes out the narration and fails if it contains what the image was not allowed to contain.
The One Measured Cost
Binding the figure to an animal changes the picture in cases where a human figure would have been fine. A drawing of a child playing becomes a drawing of a dog playing. That is a cost, and I measured it, and I accepted it, and there is a test that pins the trade.
The test's comment says that if this trade is wrong, it is a product decision, not a test repair. I wrote that because I know how the next session goes. A test fails, an agent reads the assertion, and the fastest path to green is to loosen the assertion. The comment is there to make that path feel like what it is.
What I Would Tell Someone Building One
If you have a guard on generated media, count how many of its tests look at the media. If the answer is zero, the guard is a hope.
Measure the failure rate with no guard first. Mine was 52 percent, which means the base rate of the harm was a coin flip, which means a guard that gets to 50 percent has done nothing and will look like it has done something.
And check the narration, the caption, the alt text, the log line. Everything that describes the artifact is a second artifact, and the second one is usually more explicit than the first.
This tool has one user and I am not going to describe them here. I will say why I measured 504 renders instead of trusting a green suite. I knew who would see the picture. That is the only reason I can offer for why anyone should do this. If you do not have a face in mind, imagine one.
-- Justin Higgins. Software Engineer, Midwest. Built an image guard, found its tests never looked at an image, and measured 504 of them.
Companion pieces: Verify the Artifact, Not the Process - the same rule, applied to screenshots. Spider-Man Is Not a Person - a detector that could not know it was wrong.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com