Read It Cold
Before I open-sourced a diagnostic tool in August, I had five readers take the tree a slice at a time and check claims, not wording: every doc claim and every comment, against the code it described. They raised forty-seven findings. Thirty-seven survived a second pass that tried to knock them down.
The one that keeps coming back is the one no test can see: a sentence that was true when it was written, about code that has since moved.
What a Stale Sentence Costs
The README and the security file both tell the reader to put a reverse proxy in front of the app. The origin check on requests that change data compared whole origins, and behind a reverse proxy, the origin the app sees isn't the origin the browser sent. So every write was refused with a 403 in exactly the deployment the docs recommended.
The stale-build reload matched any URL containing a certain path segment, under a comment that said the check was same-origin only. If a third party's script at that path failed to load, the page reloaded, and the reload throws away interview answers nobody has sent yet.
A settings value that caps the number of photos did nothing. The server refused a request above the setting, but the intake form capped at a hard-coded transport maximum and told the user that number instead.
A contract file called the model's reasoning channel dead in production because "the system prompt does not ask for it." The prompt file asked for it on every turn and put it in the required output shape.
None of those would fail a test. All of them would fail a reader who checks one thing.
Line Numbers Rot
Three references in a component pointed at specific line numbers in the stylesheet. All three had rotted, one by seven hundred lines, in a file that's still growing. Two comments sized things to a 56-pixel header that had been 44 pixels for fifteen hundred lines. Three cross-references pointed "below" at rules defined above.
The fix is to delete the numbers and name the property instead, not to update them. The property is the fact that lasts, and you can grep for it. A citation to a line number is a citation to a moment, and the moment is gone by the time anyone reads it.
There was a worse version. A find-and-replace that removed quoted phrases from comments during an earlier scrub had left about twenty comments broken mid-sentence. Several had lost the reason they were written to record and said something circular instead. The comment survived and its reason didn't. That's a way automated cleanup fails that I'd never seen named, and I wouldn't have caught it here without a reader going comment by comment.
The One Citation
The README's claim that the eval hits 96 percent pointed at a line in the eval runner. That line was a research-resume block. The real line was 152 lines later.
A skeptical reader checks exactly one citation. If the one they check is wrong, they stop reading, and everything else in the document loses its standing at once. The number was right and the pointer was wrong, and that's enough.
What a Clone Cannot Do
The most useful finding was about honesty rather than correctness.
A measurement script in the repo can't run at all outside my machine. It finds a headless browser through a hard-coded version-manager path, looks up Chrome in a macOS-only location, and defaults to a gitignored run directory. The README now says so. The scenarios, the harness, and the scoring code all ship, and a reader can produce equivalent runs. But the specific numbers in the docs came from a script only I can run, and the docs shouldn't suggest otherwise.
Publishing a spec that promises checked sources while the code can't check them is the kind of claim a careful reader catches. Better to say what a clone can do.
The Habit
I now run this before anything goes public, and I run it the same way. Fresh readers, no context from the build, each with a slice of the tree and one instruction: check what the words claim against what the code does. Then a second pass argues against what the first pass found, so the report isn't just a list of things one model thought were wrong.
Thirty-seven confirmed findings in a repo I thought was clean. That's the whole argument. Every one of them was a sentence I'd written or approved, and every one of them was true once, which is the most forgiving thing I can say about a comment.
-- Justin Higgins. Software Engineer, Midwest. Had five strangers read his repo before it shipped, and fixed thirty-seven sentences that used to be true.
Companion pieces: Verify the Artifact, Not the Process - the audit that found the screenshot bug. The Decision Log Ate the Project - sentences that cited each other.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com