Read It Cold
Before I open-sourced a diagnostic tool in August, I had five readers take the tree a slice at a time and check claims, not wording. Every doc claim, every comment, against the code it described. Forty-seven findings raised. Thirty-seven survived a second pass that tried to knock them down.
The one that keeps coming back is the one no test can see: a sentence that was true when it was written, about code that has since moved.
What a Stale Sentence Costs
The README and the security file both tell the reader to put a reverse proxy in front of the app. The origin check on requests that change data compared whole origins. Behind a reverse proxy, the origin the app sees is not the origin the browser sent. Every write was refused with a 403 in exactly the deployment the docs recommended.
The stale-build reload matched any URL containing a certain path segment, under a comment that said the check was same-origin only. If a third party's script at that path failed to load, the page reloaded. The reload throws away interview answers nobody has sent yet.
A settings value that caps the number of photos did nothing. The server refused a request above the setting. The intake form capped at a hard-coded transport maximum and told the user that number instead.
A contract file called the model's reasoning channel dead in production because "the system prompt does not ask for it." The prompt file asked for it on every turn and put it in the required output shape.
None of those would fail a test. All of them would fail a reader who checks one thing.
Line Numbers Rot
Three references in a component pointed at specific line numbers in the stylesheet. All three had rotted. One was off by seven hundred lines, in a file that is still growing. Two comments sized things to a 56-pixel header that had been 44 pixels for fifteen hundred lines. Three cross-references pointed "below" at rules defined above.
The fix is not to update the numbers. It is to delete them and name the property instead. The property is the fact that lasts, and you can grep for it. A citation to a line number is a citation to a moment, and the moment is gone by the time anyone reads it.
There was a worse version. A find-and-replace that removed quoted phrases from comments during an earlier scrub had left about twenty comments broken mid-sentence. Several had lost the reason they were written to record and said something circular instead. The comment survived. Its reason did not. That is a way automated cleanup fails that I had never seen named. I would not have caught it here without a reader going comment by comment.
The One Citation
The claim in the README that the eval hits 96 percent pointed at a line in the eval runner. The line was a research-resume block. The real line was 152 lines later.
A skeptical reader checks exactly one citation. If the one they check is wrong, they stop reading, and everything else in the document loses its standing at once. The number was right. The pointer was wrong. That is enough.
What a Clone Cannot Do
The most useful finding was about honesty rather than correctness.
A measurement script in the repo cannot run at all outside my machine. It finds a headless browser through a hard-coded version-manager path, looks up Chrome in a macOS-only location, and defaults to a gitignored run directory. The README now says so. The scenarios, the harness, and the scoring code all ship, and a reader can produce equivalent runs. But the specific numbers in the docs came from a script only I can run, and the docs should not suggest otherwise.
Publishing a spec that promises checked sources while the code cannot check them is the kind of claim a careful reader catches. Better to say what a clone can do.
The Habit
I now run this before anything goes public, and I run it the same way. Fresh readers, no context from the build, each with a slice of the tree and one instruction: check what the words claim against what the code does. Then a second pass that argues against what the first pass found, so the report is not just a list of things one model thought were wrong.
Thirty-seven confirmed findings in a repo I thought was clean. That is the whole argument. Every one of them was a sentence I had written or approved, and every one of them was true once, which is the most forgiving thing I can say about a comment.
-- Justin Higgins. Software Engineer, Midwest. Had five strangers read his repo before it shipped, and fixed thirty-seven sentences that used to be true.
Companion pieces: Verify the Artifact, Not the Process - the audit that found the screenshot bug. The Decision Log Ate the Project - sentences that cited each other.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com