The Writer Can't Be the Checker
On October 14 I had a fleet of agents put together a page showing sixteen of my builds. Sonnet scouts read 128 of my repos. Two Opus judges ranked what came back, each from a different angle. Sonnet writers drafted the copy for each build and captured the screenshots, often by starting the app and shooting it running. Fable agents designed versions of the page. Then every entry went to a checker: a fresh Opus agent that hadn't written a word of it, holding the sources the draft relied on and the power to veto it.
Sixteen entries went to checkers, and sixteen came back marked FIXED. None came back PASS. A final review vetoed the page and two companion pages with seven must-fix items between them, and a review of a rebuilt version vetoed it again.
The corrections had something in common, and it's why the agent that writes a claim can't be the one that checks it.
The Responsible Version
One build is a repair loop for my deployed services. When a service fails two checks in a row, or a daily audit finds a known security hole in the code it runs, a fix agent works in a fresh copy and opens a pull request. An earlier draft of the page said the loop "has an AI write the fix for a person to approve."
No person approves it. The fix agent's instructions tell it to merge its own pull request when the build passes and every test that passed before still passes. Otherwise the request stays open and the item lands on a page for me. After a merge, the loop watches the deploy and rolls it back if the service breaks. Fifteen of the sixteen fixes in its log were merged by the agent, and the sixteenth was a redeploy.
The draft didn't pull its sentence from nowhere. "A person approves" is what a careful system is supposed to say, and it's what most readers expect to read. The writer produced the responsible-sounding version instead of the true one, and it reads fine. The checker caught it because it had the loop's own fix log open and was reading that, not the sentence.
Zero Wrong
Another build is a plain-English question box over a database. A model plans the answer, ordinary code checks the plan and computes every number, and when the data can't support an answer the system declines and says why. I'd run it on a public text-to-SQL benchmark where half the questions are built to have no answer.
The draft led with zero wrong answers across the whole benchmark. That was true, and it was the wrong thing to lead with. In its best run the system answered about one in seven of the questions that had an answer, all of them right, and declined everything else. Zero wrong is easy to reach if you decline enough.
The checker rewrote the figure so the coverage sits inside it: how many it answered, that none were wrong, and that it declined the rest. Its notes put the rule in a line: coverage is part of the claim. A zero shown alone describes a system that answers everything.
The warning was in the pile before any copy existed. The brief the writer worked from called the zero the proof, and in the same file one of the ranking judges had written that the zero should never be headlined without its coverage. The draft headlined it. The checker had the same file and the same note, and it had a different job. It was there to test the claim, and it could refuse it.
Numbers From Next Door
Two more catches were real numbers in the wrong sentence, both in one entry.
That build answers questions over a set of contracts, and its tests include sixty trick questions the contracts can't answer, which it should decline. The page showed sixty of sixty declined, beside the production model's other results. The sixty came from an earlier run on a smaller model. The production model declined fifty-seven of sixty.
The same entry said the GPU behind its second model, the one that reorders search results, served 4,840 calls in the full test run. The 4,840 came from a separate retrieval test of 2,423 lookups. The arithmetic gave it away. That retrieval test used about 4,300 GPU-seconds to make its 4,840 calls, and the full run used 2,230 in total.
Neither number was invented. Each was measured and sitting in a file, and each had ended up in the sentence where it looked best. Rereading doesn't catch that, because the writer remembers the number as real, and it is.
The Errors Leaned
Afterwards I counted. The word "overclaim" shows up fifteen times in the sixteen checkers' notes, in thirteen of them. Nothing is called an underclaim. The nearest is one sentence called too narrow.
Part of that is structural. A modest sentence is rarely wrong, so a checker has less reason to flag it. But the big catches all moved the same way: toward a person in the loop, a zero with no denominator, a smaller model's perfect score beside the bigger model's results, a bigger workload for a GPU. Random mistakes would scatter. These moved toward what a reader would hope was true.
That's why the writer can't check. A fluent writer reaches for the sentence that usually goes there, and on every reread that sentence still looks right to the one who wrote it. The checker starts from the other end. It never saw the conversation the page came out of. It opens the source and asks whether the sentence is in it.
Pixels
The words got checked line by line. The pictures got looked at. One line in them got through twice.
One build is a data app for a regional contractor. The copy was careful and never named the company or where it is. The script that captured the screenshots had scrubbed them too. It swapped the company's name for a generic label and swapped the town for the metro area around it. So in three screenshots, the app's sidebar printed the contractor's home metro.
The first whole-page review opened those screenshots and flagged two of them for dollar figures. It didn't flag the sidebar. Later, an agent choosing replacement screenshots read that line, judged it harmless, and noted that it already showed in the lead image. The first leak had become the precedent for the second. The checker on the rebuilt version of the page caught it, and the line was masked out of all three images. That review's veto had two must-fix items, and both were about images.
Copy that had been checked sentence by sentence sat beside pictures that said what it had left out. Opening the image isn't enough. The check has to read it line by line, the way it reads the copy.
When the Checkers Disagreed
Two agents disagreed about one sentence: whether a sign-in service's audit log records revocations. One had read the app's own screen, which describes the log as "every sign-in, sign-out and revocation, in order," and took that as proof. The other had read the code, where the audit function accepts three event types: sign-in, sign-out and failed. Revocation isn't one of them.
The app's own copy leaned the same way the drafts did, toward the log a careful system ought to keep. The page now says what the code writes. When a screen and its code disagree, the screen is one more draft.
Every Rewrite Is a New Draft
After the checks had passed, a later pass by a different model polished the prose. It rewrote the benchmark line and put the answered count over the benchmark's full question total, calling that total answerable. Half of those questions are built to have no answer, so the new sentence doubled the denominator. A final scan caught it.
That error didn't flatter anything. It made the system look worse. It matters for a different reason: the checks were finished and the sentence changed afterward. A check covers the text it read, and a pass that rewrites prose writes new claims.
A Boundary Isn't a Hedge
One of the leans was mine. My writing rules say no hedging, and partway through the day I told the agents to "ditch the hedging." The fifty-seven of sixty read like a hedge, and the sixty from the earlier run took its place. That's how a smaller model's score ended up next to the production model's results.
The same rules carry a second half: state a boundary once, as a fact. The checkers used it against the first half. "Answered about one in seven, all right, declined the rest" is a fact about the system, stated once. "Sixty of sixty, on an earlier run with a smaller model" is a fact about the run. Neither needs a "may" or an "up to."
The checker for that entry kept both numbers, each labelled with its run. My instruction cut it to one, so the page showed the sixty labelled with its run and no fifty-seven beside it. A later pass also took the model's name off that label, one more rewrite after the check. A later rebuild dropped the sixty as well. The fifty-seven goes here.
What I'd Keep
The checker gets the sentence, the source and the rules, and none of the conversation that produced the draft. It gets a veto, not a comment box. It reads every line in the images. And whatever rewrites the text after it, its own fixes included, goes back through a check. The checkers here rewrote what they flagged, which made them writers too, and their rewrites went to the whole-page reviews.
In July the fix for my agent-written cards was to stop the verification from writing the copy. This time writing and checking were separate from the start, and all sixteen entries still needed fixing. Keeping them apart didn't make the drafts better. It's how anyone found out they weren't.
-- Justin Higgins. Software Engineer, Midwest. Sent sixteen drafts to checkers that never wrote them, and got sixteen corrections back.
Companion pieces: My Portfolio Read Like a Compliance Audit - the July cards, when the verification wrote the copy. Showing Includes Saying - a guard tested on words that did nothing for the pictures. Verify the Artifact, Not the Process - a screenshot script that checked everything but the screenshot.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com