Eighty-Nine of Ninety Were Not Gated
On September 26 an audit of the harness behind The Shape Leaks found that the arm of the experiment labeled "gated" wasn't gated. Eighty-nine of its ninety husks were built with the verifier switched off. Every figure that post reports for LLM translation, the one method it says did any work, describes a rewriter running without its verifier.
The Shape Leaks needs a correction, and this post explains why. The verdict survived, but the comparison it was built on didn't. The more useful part is what happened to the numbers next. The usual move is to quietly fix the figure, and that isn't what happened here.
What the Audit Found
The audit ran four lanes: the husking methods, the service, the statistics and the harness. A second reviewer re-checked each one. It turned up 37 findings, all reproduced. Thirty-three shipped as fixes and four were deferred to me.
The biggest was about how the treatment got assigned. The experiment built the LLM-translation husks twice, once with the verifier on (the repo calls it the gate) and once with it off, so the effect of the verifier could be measured. To build the ungated arm, the harness switched the verifier off by editing process-wide settings from six worker threads. The verifier reads those settings when it checks a husk. So while one thread had it off for an ungated husk, gated husks in the other threads were checked with it off too. Overlapping jobs saved and restored each other's overrides and left them set.
Another harness in the same repo built the ungated arm correctly, in a child process with its own environment. This one copied its constants, under a comment saying so the two couldn't drift apart, and left the isolation behind.
The records had the answer from the day of the run. Each husk records the settings it was checked under, and 89 of the 90 "gated" husks record the off settings. One carries the shipped ones. A line in the roadmap said the opposite: the gate refused nothing, 90 of 90 ok. A verifier that's on with nothing to refuse and a verifier that's off write the same line.
At the settings shipped then, 39 of the 90 would have been refused on their first attempt. Two were one source file minus its trailing newline, and the attackers got all four forced-choice answers on them right.
What the Numbers Mean Now
In The Shape Leaks, the 78 percent on the menu, the roughly 50 percent in free text, the plus 0.44 to plus 0.48, the copied-span shares and the 22 of 90 all measured an unverified rewriter. A verified figure needs a paid re-run. A gated build retries each refusal once and never attacks what's still refused, so the gated number can't be derived from these records.
The fail isn't at risk. On the husks the verifier would have passed, the two attackers scored about 81 and 71 percent, against 78 and 69 percent on the whole arm. That describes recorded husks, not a gated result.
What doesn't survive is the comparison of gated against ungated, the thing the second arm existed to measure. It was void anyway: the ungated arm had only been attacked on one domain.
The same pass moved smaller figures: 986 of 998 attacker calls is 948 of 960 against the arms the run declared, and eight of sixty canonicalized files kept an authored comment the gates couldn't see, which shifts the structure-only result to about plus 0.21 to plus 0.25.
What Was Not Done
No published number was rewritten, and the decision log entry for that day sets the rules.
Each affected figure keeps its value, in the limitations file, the README and the roadmap, with a pointer to a numbered erratum. There are seventeen, E1 through E17. Each says what was wrong, gives the corrected value or says why one needs a re-run, and says whether a verdict changes. None does.
A statement that was simply false gets struck through with a dated correction instead of deleted. The roadmap line about the gate refusing nothing is still there, struck, with "false" and "the gate was not running" beside it.
Committed run artifacts stay as they were run. Where today's code would produce something different, that's recorded and not repaired. There's one exception: the worksheet for hand-labelling judge agreement showed the wrong attacker's guess on 57 of 98 rows, with none of its 98 labels filled in, so it was regenerated in place. Nothing had been computed from it.
The fixes came with tests: 69 to 93 for the service, 20 new for the harness. The regression test for this race runs both arms against a stub that returns the source unchanged, which the shipped verifier must refuse, so a refusal shows which arm a husk was really built under. The ungated arm now builds in a child process, and every husk's recorded settings are checked against its arm's definition. A mismatch fails closed: the husk is recorded as a harness error and kept from the attackers.
The re-run is priced, not ordered. Running the gated arm again means 90 husks plus up to 90 retries, 360 attacker calls and up to 180 judge calls. The estimate is roughly $15, and it's an estimate, not a quote: the attackers' and judge's usage was never recorded, so it scales the experiment's average of about $43 over 1,475 calls. The harness records that usage now. The re-run is an owner decision on the roadmap that blocks nothing, and it hasn't been run.
Why Not Fix the Figure
The quick fix is to correct the number in place. Here that would replace a wrong number with an unmeasured one, because the right figure doesn't exist until the re-run is paid for.
The 78 percent isn't an arithmetic mistake. It's a real measurement of an ungated rewriter that carried the wrong label. The label was the error, so the erratum fixes the label and keeps the measurement.
The Shape Leaks is public, so anyone who read it holds the old numbers. A quiet edit would leave them holding numbers that no longer match the page. A strike-through, a date and an erratum number give the old value, the new status and the reason in one place, beside the wrong number, which stays part of the record.
What a Treatment Is
An arm label is a claim about how a unit was made. The treatment is the state it was actually made under. Here the claim lived in the config, the state lived in a setting six threads were writing to, and the two disagreed for 89 of 90 units with nothing comparing them.
A treatment applied through state the workers share is applied to whoever reads it next. The records carried the truth the whole time, and nothing checked them against the label until now.
The Shape Leaks closes by saying that mostly what I keep is the experiment. I still do: the experiment as recorded, including what its harness did wrong, and a verdict the bad arm didn't move.
-- Justin Higgins. Software Engineer, Midwest. Struck the false sentences in his own experiment's record and left every number where it was published.
Companion pieces: The Shape Leaks - the experiment whose figures this corrects. Read It Cold - sentences that were true once; the roadmap line here never was.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com