A Tie Is a Loss
Asking a model to try again is not a competition. It is the same model with the same habits producing a slightly different haircut, and you picking the one that looks better that day. I did that for months and could not tell whether anything was actually improving or I was just getting used to it.
What I do now is a ratchet. It has four parts, and each one exists because leaving it out produced a specific failure.
Two Builds, Structurally Divergent
Two agents implement competing versions in parallel from deliberately different starting skeletons. They do not see each other's work in progress. Independence is the point.
Then a gate, before anyone scores anything. Strip the identifiers and comments from both diffs. Could you still tell them apart by structure alone? If not, fail. Send the more derivative one back and tell it to take the problem apart again, not repaint it. Two candidates with the same structure and different paint fail the gate automatically, even if both look fine.
That gate was added after the third run of a page ratchet, when I noticed the first three versions all rode the same vertical-scroll skeleton. The ratchet had gone stale by stacking features on one layout. A page that keeps the champion's bones and only adds motion fails the layout-divergence gate now, even if it scores higher.
A Versioned Rubric
Each candidate is scored axis by axis against a rubric, and the rubric's version number is recorded with the scores. Change the axes, bump the version, and re-score the champion under the new rubric before comparing anything. Never compare scores across rubric versions.
A useful step before declaring a win: send something in to argue that the winner is not actually better than the champion on the axis it claims. Keep the win only if it survives.
Strictly Greater
The winner ships only if its total is strictly greater than the reigning champion's total on the current rubric. A tie is a loss. Do not inflate scores to manufacture a win.
Losses are recorded too. Every run appends a row: date, per-axis scores, total, rubric version, result. A failed attempt is audit data, not something to tidy away. The ledger also carries a no-repeat list of techniques already used, so a future run cannot win by running back a move the judge already rewarded.
Output never overwrites. Every candidate lands in a new, auto-incremented file, and the source is frozen.
Where It Refused Me
The system is only credible because of the times it told me no.
An executive brief scored 41 against a champion's 31, with a one-point drop on narrative. Gate failed. I rewrote the opening instead of overriding the gate, re-scored, and it passed. The ledger calls that the family's first erratum.
A styling ledger records that the shipped champion was promoted on owner preference, unscored. It also records that the highest judged total still belongs to a candidate that is not the canonical one. The next run has to beat a judged 40 or my preference, and the ledger says which.
And one where the judge was wrong. Two shells for a 3D globe. The judge picked the framework-free one, 43 to 42, from headless screenshots. I shipped the other one ten days later by override. The rubric never measured live interaction, and the loser runs dramatically better in real use. The ledger records the override, records the rubric gap, and adds a measured runtime-performance axis to the next rubric version. I had picked the same shell myself before I saw the verdict, and that is recorded too, because it says as much about me as about the shell.
A scoring system that admits when the human overruled it, and why, and what it changed as a result, is more credible than one that does not.
The Ceiling the Judge Named
In one document ratchet, the judge flagged that every entrant had hand-set numbers that could not be checked from inside the document. A future run could go after that with an outside verifier that knows the real values. It did not claim a clean win. It named its own blind spot and put it in the ledger as the next opening.
That is what I want from a judge. Not a verdict. A verdict, the reasoning, the loser's flaws as the next run's spec, and the thing the rubric cannot see.
What It Costs
Two builders, a gate, a judge, per unit. On a small repo the margins are often narrow, five points on fifty. I have written elsewhere that I want to see a run where the competition changes the outcome more before I default to it everywhere.
Where it has clearly paid is anything visual or structural, where I cannot say what I want until I see two versions that differ in their bones. And the ledger has paid for itself on its own: a history of every attempt, what won, what lost, and why. I can read it months later and trust it, because the process wrote it, not me afterward.
-- Justin Higgins. Software Engineer, Midwest. Makes agents compete, records the losses, and lets the judge refuse him.
Companion pieces: Five Hours Unattended - five of these ran in one night. Survived Is Not Endorsed - the review that checks the reviewers.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com