A Tie Is a Loss
Asking a model to try again isn't a competition. It's the same model with the same habits producing a slightly different haircut, and you picking whichever looks better that day. I did that for months and couldn't tell whether anything was actually improving or I was just getting used to it.
What I do now is a ratchet. It has four parts, and each one exists because leaving it out produced a specific failure.
Two Builds, Structurally Divergent
Two agents build competing versions in parallel from deliberately different starting skeletons, and neither sees the other's work in progress. Independence is the point.
Then a gate, before anyone scores anything. Strip the identifiers and comments from both diffs. Could you still tell them apart by structure alone? If not, fail, and send the more derivative one back to take the problem apart again rather than repaint it. Two candidates with the same structure and different paint fail automatically, even if both look fine.
That gate went in after the third run of a page ratchet, when I noticed the first three versions all rode the same vertical-scroll skeleton. The ratchet had gone stale stacking features on one layout. Now a page that keeps the champion's bones and only adds motion fails the layout-divergence gate, even if it scores higher.
A Versioned Rubric
Each candidate is scored axis by axis against a rubric, and the rubric's version number is recorded with the scores. Change the axes, bump the version, and re-score the champion under the new rubric before comparing anything. Never compare scores across rubric versions.
A useful step before declaring a win: send something in to argue that the winner isn't actually better than the champion on the axis it claims, and keep the win only if it survives.
Strictly Greater
The winner ships only if its total is strictly greater than the reigning champion's total on the current rubric. A tie is a loss. Don't inflate scores to manufacture a win.
Losses get recorded too. Every run appends a row: date, per-axis scores, total, rubric version, result. A failed attempt is audit data, not something to tidy away. The ledger also carries a no-repeat list of techniques already used, so a future run can't win by running back a move the judge already rewarded.
Output never overwrites. Every candidate lands in a new, auto-incremented file, and the source is frozen.
Where It Refused Me
The system is only credible because of the times it told me no.
An executive brief scored 41 against a champion's 31, with a one-point drop on narrative, and failed the gate. I rewrote the opening instead of overriding the gate, re-scored, and it passed. The ledger calls that the family's first erratum.
A styling ledger records that the shipped champion was promoted on owner preference, unscored, and that the highest judged total still belongs to a candidate that isn't the canonical one. The next run has to beat a judged 40 or my preference, and the ledger says which.
And one where the judge was wrong. It was choosing between two shells for a 3D globe and picked the framework-free one, 43 to 42, from headless screenshots. Ten days later I shipped the other one by override. The rubric never measured live interaction, and the loser runs dramatically better in real use. The ledger records the override and the rubric gap, and adds a measured runtime-performance axis to the next rubric version. I'd picked the same shell myself before I saw the verdict, and that's recorded too, because it says as much about me as about the shell.
A scoring system that admits when the human overruled it, and why, and what it changed as a result, is more credible than one that doesn't.
The Ceiling the Judge Named
In one document ratchet, the judge flagged that every entrant had hand-set numbers that couldn't be checked from inside the document, which a future run could go after with an outside verifier that knows the real values. It didn't claim a clean win. It named its own blind spot and put it in the ledger as the next opening.
That's what I want from a judge: not just a verdict, but the verdict, the reasoning, the loser's flaws as the next run's spec, and the thing the rubric can't see.
What It Costs
Every unit costs two builders, a gate and a judge. On a small repo the margins are often narrow, five points on fifty. I've written elsewhere that I want to see a run where the competition changes the outcome more before I default to it everywhere.
It has clearly paid off on anything visual or structural, where I can't say what I want until I see two versions that differ in their bones. And the ledger has paid for itself on its own: a history of every attempt, what won, what lost, and why. I can read it months later and trust it, because the process wrote it, not me afterward.
-- Justin Higgins. Software Engineer, Midwest. Makes agents compete, records the losses, and lets the judge refuse him.
Companion pieces: Five Hours Unattended - five of these ran in one night. Survived Is Not Endorsed - the review that checks the reviewers.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com