The Winner Was Array Order
I ran a seven-model bake-off for the language pipeline in a small app I built. Seven candidates from an inference router, 40 corpus entries, up to four turns each. 280 sessions, 953 seconds, 43 cents. Three roles on three different models: the pipeline under test, a simulated user, and a judge. The simulator and judge were held out of the candidate list so nothing graded itself.
It produced a ranking. The ranking was almost entirely an artifact. This is about the three ways it was, because each one is a way any eval can be.
The Detector Fired Before Any Model Spoke
The corpus has a leak detector that flags when the pipeline echoes something it should not. Eleven of the 35 recoverable entries tripped that detector on the corpus's own first utterance, before any candidate model had produced a token. The ranking was being decided by which models happened to get handed entries that were pre-flagged.
Fixing it was not "loosen the detector." A loosened detector that no longer fires is worse than the false positives it replaced, because it stops telling you about the real leaks too. The fix was to score the detector against the corpus first, separately, and exclude what it flagged at turn zero from the model comparison.
The Tie Was Sorted
On a subset of cases where the app was wrong on every turn, and the contract said repeat the canonical answer verbatim, all three top candidates produced identical behavior. Byte-identical, because the harness pulls the canonical answer out of a cache instead of letting the model paraphrase. Three identical scores.
The report printed one of them as the winner. It was the first one in the array.
I did not catch that by reading the summary. I caught it by reading the per-session output and noticing that three rows were the same row. A summary that prints a winner will print a winner. It does not know the word tie unless you teach it, and I had not, because I did not think three models would agree on anything.
The Best Recall Was Unusable
Once the artifacts were out, the model that recovered the most intent, 57 percent against the incumbent's 40, was unusable in this app. A 31-second 95th-percentile latency per turn, with 45 percent of turns over the app's timeouts. Another candidate recovered more than the incumbent and missed the latency target by two and a half times and broke the voice contract on 65 percent of turns. The voice contract is the part the user actually hears.
The incumbent won by not losing on the constraints that matter more than the headline metric. That is a boring result and it is the correct one. I had been about to switch models based on a recall number.
The finding that matters more than the ranking: changing models moves the number. The prompt and the schema set the ceiling. Thirty harmful literal reads across 280 sessions, every model three to five, not one clean. A different model was never going to fix that.
Three Green Dashboards
The same app had three separate model-call failures in two weeks, and all three produced green dashboards.
A proxy wrapper around the inference client threw an error the moment it was built, because the client assigns its methods as non-writable properties. The error landed inside a catch-all fallback path, so nothing crashed. The app quietly stopped using its models and served fallbacks on every turn, with empty telemetry.
A token cap tuned for a small model silently returned nothing on a reasoning model, because reasoning models spend the budget on thinking before they write anything. A tight cap gets you silence, not brevity. No crash. Green.
An eval run lost 23 of 40 sessions to an account spending limit and still printed a complete table of percentages. They were computed over the 17 survivors, with nothing on screen saying so.
Three instances of the same failure class. No crash, plausible output, the measurement quietly gone. After the third I stopped fixing numbers and built a seam: the harness now refuses to print a summary table unless the join between sessions requested and sessions scored is complete, and it announces the gap in the output. The next two damaged runs announced themselves. It was the first time that class of failure was caught by machinery rather than by someone noticing afterward.
Observability That Cannot See the User
One more decision from the same app, because it is the opposite instinct from everything above and I think it is the right one.
The telemetry panel reports model calls, tokens, cost, latency, and failures. Nothing else. That line is enforced by a test that serializes the whole payload and fails if it contains an utterance, a session identifier, a caption, or any of the governor's state words. A future addition has to break a test to cross it, not just ignore a comment.
The store is its own database file, opened by a module that imports nothing from the app's data layer. Deleting the file leaves the app untouched. The costs are labeled as estimates everywhere, because the router picks a provider per request and the resolved model is not always the requested one. Telemetry records the resolved model, not the requested one. The first time I checked, leaving the requested model's rate in place would have under-reported spend by about five times.
I am measuring the machine as carefully as I can, and I have made it impossible, by design, to measure the person using it. Both of those are the same principle. The artifact you look at should be the one you are responsible for, and only that one.
-- Justin Higgins. Software Engineer, Midwest. Ran a seven-model bake-off, found the winner was sorted, and built the seam.
Companion pieces: Spider-Man Is Not a Person - the green health check. The Simulated User Is the Confound - the other eval this summer.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com