The Winner Was Array Order
I ran a seven-model bake-off for the language pipeline in a small app I built: seven candidates from an inference router, 40 corpus entries, up to four turns each. That came to 280 sessions, 953 seconds and 43 cents. Three roles ran on three different models: the pipeline under test, a simulated user, and a judge. The simulator and judge were held out of the candidate list so nothing graded itself.
It produced a ranking, and the ranking was almost entirely an artifact. This is about the three ways it was, because each one is a way any eval can be.
The Detector Fired Before Any Model Spoke
The corpus has a leak detector that flags when the pipeline echoes something it shouldn't. Eleven of the 35 recoverable entries tripped that detector on the corpus's own first utterance, before any candidate model had produced a token. The ranking was being decided by which models happened to get handed entries that were pre-flagged.
Fixing it wasn't "loosen the detector." A loosened detector that no longer fires is worse than the false positives it replaced, because it stops telling you about the real leaks too. The fix was to score the detector against the corpus first, separately, and exclude what it flagged at turn zero from the model comparison.
The Tie Was Sorted
On a subset of cases where the app was wrong on every turn, and the contract said repeat the canonical answer verbatim, all three top candidates produced identical behavior. It was byte-identical, because the harness pulls the canonical answer out of a cache instead of letting the model paraphrase. Three identical scores.
The report printed one of them as the winner. It was the first one in the array.
I didn't catch that by reading the summary. I caught it by reading the per-session output and noticing that three rows were the same row. A summary that prints a winner will print a winner. It doesn't know the word tie unless you teach it, and I hadn't, because I didn't think three models would agree on anything.
The Best Recall Was Unusable
Once the artifacts were out, the model that recovered the most intent, 57 percent against the incumbent's 40, was unusable in this app. It had a 31-second 95th-percentile latency per turn, with 45 percent of turns over the app's timeouts. Another candidate recovered more than the incumbent but missed the latency target by two and a half times and broke the voice contract on 65 percent of turns. The voice contract is the part the user actually hears.
The incumbent won by not losing on the constraints that matter more than the headline metric. That's a boring result and it's the correct one. I'd been about to switch models based on a recall number.
The finding that matters more than the ranking is that changing models moves the number, but the prompt and the schema set the ceiling. There were thirty harmful literal reads across 280 sessions, three to five on every model, and not one model came out clean. A different model was never going to fix that.
Three Green Dashboards
The same app had three separate model-call failures in two weeks, and all three produced green dashboards.
A proxy wrapper around the inference client threw an error the moment it was built, because the client assigns its methods as non-writable properties. The error landed inside a catch-all fallback path, so nothing crashed. The app quietly stopped using its models and served fallbacks on every turn, with empty telemetry.
A token cap tuned for a small model silently returned nothing on a reasoning model, because reasoning models spend the budget on thinking before they write anything. A tight cap gets you silence, not brevity. No crash. Green.
An eval run lost 23 of 40 sessions to an account spending limit and still printed a complete table of percentages, computed over the 17 survivors with nothing on screen saying so.
That's three instances of the same failure class: no crash, plausible output, and the measurement quietly gone. After the third I stopped fixing numbers and built a seam. The harness now refuses to print a summary table unless the join between sessions requested and sessions scored is complete, and it announces the gap in the output. The next two damaged runs announced themselves. It was the first time that class of failure was caught by machinery rather than by someone noticing afterward.
Observability That Cannot See the User
One more decision from the same app, because it's the opposite instinct from everything above and I think it's the right one.
The telemetry panel reports model calls, tokens, cost, latency, and failures, and nothing else. That line is enforced by a test that serializes the whole payload and fails if it contains an utterance, a session identifier, a caption, or any of the governor's state words. A future addition has to break a test to cross it, not just ignore a comment.
The store is its own database file, opened by a module that imports nothing from the app's data layer, so deleting the file leaves the app untouched. The costs are labeled as estimates everywhere, because the router picks a provider per request and the resolved model isn't always the requested one. Telemetry records the resolved model. The first time I checked, leaving the requested model's rate in place would have under-reported spend by about five times.
I'm measuring the machine as carefully as I can, and I've made it impossible, by design, to measure the person using it. Both come from the same principle: the artifact you look at should be the one you're responsible for, and only that one.
-- Justin Higgins. Software Engineer, Midwest. Ran a seven-model bake-off, found the winner was sorted, and built the seam.
Companion pieces: Spider-Man Is Not a Person - the green health check. The Simulated User Is the Confound - the other eval this summer.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com