The Simulated User Is the Confound
I built a diagnostic tool for heavy equipment. An operator describes a problem, the tool interviews them, and it produces a ranked list of causes with the evidence for each. The interesting engineering is not the report. It is whether the interview asks the questions that tell look-alike causes apart. The only way to measure that is to run the interview against an operator.
I do not have a fleet of operators. I have a language model playing one. It took me an embarrassing number of runs to understand that the model playing the operator was the thing I was measuring. I will tell you the number. It was more than sixty.
Ten Out of Ten, Then Eight
The first eval had ten scenarios. Each has one hidden true cause whose symptoms look like three or more other causes, and only the right question separates them. The control prompt scored 10 of 10.
I re-ran the identical prompt. It scored 8 of 10.
Three of the ten cases were coin flips. Nothing had changed except the dice. I had already declared two prompt variants failures on 6 and 7 out of 10 before I understood that, and both verdicts had to be thrown out. The rule I wrote afterward: a single-run accuracy delta of two or fewer cases is noise, and a finding needs four. Later, with more cases, I did the arithmetic properly. At an accuracy around 85 percent, the 95 percent band on a difference between two arms is about 29 points at twelve cases per arm, about 16 at forty, still about 9 at 120. Chasing a five-point effect by growing the set is not a plan. Pairing cases and screening out the ones every arm gets right is.
Four of the six cases in one set were correct in every single run. Reporting 10 of 12 versus 11 of 12 implies twelve observations. There were two.
The Chips
The interview offers multiple-choice chips to speed things up. For one scenario the chips were "Same both sides," "Right looser," "Left looser," "Haven't checked." The ground truth is that the right track has no slack and the left gives two inches. The correct chip, "Left looser," was on the list.
The simulated operator picked "Right looser."
The model then reasoned correctly from a false premise and produced a confident wrong diagnosis. Fifteen runs, one correct. And the simulator had answered truthfully, in a sense. A question shaped "which track is looser" has an honest answer, and the operator gave the one it thought was being asked about. Flipping the fact back in the scenario sheet cannot fix this. The fact was already right.
I ran an ablation. Same app, same prompts, chips hidden from the simulated operator only, so it had to answer in free text. Two scenarios flipped from wrong in both chip replicates to right in both free-text replicates. Pooled, 8 of 12 became 11 of 12.
So a large part of the headline accuracy on those cases was not measuring how hard the diagnosis was. It was measuring whether a left-or-right fact survived a single-select widget being read by a model. Rank-one accuracy was 78 percent when the abnormality actually reached the diagnostic model and 3 percent when it did not.
Pre-Registration Found a Broken Scenario
Every scenario in the code-bearing set carries a block written before the first run: the correct top answer, the names that also count, the names scored wrong, and the behavioral checks. That block is what exposed a scenario I would otherwise have kept.
Across 61 runs of one scenario, 56 top-ranked answers, 92 percent, named a cause that appeared nowhere in its pre-registered block. Fifty named a sensor affected by cold and condensation. Five named an air leak. No discriminator had ever been written to argue either down, because nobody had expected them. The four alternatives that were written were chosen a combined one time in 61. And the scenario's only real discriminator, a mechanical fact the operator could check by hand, was delivered in zero of 61 interviews.
The judge was scoring an answer the pre-registration never anticipated. That is not a hard scenario. That is a broken one, and the repair is keyed to questions the banked interviews measurably do ask: a moisture-independence fact was asked for in 56 percent of runs, and the scenario sheet had no answer for it, so the simulated operator improvised one. Now it has an answer.
Two adjacent scenarios were triaged at the same time and deliberately not touched. One is hard. One is an instrument leak. Editing either would destroy a good case, and the temptation to fix a case that scores badly is exactly the temptation the pre-registration exists to resist.
Three Things at Once
Every one of those runs was measuring three things at once: the diagnostic model, the interview design, and the simulator's answering behavior. The simulator dominated more often than not, and it dominated in ways that looked like model weakness.
I do not have a clean way out. A real operator would be better and I cannot afford a hundred of them. What I have instead is a list of rules I now apply to every eval with a simulated party in it. Replicate before you believe a delta. Pair cases across arms. Screen out cases every arm gets right. Hide the buttons from the simulator and see what moves. Write the expected answer before the first run, and when 92 percent of answers fall outside it, fix the scenario, not the judge.
And say, in the README, that the run directories are not in the repository. They are large, tied to the account that paid for them, and not something a reader can meaningfully diff. The scenarios, the harness, and the scoring code all ship. Anyone can make their own. That is the part that has to be true for any of the numbers above to mean anything.
-- Justin Higgins. Software Engineer, Midwest. Built an eval, discovered it was measuring the fake operator, and kept the broken scenario's repair keyed to what the interviews actually ask.
Companion pieces: The Winner Was Array Order - the other eval this summer. Everything Looks Like Everything - a stage metric is not a system metric.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com