Three in Five Opus Runs Did Not Need Opus
I run almost everything on the biggest model. I had a feeling that was wasteful, in the way you have a feeling about a gym membership, and I had no evidence, so I built the evidence.
The Audit
Every Claude Code session file touched in one week. 1,899 raw log files, of which 1,647 were subagent transcripts and excluded, and 78 were empty. That left 170 real top-level sessions across 21 project directories.
Eighteen reviewer agents read those sessions blind to which model had actually run them. Each was asked one question: what is the minimum model that could have done this session's work to the same standard? The brief told them not to rubber-stamp the biggest model just because it is what ran. The point was to find where it was not load-bearing.
About 95 percent of the sessions had run on Opus.
The reviewers' verdict: 34 sessions, 20 percent, were Haiku-sufficient. 70 sessions, 41 percent, were Sonnet-appropriate. 65 sessions, 38 percent, needed Opus.
Of the 34 Haiku-sufficient ones, 27 were literal no-ops. A clear command, an exit, a directory change, an effort setting. They never called a model at all, so I will not count them as savings. But the 70 in the middle are real. Two in five sessions did work that a cheaper model would have done as well, on a model priced for judgment.
What Did Not Predict It
Session length and tool-call volume are not the signal. I expected them to be. Several of the week's longest, highest-tool-count sessions were just working through a spec that was already detailed. Every hard call had been made before the session started, in a screenshot or a document or a previous conversation, and the model was just typing fast. Typing fast is a Sonnet job.
Rework was a weak signal on its own too. A lot of the sessions with a correction in them were friction from process or from talking past each other, not a reasoning failure. "You ran it without telling me." An unauthorized deploy. A misread instruction. A logo I did not like. None of those are fixed by a bigger model.
What Did
The most common reason the reviewers gave for Opus was that the work could not be undone. Production data. Credential rotation. Live DNS. A multi-tenant migration. Git history reconciliation under an instruction not to lose anything. A mistake in those sessions is not a re-prompt. It is an incident. That is where the expensive judgment pays for itself.
The second was that a decision was genuinely open. Choosing between architectures. Deciding what to leave out. Reading a failure and deciding whether it was a bug or a signal.
Which gives the check I now run before hitting enter. Is there a decision in here that is not already made? If the prompt, the screenshot, or the spec already settles the hard call and this is just typing, default to the middle tier. If the middle tier comes back wrong or incomplete, that is the signal to move up. It is not a sign I should have started at the top everywhere just in case.
What I Changed
My global instructions now carry a tier map. Pure extraction, file lists, grep-and-quote, counting: the small model. Reading and fetching, repo exploration, doc crawls, first drafts of boilerplate: the middle model. Judging, code review, scoring against a ledger, verification, test-failure triage: the big model. The session-tier model is kept for the main conversation, and a subagent never inherits it by default.
Every agent dispatch now opens with one line stating the model and the count and why, so a step off the map is visible rather than buried.
I want to be honest about the shape of the evidence. Eighteen reviewer agents judging sessions is eighteen model opinions about model necessity, and I do not treat a model's opinion of a model's work as an oracle. The reviewers could all be tilted the same way, in either direction. What I trust more than the percentages is the pattern in the reasons they gave, because the pattern matches what I can check by hand. The sessions flagged as Opus-needed are the ones where I remember being nervous.
The Number That Matters
I do not know exactly what the week cost. I know roughly what the middle-tier sessions would have cost on the middle tier, and the difference is real money for one person paying out of pocket. But the money is the smaller point.
The larger one is that "use the best model for everything" is a decision I made once, by default, and never re-examined, because it never produced a visible failure. It produced an invisible cost, spread across a hundred sessions, that only showed up when I made something read all of them.
-- Justin Higgins. Software Engineer, Midwest. Had 18 agents read a week of his sessions and tell him which ones were overkill.
Companion pieces: The Judge Never Overrides the Regex - model selection as a scored discipline.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com