Three in Five Opus Runs Did Not Need Opus
I run almost everything on the biggest model. I had a feeling that was wasteful, the way you have a feeling about a gym membership, and I had no evidence, so I built the evidence.
The Audit
I took every Claude Code session file touched in one week: 1,899 raw log files, of which 1,647 were subagent transcripts and excluded, and 78 were empty. That left 170 real top-level sessions across 21 project directories.
Eighteen reviewer agents read those sessions blind to which model had actually run them. Each was asked one question: what's the minimum model that could have done this session's work to the same standard? The brief told them not to rubber-stamp the biggest model just because it's what ran. The point was to find where it wasn't load-bearing.
About 95 percent of the sessions had run on Opus.
The reviewers' verdict: 34 sessions, 20 percent, were Haiku-sufficient. 70 sessions, 41 percent, were Sonnet-appropriate. 65 sessions, 38 percent, needed Opus.
Of the 34 Haiku-sufficient ones, 27 were literal no-ops: a clear command, an exit, a directory change, an effort setting. They never called a model at all, so I won't count them as savings. But the 70 in the middle are real. Two in five sessions did work a cheaper model would have done as well, on a model priced for judgment.
What Did Not Predict It
I expected session length and tool-call volume to be the signal, and they aren't. Several of the week's longest, highest-tool-count sessions were just working through a spec that was already detailed. Every hard call had been made before the session started, in a screenshot or a document or a previous conversation, and the model was just typing fast. Typing fast is a Sonnet job.
Rework was a weak signal on its own too. A lot of the sessions with a correction in them were friction from process or from talking past each other, not a reasoning failure. "You ran it without telling me." An unauthorized deploy. A misread instruction. A logo I didn't like. A bigger model doesn't fix any of those.
What Did
The most common reason the reviewers gave for Opus was that the work couldn't be undone: production data, credential rotation, live DNS, a multi-tenant migration, git history reconciliation under an instruction not to lose anything. A mistake in those sessions isn't a re-prompt, it's an incident, and that's where the expensive judgment pays for itself.
The second was that a decision was genuinely open: choosing between architectures, deciding what to leave out, reading a failure and deciding whether it was a bug or a signal.
That gives me the check I now run before hitting enter. Is there a decision in here that isn't already made? If the prompt, the screenshot, or the spec already settles the hard call and this is just typing, default to the middle tier. If the middle tier comes back wrong or incomplete, that's the signal to move up, not a sign I should have started at the top everywhere just in case.
What I Changed
My global instructions now carry a tier map. Pure extraction, file lists, grep-and-quote, counting: the small model. Reading and fetching, repo exploration, doc crawls, first drafts of boilerplate: the middle model. Judging, code review, scoring against a ledger, verification, test-failure triage: the big model. The session-tier model is kept for the main conversation, and a subagent never inherits it by default.
Every agent dispatch now opens with one line stating the model, the count and why, so a step off the map is visible instead of buried.
I want to be honest about the shape of the evidence. Eighteen reviewer agents judging sessions is eighteen model opinions about model necessity, and I don't treat a model's opinion of a model's work as an oracle. The reviewers could all be tilted the same way, in either direction. What I trust more than the percentages is the pattern in the reasons they gave, because it matches what I can check by hand. The sessions flagged as Opus-needed are the ones where I remember being nervous.
The Number That Matters
I don't know exactly what the week cost. I know roughly what the middle-tier sessions would have cost on the middle tier, and the difference is real money for one person paying out of pocket. But the money is the smaller point.
The larger one is that "use the best model for everything" was a decision I made once, by default, and never re-examined, because it never produced a visible failure. It produced an invisible cost, spread across a hundred sessions, that only showed up when I made something read all of them.
-- Justin Higgins. Software Engineer, Midwest. Had 18 agents read a week of his sessions and tell him which ones were overkill.
Companion pieces: The Judge Never Overrides the Regex - model selection as a scored discipline.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com