What the Meter Did Not Count
For two weeks in August I ran two agentic coding tools side by side. GitHub Copilot CLI on a Windows workstation at work. Claude Code on a Mac at home. Both of them write a log about everything they do, one JSON object per line, and I had been reading the Claude Code logs for months. So I read the other one too, and then I put them next to each other.
I expected to write about speed. What I ended up with was a list of things the meter did not count, on both sides, and the part of my own report that did not survive review.
Two Corpora, Not Matched
I should say up front what this is not. It is not a fair benchmark. Twenty-two Copilot sessions against 274 Claude Code session groups. A corporate laptop on a VPN against a personal machine. About 120 dollars of Copilot credit against a 200 dollar a month subscription. A sonnet-class default against an opus-class default. Every comparison in the original draft that depended on a denominator turned out to be a comparison between those mismatches. I will get to what happened to those.
What survives is the stuff you can read off a single log line. A status code. A field that says one thing next to a field that says another. Those have no denominator to get wrong.
The Undercount
The Copilot logs carry a per-model metrics block that is supposed to be the accounting. In the biggest session, a 70-minute rebrand of a React app, that block reported 1.1 million input tokens. The SQLite store the same tool keeps on disk, summing the same session's usage events, reported 34.1 million. The metrics block covered 3.26 percent of what the store said was spent.
Inside the same shutdown event, the total premium-request counter summed to 284 while the per-model request costs summed to 174. Five sessions showed the expensive model sending over a hundred messages while the per-model breakdown reported it at zero or two requests.
Four subagent runs in the corpus went to OpenAI models. Nine and a half million tokens across 221 tool calls, none of it in any per-model metrics block. A fifth subagent was resurrected after it finished and made 87 more tool calls and 38 more messages. No start event, no completion event. Sixty-three percent of that agent's work happened outside the meter entirely.
Every log line in the corpus that names a token limit says 200,000. Six of those sessions had announced a one-million-token context on startup and then compacted against the 200,000 ceiling anyway.
And the one I keep thinking about. In the rebrand session, 95.6 percent of the spend landed in a single 63-minute stretch with no usage checkpoint written. Checkpoints fire on user-turn boundaries, not on spend. The agent ran on one instruction for an hour and reported nothing. There was no moment at which I could have seen it happening.
None of that is a performance finding. It is an accounting finding. A harness can report success on a command that crashed, which fourteen of them did. The hard parser error sits right there in the output, under a success flag. It can advertise a context it does not have. It can route work to another vendor's model and leave it off the books. The interesting differences between these tools were never speed or chattiness. They were what the meter fails to count, and I only know the Copilot side of that because I went looking. I have not done the same dig through my own Claude Code logs, and I should assume they have their own version of this.
The Skill That Fired on Itself
One finding was my fault and I include it because it is the funniest thing in the corpus, and there is not much competition.
I have a skill called prompt-improver. Its job is to catch a human's first prompt and offer to rewrite it. Its trigger condition is "first-turn prompts." Every subagent task is a first-turn prompt. So in the rebrand session, my own skill fired inside all three subagents, within a second of each other. It burned roughly 5,000 tokens and three turns offering to improve prompts that no human was going to read.
The same skill, byte-identical in both harnesses, cost about four times more wall-clock on Copilot than on Claude Code. The reason is an API shape, not a model. Copilot's ask-the-user tool takes exactly one question. Every one of the 64 calls in the corpus carried one. So a three-question menu became four model responses in a row and four waits. Claude Code's equivalent takes up to four questions, so the same menu was one message. Across the corpus, the single-question tool sat waiting for 86.7 minutes total, a median of 23 seconds, a maximum of 28 minutes. I sorted the waits: about 58 percent were load-bearing questions. The rest were the menu being taken one question at a time, permission friction, and questions the agent could have answered itself.
The Review That Killed My Draft
Here is the part I would rather skip.
The first draft of this report had 24 findings. I handed every one of them to an independent reviewer with instructions to refute them, and to default to "refuted" when uncertain.
Twenty-four claims tested. Zero confirmed. Nine refuted. Fifteen overstated.
The claim that Copilot pays a narration tax on 98 percent of tool calls: the field I was counting is a client-side display label, 2,509 of 2,527 of them byte-identical copies of the call's own arguments. Zero output tokens. Section deleted.
The claim that Copilot batches tool calls worse: I had compared Copilot's main thread against Claude Code's main thread plus all its subagents. Main to main, Copilot batches better. Section deleted.
The claim that fixed prompt overhead of 16 to 28 thousand tokens is a tax: the ceiling was a tokenizer artifact. About 2,900 of it was my own instruction files, and time-matched Claude Code sessions carry 42 to 69 thousand. Copilot's maximum was below Claude Code's minimum. Section deleted.
Cache churn, redundant re-reads, self-contradicting system prompt, human prompts inflated five times over. Refuted, a tie, too weak to carry, and 1.18 times once you remove my own skill files.
The reviewer ran out of budget after three of eight dimensions. All three collapsed. I never tested the other five, and I do not get to assume they would have held.
The reason the survivors survived is that they are different in kind. Everything that died was a statistical comparison between two corpora that did not match. Everything that lived is a verbatim read of a single line: an HTTP 402, a token limit of 200,000, a model name inside a subagent completion event, a store reporting 34.1 million where the shutdown event reports 1.1 million. Those have no denominator.
I wrote a rule into the roadmap after that. Do not bring the killed claims back. If a future edit gets punchier by reaching for one of them, it has made the document worse. I wrote it because I know myself. The version of this post that would have gone out without the review was more fun to read and mostly wrong, and I would have enjoyed the comments right up until the second one.
What I Would Tell You to Do
Read your own logs. Not the dashboard, the logs. Sum the usage events yourself and compare them to what the tool reports about itself. Grep for the context ceiling and compare it to what the tool advertised. Look for subagents that started without finishing, or finished without starting. Look for commands that returned success with an error in the body.
Then hand what you found to something with instructions to tear it apart, and expect to lose most of it. The part that survives is the part worth publishing.
-- Justin Higgins. Software Engineer, Midwest. Compared two agent harnesses, lost 24 of 24 findings to review, kept the log lines.
Companion pieces: Everything Looks Like Everything - a stage metric is not a system metric.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com