What the Meter Did Not Count
For two weeks in August I ran two agentic coding tools side by side: GitHub Copilot CLI on a Windows workstation at work, and Claude Code on a Mac at home. Both write a log of everything they do, one JSON object per line, and I'd been reading the Claude Code logs for months. So I read the other one too, and then I put them next to each other.
I expected to write about speed. What I ended up with was a list of things the meter didn't count, on both sides, and the part of my own report that didn't survive review.
Two Corpora, Not Matched
I should say up front that this isn't a fair benchmark. It's twenty-two Copilot sessions against 274 Claude Code session groups, a corporate laptop on a VPN against a personal machine, about 120 dollars of Copilot credit against a 200 dollar a month subscription, and a sonnet-class default against an opus-class default. Every comparison in the original draft that depended on a denominator turned out to be comparing those mismatches. I'll get to what happened to those.
What survives is what you can read off a single log line: a status code, or a field that says one thing next to a field that says another. Those have no denominator to get wrong.
The Undercount
The Copilot logs carry a per-model metrics block that's supposed to be the accounting. In the biggest session, a 70-minute rebrand of a React app, that block reported 1.1 million input tokens. The SQLite store the same tool keeps on disk, summing the same session's usage events, reported 34.1 million. The metrics block covered 3.26 percent of what the store said was spent.
Inside the same shutdown event, the total premium-request counter summed to 284 while the per-model request costs summed to 174. Five sessions showed the expensive model sending over a hundred messages while the per-model breakdown reported it at zero or two requests.
Four subagent runs in the corpus went to OpenAI models: nine and a half million tokens across 221 tool calls, none of it in any per-model metrics block. A fifth subagent was resurrected after it finished and made 87 more tool calls and 38 more messages, with no start event and no completion event. Sixty-three percent of that agent's work happened outside the meter entirely.
Every log line in the corpus that names a token limit says 200,000. Six of those sessions had announced a one-million-token context on startup and then compacted against the 200,000 ceiling anyway.
Then there's the one I keep thinking about. In the rebrand session, 95.6 percent of the spend landed in a single 63-minute stretch with no usage checkpoint written. Checkpoints fire on user-turn boundaries, not on spend. The agent ran on one instruction for an hour and reported nothing, and there was no moment when I could have seen it happening.
These are accounting findings, not performance ones. A harness can report success on a command that crashed, which fourteen of them did, with the hard parser error sitting right there in the output under a success flag. It can advertise a context it doesn't have. It can route work to another vendor's model and leave it off the books. The interesting difference between these tools was what the meter fails to count, never speed or chattiness, and I only know the Copilot side of that because I went looking. I haven't done the same dig through my own Claude Code logs, and I should assume they have their own version of this.
The Skill That Fired on Itself
One finding was my fault, and I include it because it's the funniest thing in the corpus, though there isn't much competition.
I have a skill called prompt-improver that catches a human's first prompt and offers to rewrite it. Its trigger condition is "first-turn prompts," and every subagent task is a first-turn prompt. So in the rebrand session my own skill fired inside all three subagents, within a second of each other, and burned roughly 5,000 tokens and three turns offering to improve prompts no human was going to read.
The same skill, byte-identical in both harnesses, cost about four times more wall-clock on Copilot than on Claude Code. That comes down to an API shape, not a model. Copilot's ask-the-user tool takes exactly one question, and every one of the 64 calls in the corpus carried one, so a three-question menu became four model responses in a row and four waits. Claude Code's equivalent takes up to four questions, so the same menu was one message. Across the corpus, the single-question tool sat waiting for 86.7 minutes total, a median of 23 seconds, a maximum of 28 minutes. I sorted the waits: about 58 percent were load-bearing questions. The rest were the menu being taken one question at a time, permission friction, and questions the agent could have answered itself.
The Review That Killed My Draft
This is the part I'd rather skip. The first draft of this report had 24 findings. I handed every one of them to an independent reviewer with instructions to refute them, and to default to "refuted" when uncertain.
Twenty-four claims tested: zero confirmed, nine refuted, fifteen overstated.
The claim that Copilot pays a narration tax on 98 percent of tool calls: the field I was counting is a client-side display label, 2,509 of 2,527 of them byte-identical copies of the call's own arguments. Zero output tokens. Section deleted.
The claim that Copilot batches tool calls worse: I'd compared Copilot's main thread against Claude Code's main thread plus all its subagents. Main to main, Copilot batches better. Section deleted.
The claim that fixed prompt overhead of 16 to 28 thousand tokens is a tax: the ceiling was a tokenizer artifact. About 2,900 of it was my own instruction files, and time-matched Claude Code sessions carry 42 to 69 thousand. Copilot's maximum was below Claude Code's minimum. Section deleted.
Cache churn, redundant re-reads, a self-contradicting system prompt, human prompts inflated five times over: refuted, a tie, too weak to carry, and 1.18 times once you remove my own skill files.
The reviewer ran out of budget after three of eight dimensions, and all three collapsed. I never tested the other five, and I don't get to assume they'd have held.
The survivors survived because they're a different kind of claim. Everything that died was a statistical comparison between two corpora that didn't match. Everything that lived is a verbatim read of a single line: an HTTP 402, a token limit of 200,000, a model name inside a subagent completion event, a store reporting 34.1 million where the shutdown event reports 1.1 million.
After that I wrote a rule into the roadmap: do not bring the killed claims back. If a future edit gets punchier by reaching for one of them, it's made the document worse. I wrote it because I know myself. The version of this post that would have gone out without the review was more fun to read and mostly wrong, and I'd have enjoyed the comments right up until the second one.
What I Would Tell You to Do
Read your own logs. Not the dashboard, the logs. Sum the usage events yourself and compare them to what the tool reports about itself. Grep for the context ceiling and compare it to what the tool advertised. Look for subagents that started without finishing, or finished without starting. Look for commands that returned success with an error in the body.
Then hand what you found to something with instructions to pick it apart, and expect to lose most of it. What survives is what's worth publishing.
-- Justin Higgins. Software Engineer, Midwest. Compared two agent harnesses, lost 24 of 24 findings to review, kept the log lines.
Companion pieces: Everything Looks Like Everything - a stage metric is not a system metric.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com