The Judge Never Overrides the Regex
Correction, 2026-10-08. In September I calibrated this scorer against forty recent sessions and found parts of it were measuring the log format instead of the work. It counted 275 sessions in those forty, because every subagent transcript looked like a session of its own. The rule this post is about did not change. Three things it describes were wrong:
- The permissions-skip penalty never fired. It looked for a command-line flag that never shows up in the logs, while the logs recorded 1,324 bypass events. It now reads bypass mode from what the logs do record.
- The safety cap fired on a false positive. The secret check matched ordinary variable names in code I had written, zeroed Security Awareness, and capped the whole corpus at 3x. The cap did exactly what it was built to do, on a wrong input. The fix was the check, not letting the judge overrule it. With that and the rest of the recalibration, the composite went from 2.76 capped at 3x to 3.20 at 8x.
- Model Selection's top score came from the subagents. The session model is a fixed choice, so the tier mix it rewarded was the subagents' mix, not mine. It now scores how I dispatch work.
The judge changed too. The text below is struck through where it no longer holds.
In May I built a tool that scores Claude Code sessions: eleven categories, a composite, and a multiplier band from 2x to 20x. It was meant to answer a question I couldn't answer by feel. Am I actually getting better at working with this thing, or do I just feel busier? Both are possible, and both have happened to me at the same job.
What I want to write about is one design decision, not the score. I think it applies to every system that uses a model to grade something, and I haven't seen it stated plainly anywhere.
Two Kinds of Question
The eleven categories split cleanly in two once you ask where the signal could possibly come from.
Seven of them are process questions: which model tiers got used and in what proportion, whether files were read before they were edited, how often tool calls ran in parallel, prompt-cache hit rate, whether a destructive command was run, whether permissions were skipped, and whether the prompts were structured. All of those are patterns in the log. A regex, a counter or a ratio answers them, deterministically and for free.
Four of them are content questions. Was the code correct? Was it well designed? Was the design choice defensible? Did it perform? You can't answer those from patterns. Somebody has to read what was actually done.
So there are three scoring modes. Heuristics only, which fills the four content categories with a placeholder of 2.5. The rationale says, in so many words, we cannot judge this from patterns alone, defaulting to mid. Model judge only, which reads a transcript against a rubric with calibration anchors for every score from zero to five, because without anchors, raters drift to the middle. And there's a hybrid.
The Rule
In the hybrid, the model judge fills in the four content categories over the placeholders, and that's all it does. It never overrides a heuristic category. It can't soften a Code Safety penalty the regex caught, and it can't argue that the destructive command was fine in context. Heuristics own process, the judge owns quality, and each category has exactly one source of truth.
The two instruments aren't interchangeable. A model judging whether a recursive delete was destructive would be expensive and slow, and it would add nothing over a pattern match. A regex judging whether the code is well designed would be a fantasy. Neither should do the other's job, and the fluent one especially shouldn't reach into the deterministic one's territory, because fluency is exactly what you don't want near a safety penalty.
The Cap
The composite is a plain average of the enabled categories. An average can be gamed by being excellent elsewhere, so it isn't the final word.
If Code Safety or Security Awareness falls below a threshold, the multiplier is capped at 3x no matter what the composite says. If Correctness falls below its threshold, it's capped at 5x. A perfect 4.8 composite with a basement safety score doesn't get the 20x band. You don't reward someone for being mostly excellent while doing destructive things.
The cap is the second half of the same idea. The judge can't soften the safety score, and the composite can't average it away. The safety signal has one source and one path to the output, and nothing fluent sits on that path.
The Part That Grades Me
When I built it, I didn't fully register that the process categories grade me, not the model.
Cache hit rate is a measure of whether I keep a stable prefix or churn my prompts. Read-before-edit is whether I let it look before I let it touch. Parallel rate is whether I batch or serialize. Skipping permissions costs 0.7 per occurrence. Skipping permissions costs points (corrected 2026-10-08: the original check never fired). Model selection scores highest when at least two tiers are present and no single tier dominates. Model selection scores how I dispatch work to subagents (corrected 2026-10-08: the tier-mix rule was rewarding the subagents' mix, not mine). That is a judgment about how I dispatch, not about the model's output.
I built a discipline meter for the operator, and the operator is me, and it can cap my own score. I don't think I'd have designed it that way if I'd started from "how do I make the number go up," which is how most of these things get designed.
What It Does Not Do Well
The judge samples the top three sessions by event count, which is a deliberate choice and not a random one. Three deep sessions tell you more than thirty trivial ones, but it leans toward heavy work. The judge now samples sessions across the range of sizes (updated 2026-10-08). Transcripts are truncated to a character budget, so a twelve-hour session gets summarized into roughly its first 60,000 characters its beginning and its end (updated 2026-10-08) of stylized text. That is a real fidelity tradeoff and I have not solved it.
And the score wobbles. Run the judge twice and expect it to move by about 0.3. There's also self-flattery, since Claude grading Claude tends to be generous. Averaging across sessions and bounding the multiplier dampens both but doesn't remove them.
There's an idea I keep circling and haven't yet measured well enough to publish. A model grading its own work over-reports on what was intended and under-reports on what the artifact actually shows. It reads the transcript, sees the plan, and grades the plan, but the code is what shipped. I built the split between heuristics and judge partly on that suspicion. The split holds up even if the suspicion is only half right, because it never let the judge near the numbers that matter most anyway.
The General Form
If you're building anything that uses a language model as a grader, I'd put it this way. Split the questions by where the signal lives. Give each category exactly one instrument. Never let the fluent instrument override the deterministic one. And put a cap on the output that the fluent instrument can't reach.
The model is very good at the questions only a reader can answer. It shouldn't be in the room for the ones a counter can.
-- Justin Higgins. Software Engineer, Midwest. Built a scorer for his own agent sessions and kept the regex in charge of safety.
Companion pieces: The Work Looks Like Nothing - what the sessions record. Spider-Man Is Not a Person - a score that is not a confidence.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com