The Judge Never Overrides the Regex
In May I built a tool that scores Claude Code sessions. Eleven categories, a composite, and a multiplier band from 2x to 20x. It was meant to answer a question I could not answer by feel: am I actually getting better at working with this thing, or do I just feel busier? Both are possible. Both have happened to me at the same job.
What I want to write about is not the score. It is one design decision. I think it applies to every system that uses a model to grade something, and I have not seen it stated plainly anywhere.
Two Kinds of Question
The eleven categories split cleanly in two once you ask where the signal could possibly come from.
Seven of them are process questions. Which model tiers got used and in what proportion. Whether files were read before they were edited. How often tool calls ran in parallel. Prompt-cache hit rate. Whether a destructive command was run. Whether permissions were skipped. Whether the prompts were structured. All of those are patterns in the log. A regex, a counter, a ratio. Deterministic. Free.
Four of them are content questions. Was the code correct. Was it well designed. Was the design choice defensible. Did it perform. You cannot answer those from patterns. Somebody has to read what was actually done.
So there are three scoring modes. Heuristics only, which fills the four content categories with a placeholder of 2.5. The rationale says, in so many words, we cannot judge this from patterns alone, defaulting to mid. A model judge only, which reads a transcript against a rubric with calibration anchors for every score from zero to five. Without anchors, raters drift to the middle. And a hybrid.
The Rule
In the hybrid, the model judge fills in the four content categories over the placeholders. And that is all it does. It never overrides a heuristic category. It cannot soften a Code Safety penalty that the regex caught. It cannot argue that the destructive command was fine in context. Heuristics own process. The judge owns quality. Each category has exactly one source of truth.
The reason is that the two instruments are not interchangeable. A model judging whether a recursive delete was destructive would be expensive, slow, and add nothing over a pattern match. A regex judging whether the produced code is well designed would be a fantasy. Neither instrument should be allowed to do the other's job. In particular the fluent one should not get to reach into the deterministic one's territory, because fluency is exactly what you do not want near a safety penalty.
The Cap
The composite is a plain average of the enabled categories. An average can be gamed by being excellent elsewhere, so it is not the final word.
If Code Safety or Security Awareness falls below a threshold, the multiplier is capped at 3x no matter what the composite says. If Correctness falls below its threshold, capped at 5x. A perfect 4.8 composite with a basement safety score does not get the 20x band. You do not reward someone for being mostly excellent while doing destructive things.
That cap is the second half of the same idea. The judge cannot soften the safety score, and the composite cannot average it away. The safety signal has one source and one path to the output, and nothing fluent sits on that path.
The Part That Grades Me
Here is what I did not fully register when I built it. The process categories are not grading the model. They are grading me.
Cache hit rate is a measure of whether I keep a stable prefix or churn my prompts. Read-before-edit is whether I let it look before I let it touch. Parallel rate is whether I batch or serialize. Skipping permissions costs 0.7 per occurrence. Model selection scores highest when at least two tiers are present and no single tier dominates. That is a judgment about how I dispatch, not about the model's output.
I built a discipline meter for the operator, and the operator is me, and it can cap my own score. I do not think I would have designed it that way if I had started from "how do I make the number go up," which is how most of these things get designed.
What It Does Not Do Well
The judge samples the top three sessions by event count, which is a deliberate choice and not a random one. Three deep sessions tell you more than thirty trivial ones, but it leans toward heavy work. Transcripts are truncated to a character budget, so a twelve-hour session gets summarized into roughly its first 60,000 characters of stylized text. That is a real fidelity tradeoff and I have not solved it.
And the score wobbles. Run the judge twice and expect it to move by about 0.3. There is also self-flattery. Claude grading Claude tends to be generous. Averaging across sessions and bounding the multiplier dampens both, but does not remove them.
Here is the idea I keep circling and have not yet measured well enough to publish. A model grading its own work over-reports on what was intended and under-reports on what the artifact actually shows. It reads the transcript, sees the plan, and grades the plan. The code is what shipped. I built the split between heuristics and judge partly on that suspicion. The split holds up even if the suspicion turns out to be only half right, because it never let the judge near the numbers that matter most anyway.
The General Form
If you are building anything that uses a language model as a grader, I would put it this way. Partition the questions by where the signal lives. Give each category exactly one instrument. Never let the fluent instrument override the deterministic one. And put a cap on the output that the fluent instrument cannot reach.
The model is very good at the questions only a reader can answer. It should not be in the room for the ones a counter can.
-- Justin Higgins. Software Engineer, Midwest. Built a scorer for his own agent sessions and kept the regex in charge of safety.
Companion pieces: The Work Looks Like Nothing - what the sessions record. Spider-Man Is Not a Person - a score that is not a confidence.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com