Comprehension Confidence Is Useless
I built an app whose whole job is to interview a person about what they want built until it has a specification an agent could execute. The mechanic at the center of it is a confidence gauge. The lesson at the center of it is that the obvious way to build that gauge produces a number that is always high and means nothing.
Two Questions That Sound the Same
Ask a language model how confident it is that it understood you, and it will say 0.95. It will say 0.95 every time, on every input, including the ones I wrote to be deliberately vague. It has never once told me it did not understand me, which is more than I can say for most people. I ran a mid-sized open model through near-identical sessions, and it marched from 0.10 to 0.95 on every one of them. The reasons it gave were generic, and it made no difference whether the conversation had settled anything.
Ask it a different question. How confident are you that an agent, given only this transcript, could build the right thing? Now the number behaves. One model held itself around 0.68 and varied across sessions, giving reasons that named specific gaps in the design. A smaller model was more conservative still, around 0.48, and once revised a reading downward mid-interview when it realized a definition was missing.
That second question is build-sufficiency. It is the only usable version of confidence I have found, and the difference is not phrasing. Comprehension confidence measures whether the model can say your idea back to you smoothly, and it always can. Build-sufficiency measures whether the transcript holds enough decided facts to act on. Often it does not, and the model will say so if that is what you asked.
The Low Number Is the Product
There is a residual low confidence for non-technical users that I initially tried to engineer away. Someone who cannot describe a data model will never get the gauge to 0.95, no matter how good the interviewer is. I treated that as a defect for a while.
It is not. It is the honest signal. The gap between what the person can put into words and what a builder needs is real. A gauge that closed it by being generous would be lying about the one thing it exists to measure. The flat march to the ceiling is a contract bug, not a feature.
There was a calibration mistake in the other direction too. An early version capped the gauge at 0.50, which sat below where the interviewer actually settles. So ordinary answers pinned it at 100 percent and the system jumped to the expensive model too fast. That was the single largest cause of "escalates too fast" in the logs. Measuring where a cooperative session really tops out, about 0.58, and setting the ceiling just above that, fixed most of it.
Escalation Improves Fluency, Not Truth
The app escalates through three model tiers as the interview gets harder. I expected the bigger models to be more correct. They were more persuasive, which is different.
A model filling gaps invents plausible specifics. The middle model volunteered a real regulatory rule about expedited review timelines, and it was right. The small model gave a precise uptime figure, that a certain number of nines equals a certain number of minutes per year. It was confidently wrong. Larger models invent more persuasively. That is the whole finding, and it is why the confidence gauge cannot be replaced by "just use the biggest model."
Recognition Over Recall
The mechanic that actually closes the gap is not a better model. It is a confirm-back loop. The interviewer puts its guesses up as statements to accept or correct, not as open questions to answer.
People who cannot write a spec can still judge one specific claim. "You want the report emailed weekly to the requester's manager" is something anyone can say yes or no to. "Describe your notification requirements" is something almost nobody can answer well. The first form turns what the person knows but never says into decided facts, one at a time. The second form asks them to do the builder's job.
That is recognition over recall. Every good analyst I have worked with does it without knowing it has a name. They do not ask what you want. They tell you what they think you want and watch your face.
Why I Care
Twenty years in a large company taught me that most failed projects failed at the spec. The spec failed because someone confident wrote down what they understood rather than what had been decided. The model does the same thing, faster, and reports 0.95 while doing it.
The fix is to ask it the question that has a real answer. Not "do you understand." "Could someone build this from what we have." It will tell you no, and that is the most useful thing it says.
-- Justin Higgins. Software Engineer, Midwest. Built an interviewer and learned to ask it the question with a real answer.
Companion pieces: The Tree I Never Planned to Move - fluency impersonating accuracy. The Corpus Contains Itself - what the interviewer's output built.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com