Comprehension Confidence Is Useless
I built an app whose whole job is to interview a person about what they want built until it has a specification an agent could execute. At its center is a confidence gauge, and the lesson I took from it is that the obvious way to build that gauge produces a number that's always high and means nothing.
Two Questions That Sound the Same
Ask a language model how confident it is that it understood you, and it'll say 0.95. It'll say 0.95 every time, on every input, including the ones I wrote to be deliberately vague. It has never once told me it didn't understand me, which is more than I can say for most people. I ran a mid-sized open model through near-identical sessions, and it marched from 0.10 to 0.95 on every one of them, giving generic reasons whether or not the conversation had settled anything.
Now ask it a different question: how confident are you that an agent, given only this transcript, could build the right thing? The number behaves. One model held around 0.68 and varied across sessions, giving reasons that named specific gaps in the design. A smaller model was more conservative still, around 0.48, and once revised a reading downward mid-interview when it realized a definition was missing.
That second question is build-sufficiency. It's the only usable version of confidence I've found, and the difference is more than phrasing. Comprehension confidence measures whether the model can say your idea back to you smoothly, and it always can. Build-sufficiency measures whether the transcript holds enough decided facts to act on. Often it doesn't, and the model will say so if that's what you asked.
The Low Number Is the Product
Non-technical users are left with a residual low confidence that I first tried to engineer away. Someone who can't describe a data model will never get the gauge to 0.95, no matter how good the interviewer is. For a while I treated that as a defect, but it's the honest signal. The gap between what the person can put into words and what a builder needs is real, and a gauge that closed it by being generous would be lying about the one thing it exists to measure. The flat march to the ceiling is a contract bug, not a feature.
There was a calibration mistake in the other direction too. An early version capped the gauge at 0.50, below where the interviewer actually settles, so ordinary answers pinned it at 100 percent and the system jumped to the expensive model too fast. That was the single largest cause of "escalates too fast" in the logs. Measuring where a cooperative session really tops out, about 0.58, and setting the ceiling just above that fixed most of it.
Escalation Improves Fluency, Not Truth
The app escalates through three model tiers as the interview gets harder. I expected the bigger models to be more correct. They were more persuasive, which is different.
A model filling gaps invents plausible specifics. The middle model volunteered a real regulatory rule about expedited review timelines, and it was right. The small model gave a precise uptime figure, that a certain number of nines equals a certain number of minutes per year, and it was confidently wrong. Larger models invent more persuasively. That's the whole finding, and it's why "just use the biggest model" can't replace the confidence gauge.
Recognition Over Recall
What actually closes the gap is a confirm-back loop, not a better model. The interviewer puts its guesses up as statements to accept or correct, not as open questions to answer.
People who can't write a spec can still judge one specific claim. Anyone can say yes or no to "You want the report emailed weekly to the requester's manager." Almost nobody can answer "Describe your notification requirements" well. The first form turns what the person knows but never says into decided facts, one at a time. The second asks them to do the builder's job.
That's recognition over recall. Every good analyst I've worked with does it without knowing it has a name. They don't ask what you want. They tell you what they think you want and watch your face.
Why I Care
Twenty years in a large company taught me that most failed projects failed at the spec, because someone confident wrote down what they understood rather than what had been decided. The model does the same thing, faster, and reports 0.95 while doing it.
The fix is to ask it the question that has a real answer: "Could someone build this from what we have," not "do you understand." It'll tell you no, and that's the most useful thing it says.
-- Justin Higgins. Software Engineer, Midwest. Built an interviewer and learned to ask it the question with a real answer.
Companion pieces: The Tree I Never Planned to Move - fluency impersonating accuracy. The Corpus Contains Itself - what the interviewer's output built.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com