Everything Looks Like Everything
The demo everybody builds is retrieval over a diverse corpus. A wiki, a support archive, a folder of PDFs about unrelated subjects. You ask a question, the embedding finds the three chunks about that thing, a model reads them, you get an answer with citations. It works on the first try, which should have been the tell.
I spent the summer on the other case. One thousand and eight insurance contracts. Synthetic, generated to be exactly as redundant as the real ones. Same carrier. Same benefit categories. Same section headings in the same order. Deductibles, out-of-pocket maximums, coinsurance, network tiers. Every document contains all of it, and every document says something slightly different.
Ask that corpus what the individual deductible is and the only honest response is a question. Whose? For which plan year? Under the summary plan description, or the amendment that superseded part of it in March?
Retrieval over a diverse corpus is a search problem. Retrieval over a redundant one is a problem of telling things apart, wearing a search problem's clothes. Almost nothing written about RAG is about the second case. Inside a regulated enterprise, the second case is nearly all of the work.
I picked insurance contracts because I have spent two decades near them and I know exactly how they misbehave. That turned out to matter more than any modeling decision I made afterward.
Everything Matches
The first honest measurement was 3.5%.
A number that low usually means something is unplugged. Nothing was. The embeddings did exactly what embeddings do. Asked about an out-of-pocket maximum, they returned a fistful of passages about out-of-pocket maximums, ranked by how closely each one resembled the question. All of them resembled the question. That is what redundancy means. The vector index had no way to say the only thing that mattered. One thousand and seven of those documents belonged to somebody else.
I had seen this failure before in a different costume. Ranking candidates by similarity to a job description has the same shape: when everyone uses the same nouns, similarity stops carrying information and starts laundering it. Resemblance is a weak signal when the documents differ and a worthless one when they do not. I wrote a whole essay about that without noticing I was describing a retrieval bug.
The fix was not a better model. It was working out who the question was about before searching for an answer. Pull the business name, plan ID, or group number out of the query, match it against the record index, narrow to that record's documents, and only then retrieve. Same embeddings. Same reranker. Same everything downstream.
3.5% to 99.8%.
The lesson is unflattering in the way real lessons are. The hard part was never the retrieval stack. The hard part was noticing that the question had a subject and that nothing in the pipeline had bothered to ask which one.
The Instrument Was Measuring Nothing
Then the part I would rather not include.
A tuning run came back with 0.998 document recall. Excellent by any published standard. I moved on, wired the answer path, and ran it end to end against real questions.
It answered one percent of them.
Retrieval was finding the right documents and the system was returning nothing. The recall number was true and told me nothing, because the evaluation harness called the retriever directly, without the scoping step. It fuzzy-matched every query against all thousand near-identical contracts and graded itself on a pipeline nobody actually ran. The single change that took the corpus from unusable to usable was the one the measurement skipped.
Four fixes later it answered 98% and abstained correctly on 93%. But the number I keep is the first one. A stage metric is not a system metric. I knew that before I started. Knowing it did not help, because the harness never lied to me. It answered exactly the question I had asked it, and I had asked the wrong one.
It also turned out that three separate callers had each written their own version of that scoping step, and each had quietly dropped a different piece of it. Three independent authors, all of them me, all of them convinced it was two lines of glue. It lives in one exported function now, and everything takes it whole.
This is the second time one of my own instruments has told me something flattering that turned out to be a statement about the instrument. I have stopped treating that as bad luck.
The Product Is the Refusal
Here is the thing I did not expect to be the center of the project. It took me a while to stop treating it as the boring part of the build.
A contract system that answers nearly every time and is confidently wrong on the rest is worse than no system, because from the outside you cannot tell the wrong answers from the right ones. The value is not in the answer rate. It is in whether the thing knows the difference between having evidence and having text.
So: a gate. Evidence gets scored on authority and recency before anything is generated. A certificate of coverage outranks a benefit summary. An amendment outranks the document it amends. Both outrank a marketing one-pager that happens to use the same words. Then the passages that survive have to actually agree with each other. If they do not line up, the system does not answer, and it does not hedge. It names the reason:
Divergent. The retrieved clauses give different figures and nothing in the file breaks the tie.
Stale. The best-supported answer comes from a document already superseded by another one in the same file.
Conflicting. Two sources of equal authority disagree on their face.
Below authority. The only support comes from a document not entitled to settle the question.
That list is the product. The answers are table stakes. Anything with an API key produces answers. Every one of those four is a sentence a competent analyst would say out loud in a meeting. Not one of them is something a language model says unless you build the machinery that forces it to. Left alone, it will produce a fluent, well-cited number from a superseded amendment and never mention that it was superseded.
Getting the last family right took a while. The system was refusing those cases correctly but blaming the refusal on the model's own reluctance rather than on the missing evidence. Right behavior, wrong reason on the record, which is its own kind of failure. Fixing it moved that family from zero out of fifteen to fifteen out of fifteen in both harnesses, with the answerable set unchanged. Measured both ways. Not assumed.
The Half with No SKU
An abstention is an absence, and absences do not get credit.
I have spent twenty years producing that particular absence. The migration scoped down before it became a decade of carrying cost. The vendor trap that does not close. The confident number nobody ends up acting on. None of it lands on a balance sheet, and the reason it stays unpriced was never that it cannot be measured - it is that it had nowhere to be recorded.
The difference here is that this one leaves a record. The refusal ships with its citations attached. You can open the two documents that disagreed, see which one carried more authority, see the effective dates, see why the tie could not be broken. The judgment is not just claimed. It is shown, with its inputs, in a form somebody else can check and overturn.
I did not set out to build a working model of my own career. It is inconvenient how cleanly it landed. The system's most valuable behavior is the one that produces no output. The entire engineering effort went into making that behavior readable enough to be worth something.
What to Do with It
The repository is proprietary today. Every right reserved, private manifest, the whole posture. I have been rethinking that, and the useful question turned out not to be whether the code is good enough. It was: what here is actually scarce?
Not the retrieval stack. Hybrid dense-plus-BM25 with a cross-encoder rerank is a recipe now, and mine is a competent version of something a few thousand people have built.
What is scarce is the corpus. A thousand documents built to be nearly identical, each tagged with an authority tier and effective dates. Plus the generator that makes more of them, and a graded evaluation set that includes the cases where the correct answer is a refusal. I cannot find a public equivalent. Every retrieval benchmark I know of assumes documents that differ from each other, which is the easy case and not the one anybody's contract file resembles.
The honest read on the rest is that nobody is going to run it. It needs Postgres with pgvector, a reranker endpoint, and a deploy shaped like mine. Adoption isn't the return. Being read is, and the interesting part of this project is entirely visible from the outside: the eval numbers, the decision log, the commit where the measurement turned out to be measuring nothing.
I haven't decided yet.
The Label
One more thing happened, and if I could keep only one it would be this.
In production, someone asked for an individual out-of-pocket maximum for a specific business. Correctly scoped, resolved to exactly one record, evidence retrieved and cited. And the cited evidence contained the row | Out-of-Pocket Maximum - Individual / Family | $8,000 / $18,400 |. The system returned its refusal string. The badge on the response read claude-sonnet-4-6.
The cause was that production had no Anthropic key. The gateway had silently fallen back to the deterministic stub used for offline testing, whose canned reply mentions its own token limit: 512. The attribution guard saw a figure in the answer that appeared in none of the cited passages. It ruled that the number could not be traced to a source and replaced the whole thing with the refusal.
So every piece of judgment in the system worked. Retrieval was right. Scope was right. Citations were right. The guard caught a number the generator had invented and refused to pass it along. Exactly its job, done correctly, against a generator that was not the one I thought was running.
The only piece that lied was the label. The API reported the model I had selected rather than the model that actually answered. That one mismatch hid the whole swap. I had built four kinds of machinery to keep the system honest about its evidence and none at all to keep it honest about itself.
Twenty years in regulated systems and that is still the shape of it. The thing that fails is never the part you measured. It is the part so obviously true that nobody thought to check.
-- Justin Higgins. Software Engineer, Midwest. Built a retrieval system whose best feature is declining to answer.
Companion pieces: The Principal IC Moat - the diagnosis. The Work Looks Like Nothing - why the judgment never had anywhere to be recorded.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com