Everything Looks Like Everything
The demo everybody builds is retrieval over a diverse corpus: a wiki, a support archive, a folder of PDFs about unrelated subjects. You ask a question, the embedding finds the three chunks about that thing, a model reads them, and you get an answer with citations. It works on the first try, which should have been the tell.
I spent the summer on the other case: one thousand and eight insurance contracts, synthetic, generated to be exactly as redundant as the real ones. Same carrier, same benefit categories, same section headings in the same order. Deductibles, out-of-pocket maximums, coinsurance, network tiers. Every document has all of it, and every document says something slightly different.
Ask that corpus what the individual deductible is and the only honest answer is a question. Whose? For which plan year? Under the summary plan description, or the amendment that superseded part of it in March?
Retrieval over a diverse corpus is a search problem. Over a redundant one, it's a problem of telling things apart, dressed up as search. Almost nothing written about RAG covers the second case, and inside a regulated enterprise the second case is nearly all of the work.
I picked insurance contracts because I've spent two decades near them and I know exactly how they misbehave. That mattered more than any modeling decision I made afterward.
Everything Matches
The first honest measurement was 3.5%. A number that low usually means something's unplugged. Nothing was. Asked about an out-of-pocket maximum, the embeddings returned a fistful of passages about out-of-pocket maximums, ranked by how closely each one resembled the question, and all of them resembled the question. That's what redundancy means. The vector index had no way to say the only thing that mattered: one thousand and seven of those documents belonged to somebody else.
I'd seen this failure before in a different costume. Ranking candidates by similarity to a job description has the same shape: when everyone uses the same nouns, similarity stops carrying information and starts laundering it. Resemblance is a weak signal when the documents differ and a worthless one when they don't. I wrote a whole essay about that without noticing I was describing a retrieval bug.
The fix was to work out who the question was about before searching for an answer: pull the business name, plan ID, or group number out of the query, match it against the record index, narrow to that record's documents, and only then retrieve. Same embeddings, same reranker, same everything downstream.
3.5% to 99.8%.
The hard part was never the retrieval stack. It was noticing that the question had a subject, and that nothing in the pipeline had bothered to ask which one.
The Instrument Was Measuring Nothing
Then the part I'd rather not include. A tuning run came back with 0.998 document recall, excellent by any published standard. I moved on, wired the answer path, and ran it end to end against real questions.
It answered one percent of them.
Retrieval was finding the right documents and the system was returning nothing. The recall number was true and useless, because the evaluation harness called the retriever directly, without the scoping step. It fuzzy-matched every query against all thousand near-identical contracts and graded itself on a pipeline nobody actually ran. The single change that took the corpus from unusable to usable was the one the measurement skipped.
Four fixes later it answered 98% and abstained correctly on 93%, but the number I keep is the first one. A stage metric isn't a system metric. I knew that before I started, and knowing it didn't help, because the harness never lied to me. It answered exactly the question I'd asked it, and I'd asked the wrong one.
Three separate callers had also each written their own version of that scoping step, and each had quietly dropped a different piece of it. Three independent authors, all of them me, all convinced it was two lines of glue. It lives in one exported function now, and everything takes it whole.
This is the second time one of my own instruments has told me something flattering that turned out to be about the instrument. I've stopped treating that as bad luck.
The Product Is the Refusal
It took me a while to stop treating the refusals as the boring part of the build. A contract system that answers nearly every time and is confidently wrong on the rest is worse than no system, because from the outside you can't tell the wrong answers from the right ones. What matters is whether the thing knows the difference between having evidence and having text.
So I put in a gate. Evidence gets scored on authority and recency before anything is generated. A certificate of coverage outranks a benefit summary, an amendment outranks the document it amends, and both outrank a marketing one-pager that happens to use the same words. Then the passages that survive have to actually agree with each other. If they don't line up, the system doesn't answer, and it doesn't hedge. It names the reason:
Divergent. The retrieved clauses give different figures and nothing in the file breaks the tie.
Stale. The best-supported answer comes from a document already superseded by another one in the same file.
Conflicting. Two sources of equal authority disagree on their face.
Below authority. The only support comes from a document not entitled to settle the question.
That list is the product. The answers are table stakes, since anything with an API key produces answers. Every one of those four is a sentence a competent analyst would say out loud in a meeting, and a language model won't say any of them unless you build the machinery that forces it to. Left alone, it'll produce a fluent, well-cited number from a superseded amendment and never mention that it was superseded.
Getting the last family right took a while. The system was refusing those cases correctly but blaming the refusal on the model's own reluctance instead of the missing evidence. Right behavior with the wrong reason on the record is its own kind of failure. Fixing it moved that family from zero out of fifteen to fifteen out of fifteen in both harnesses, with the answerable set unchanged. I measured it both ways instead of assuming it.
The Half with No SKU
An abstention is an absence, and absences don't get credit.
I've spent twenty years producing that particular absence: the migration scoped down before it became a decade of carrying cost, the vendor trap that doesn't close, the confident number nobody ends up acting on. None of it lands on a balance sheet, and the reason it stays unpriced was never that it can't be measured - it's that it had nowhere to be recorded.
This one leaves a record. The refusal ships with its citations attached. You can open the two documents that disagreed, see which one carried more authority, check the effective dates, and see why the tie couldn't be broken. The judgment is shown with its inputs, in a form somebody else can check and overturn.
I didn't set out to build a working model of my own career, and it's inconvenient how cleanly it landed. The system's most valuable behavior produces no output, and all the engineering went into making it readable enough to be worth something.
What to Do with It
The repository is proprietary today: every right reserved, private manifest, the whole posture. I've been rethinking that, and the useful question turned out not to be whether the code is good enough but what here is actually scarce. It isn't the retrieval stack. Hybrid dense-plus-BM25 with a cross-encoder rerank is a recipe now, and mine is a competent version of something a few thousand people have built.
What's scarce is the corpus: a thousand documents built to be nearly identical, each tagged with an authority tier and effective dates, plus the generator that makes more of them and a graded evaluation set that includes the cases where the correct answer is a refusal. I can't find a public equivalent. Every retrieval benchmark I know of assumes documents that differ from each other, which is the easy case and not the one anybody's contract file resembles.
The honest read on the rest is that nobody's going to run it. It needs Postgres with pgvector, a reranker endpoint, and a deploy shaped like mine. Adoption isn't the return. Being read is, and the interesting part of this project is entirely visible from the outside: the eval numbers, the decision log, the commit where the measurement turned out to be measuring nothing. I haven't decided yet.
The Label
One more thing happened, and if I could keep only one, it'd be this.
In production, someone asked for an individual out-of-pocket maximum for a specific business. The query was correctly scoped and resolved to exactly one record, and the evidence was retrieved and cited. That evidence contained the row | Out-of-Pocket Maximum - Individual / Family | $8,000 / $18,400 |. The system returned its refusal string. The badge on the response read claude-sonnet-4-6.
Production had no Anthropic key. The gateway had silently fallen back to the deterministic stub used for offline testing, whose canned reply mentions its own token limit: 512. The attribution guard saw a figure in the answer that appeared in none of the cited passages, ruled that the number couldn't be traced to a source, and replaced the whole thing with the refusal.
So every piece of judgment in the system worked. Retrieval, scope and citations were all right, and the guard caught a number the generator had invented and refused to pass it along. That was its job, done correctly, against a generator that wasn't the one I thought was running.
The only piece that lied was the label. The API reported the model I'd selected instead of the one that actually answered, and that one mismatch hid the whole swap. I'd built four kinds of machinery to keep the system honest about its evidence and none at all to keep it honest about itself.
Twenty years in regulated systems, and that's still the shape of it. The thing that fails is never the part you measured. It's the part so obviously true that nobody thought to check.
-- Justin Higgins. Software Engineer, Midwest. Built a retrieval system whose best feature is declining to answer.
Companion pieces: The Principal IC Moat - the diagnosis. The Work Looks Like Nothing - why the judgment never had anywhere to be recorded.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com