Three Bugs in the Instrument
There is a retrieval system I built this summer over a thousand near-identical insurance contracts, and an evaluation harness to go with it. The harness has already told me something flattering once that turned out to be a statement about the harness. That was the second time an instrument of mine had done that.
Here are three more from the same project, found this week. They are smaller, they are the same shape, and I am starting to think the shape is the job.
Every Chunk Was Exactly 1,600 Characters
While verifying evidence for an unrelated fix, I noticed that every contract's benefit-schedule chunk was exactly 1,600 characters long. All 1,008 of them.
A uniform length is always a bug signal. The chunker cut off any over-long block that had no end-of-sentence mark, which describes every schedule table in every contract. Cut mid-row. Silently dropping the tier-three and tier-four prescription-drug rows, corpus-wide, for months. Every retrieval number in the project had been measured against a corpus that was missing the most important rows of every contract's most important table.
The reason it had not been fixed sooner was a second bug hiding behind the first. A re-ingest was estimated at six hours, measured at 2.2 documents per minute, and that estimate had justified putting off every corpus change for six weeks. The estimate was real. It was solving the wrong problem. The corpus had 36,512 chunks but only 4,965 distinct texts. Keying embeddings by content instead of by position made the re-ingest cost 3,196 embeddings, about 11 times less work. It ran locally in six and a half minutes.
The impossible job was a deduplication problem wearing a compute problem's clothes.
The Grader Was Grading on the Last Three Digits
I ran an adversarial review over a day's changes, five dimensions, each finding attacked by a skeptic told to default to "refuted." Thirteen verdicts. One survived, and it was a real bug in a fix I had already measured and written a test for. My stopword list carried verb endings but only singular nouns, so one question was fixed while its plural sibling still false-abstained. Opposite verdicts on the same question purely from whether the word was plural. The measurement was right and incomplete. Only an outside reader looking for the kind of bug, rather than the one that got reported, caught it.
A second finding killed a result I had already published in the decision log. The function that checks whether an answer states the expected figure inherited a regex whose leading pattern could not match a single digit before a comma. So the expected figure 6,300 became the token 300. An answer saying 300 was graded correct. An answer saying 6300 was graded wrong. 2,246 of the 2,428 expected figures in the eval set had that shape. The model bake-off that had justified switching the production model had graded on the last three digits of 92 percent of its ground truth.
I re-ran the entire bake-off after fixing it rather than reasoning about which direction the error pushed. Every model scored the same or lower. The ranking held. The winning model stayed at 1.000. The decision stands, but it stands on re-measured numbers, and the superseded run is kept in the repo next to the new one rather than deleted.
The third finding from the same pass: the spelling of a document's source path was its identity. The command-line tools passed relative paths and the ingest endpoint passed absolute ones, so a file ingested both ways minted two document rows, every clause twice in retrieval, and the ingest exited zero. Fixed centrally rather than at each call site, because the split came from two callers and a third would have brought it back.
The Honest 0.688
A family of fifteen evaluation cases scored zero at the abstention gate while the same cases scored fifteen of fifteen end to end. The zeros were an accurate measurement of an unscoped pipeline. The gate harness had reimplemented the scoping step and dropped part of it. That was the third caller to do so, and the first two were recorded in a comment on the function they should all have been calling. The fix is one exported function that everything takes whole.
Until that fix, the suite reported abstention accuracy as a true 0.688. I did not drop the failing family to make the number green. A suite that reports 1.000 because the failing cases were removed is worse than one that reports an honest 0.688 with a named cause.
And when I dug into the documented three-of-fifteen leak case by case instead of trusting the total, all three failures were one template. Each returned a neighboring benefit's visit limit from an adjacent sentence. The standing root-cause note, which said there was no on-point evidence, was wrong. There was evidence. It just carried no figure.
The Shape
A uniform length. A regex off by one digit. A path spelled two ways. A harness reimplementing a step. None of those are retrieval problems. None of them are model problems. Every one of them is a bug in the instrument, and every one of them produced a number that looked like a fact about the system.
I keep a rule now that I did not have in June: a change made without re-measuring is a guess wearing a number's clothes. The evidence threshold in this system is tuned against a measured end-to-end result. Auto-adjusting it from user feedback, which I was asked to do, would repeat the mistake this repo has now documented three times. So I did not.
-- Justin Higgins. Software Engineer, Midwest. Found three more bugs in the instrument and re-ran the bake-off instead of reasoning about it.
Companion pieces: Everything Looks Like Everything - the first two. Spider-Man Is Not a Person - the score that was not a confidence.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com