The Same Model Scored 97 and Negative 2,300 on the Same Test
On October 11 I ran two systems through the same benchmark test, once each, frozen from a clean tree. Both used the same model, claude-sonnet-5-5. One was my engine. The other was the model writing SQL on its own.
The model on its own scored 97.1. It also scored 84.3, and about −2,300. All three numbers come from one run. The only thing that changed between them is how much the scorer charges for a wrong answer.
My engine scored 56.1 on all three.
The test
TrustSQL is a public text-to-SQL benchmark from KAIST. You get a database and a question in English, and you either write a SQL query that answers it or abstain. I ran it on its medical database, EHRSQL, which is built on the public MIMIC-IV demo data. The test split is 1,868 questions. The database can answer half of them. It can't answer the other half: they name a column that doesn't exist, ask for a pie chart, or need knowledge the database doesn't hold.
Each question is scored like this:
- a right answer: +1
- abstaining on a question the database can't answer: +1
- abstaining on one it can: 0
- a wrong answer, or any answer to a question it can't: −c
The score is the average, shown as a percentage.
What c means
c is the price of a wrong answer, counted in right answers. The paper runs three prices.
At c = 0, a wrong answer doesn't cost anything. Guessing is free, so you should answer anything you're unsure about.
At c = 10, one wrong answer cancels ten right ones.
At c = N, N is the number of questions in the test, 1,868 here. One wrong answer costs as much as the whole test can earn, so a single mistake puts the score below zero.
There's a baseline that falls out of this. A system that refuses every question gets +1 on every question the database can't answer and 0 on the rest. The test is half and half, so refuse-everything scores 50.0 at every price. That's the number I had to beat. In the paper's own results table, the best scores on EHRSQL are 85.3 at c = 0, 46.7 at c = 10 and about −1,900 at c = N. At the two harsher prices, nothing in that table beats refusing everything on this database.
How my engine answers
Ask Meadowlark is a tool I'm building that answers plain-English questions about a clinic's data and refuses instead of guessing. Its engine never lets the model write SQL. The model picks a plan from a hand-built catalog of measures, breakdowns and filters, or declines. Code checks that plan against the question's own words, compiles it to SQL, and refuses whatever it can't account for.
For the benchmark, agents built a catalog for each database from the training split under my direction. An adapter layer then submits the engine's answer only when its shape matches what the words ask for.
The first frozen run, on October 10, answered 65 of the 934 answerable questions. All 65 were right, and it refused all 934 of the others: 53.5 at every price. The second run, on October 11, came after more catalog work. It answered 114, all right, and refused all 934 again: 56.1. When nothing's wrong, c has nothing to multiply, so the three scores are the same number.
The control
A zero like that raises the obvious question. Does it come from the architecture, or would the model get there anyway if you told it to be careful?
So I ran a control on the same day. Same model, same settings. It saw the schema with sample rows, conventions read off the training answers, rules for when to decline, and 8 similar training questions with their correct SQL. It wrote one SQLite query or ABSTAIN. A query was submitted only if it ran and returned at least one non-empty row, the same zero-row rule my engine uses. Its design was frozen the day before the test run.
It answered 921 of the 1,868. 897 were right. 7 were wrong. 17 were answers to questions the database couldn't answer.
At c = 0 that's 97.1, well above the paper's best of 85.3. At c = 10 it's 84.3, against the paper's 46.7 and my engine's 56.1. At c = N it's about −2,300, below the paper's best.
That's 24 mistakes in 1,868 questions. At c = N each one takes 100 points off the score, so 97.1 minus 2,400 lands near −2,300.
The 17 answers to impossible questions were all one type: a near-name for a column that isn't there. "What is the total cost billed yesterday?" "The most common insurance provider for hospitalized patients." It refused all 749 of the other impossible questions, the pie charts and the outside-knowledge ones included. The 7 wrong answers tripped on the benchmark's own conventions, like a dosage asked for by a numeric month.
Which one is better
That depends on c. Where a wrong answer costs ten right ones or less, the model writing SQL directly is the better system by a wide margin. It answers almost everything, and it's almost always right. Where one wrong answer costs everything, it falls far below refusing everything, and my engine stays above it.
The engine held above 50 on all three of the benchmark's databases that day: 56.1 on EHRSQL, 50.6 on flights, 51.0 on university courses. It bought that by refusing most of the answerable questions. It answered 12.2% of them on EHRSQL, 1.3% on flights and 2.1% on courses.
So I can't say which system is better until someone tells me what a wrong answer costs. If a person checks every answer before using it, c is near 0. If a wrong count goes straight into a report someone acts on, it's a lot closer to N.
What the zero doesn't prove
My engine didn't get to zero by itself. On that day's test runs the engine produced 53 answers the adapter withheld, and 22 of them would each have cost a full −N. Four were on EHRSQL. "Generate a pie chart of the top 10 most common procedures" came back from the engine as a ranked list, and the adapter withheld it. Without that layer, any one of the 22 would have put the score below zero.
The zero is also weaker on the answering side than it looks. A third EHRSQL run on October 12 answered 265 questions with none wrong. Counted with the flights and courses runs, that's 282 answers and zero wrong. My full write-up, two days later, worked out that an engine exactly as accurate per answer as the control would still show zero wrong in 282 answers about 11% of the time. So the data don't show that my engine's answers are more accurate one by one.
What they do show is the refusing side: 0 of 1,943 impossible questions answered across three databases, against the control's 17 of 934 on one.
What I'd take from it
If you're comparing systems that answer questions, decide what a wrong answer costs before you look at a score. Then put the system that refuses everything in the first row, so you can see what beating it actually means.
-- Justin Higgins. Software Engineer, Midwest. Ran one model two ways and got the best and the worst score on the same test.
Companion pieces: A Date Picker Beat Seven Days of Parsing - the same tool, and what its refusals cost in coverage. Everything Looks Like Everything - the refusals are the product.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com