Twelve Days of Green Tests
In August I rebuilt a proprietary database wire protocol I'm not allowed to name, for a platform I can't license, using nothing but the real client drivers that already speak it. Point a driver at a server that implements nothing, read what the driver does with the answer, fix the one thing it complained about, repeat. Seventeen days, thirty-two working sessions, 153 tests, 222 catalogued protocol elements, about 20,700 lines of Go, all of it written by an agent with me in the chair.
I can talk about the shape of it but not the names, which is a strange constraint for an essay and one I'm going to honor anyway. What I want to talk about is the third time one of my own instruments told me something flattering that turned out to be about the instrument. I've said before that I stopped treating that as bad luck. I now think it's a law, and I think I can state it in one question.
The Suite That Could Not Fail
Roughly two weeks into the build, the agent found a bug in a test itself, not in the code the test was checking.
The handshake at the start of every connection involves four request-and-reply pairs. The conformance test had asserted the reply type for all four since the endpoint existed, and was green the whole time. It was also wrong about two of the four. It had been written by reading the endpoint's own implementation and asserting that the endpoint did what the endpoint did, and nobody had checked that against anything outside the codebase. The comment the agent left in the corrected test says it plainly: this previously asserted the same type for all four, which was this codebase's own model asserted back at itself.
Twelve days, every run green, and it caught nothing, because nothing outside was being checked. It wasn't broken, just circular.
What I keep coming back to is how it presented: as success. Green tests, a growing decision log, confident forward motion. Every signal I'd normally read as "this is going well" was there, and every one of them came from the same model that generated the code. If the real driver hadn't eventually refused a connection outright, I don't know when I'd have noticed. Possibly at a demo.
The Diff That Passed and the Client That Said No
A few days later the opposite happened, though it took me a while to see it that way.
The endpoint has to send a descriptor that tells the client the shape of a result: how many columns, what types, what names. I had a reference corpus for this, recorded from a real server, independent of anything the agent had written. The descriptor was checked against it byte for byte. Eighty-nine of eighty-nine objects. Sixty-four of sixty-four columns. Exact.
The real client rejected it anyway.
Matching a recorded reference proves what some server once sent, under some conditions, to some client, not what this client needs now. Versions move, negotiated options move, and the recording was true but not enough.
So I had two oracles, and they fail in opposite directions. The live client alone confirms whatever you already believe, and it looks like success. The recorded reference alone confirms what was once true, and it shows up as a passing diff. Neither one ever says "that was correct." A client only says "I accepted that" or "I rejected that," and a recording only says "this happened once." Only something you didn't write, and didn't get from the same source you're building from, tells the two apart.
The rule that fell out is short. A test may never check an implementation only against the model that generated it. At least one layer has to anchor to material the implementer didn't write. Where no such material exists, don't make one up: file the gap and leave the claim unmade. A fixture you invented turns an open question into a permanent wrong answer that passes every time.
I think this is also the ceiling on every record-and-replay approach anyone has ever sold me. Coverage stops at what was recorded. I didn't fully believe that until the eighty-nine of eighty-nine.
Belief Has No Version Control
The most expensive failure had no technical warning sign at all.
Most protocols negotiate. The server announces what it is, and the client adapts: byte order, feature set, encoding, error vocabulary. The endpoint had a placeholder identity string, an internal name never meant to mean anything, and it happened to look like a different platform's naming convention. Every client that read it silently switched byte order and started speaking the wrong dialect of the protocol back at us. Every measurement after that was a correct reading of the wrong system, for weeks, with no error anywhere.
Two forces pushed toward that mistake, and neither was technical. The freely available documentation and sample traffic were for the edition without a license gate, which was the wrong edition. And the client I could actually run would only speak the variant I cared about if I paid for an entitlement I didn't have. The oracle was bent by what it cost to get at it.
When I finally said it out loud, the sentence in the transcript isn't elegant. "We need to cleanse the project of references to that platform potentially cause I think we may have poisoned the well." Typo and all. That was the most important decision in the project, and it got made in a side conversation about an environment variable.
What changed how I think about agent-run work was the fix. It touched almost nothing in the source: one default identity string. The endpoint had been sending the correct type declarations the entire time, so on the wire it was already clean. The poisoning was a nametag and prose.
The wrong premise had spread through piled-up notes, decisions, and working context, not through files. A diff couldn't find it and a revert couldn't undo it. The cleanup was a sweep through twenty documents for claims that had gone false, which isn't something git knows how to do.
In an agent-run build, the durable artifact is belief, and belief has no version control.
The One Question
It all comes down to one pre-flight question, the cheapest check I know of for this kind of work.
What is the anchor I did not author?
If the answer is "the specification I am also implementing from," the loop will confirm itself and the cost won't surface for weeks. If it's "the recording I made of the real thing," you'll match the recording and the real thing will still refuse you. If it's a second, stricter client, an open implementation of the same protocol, a published set of test vectors, or a rival product, then you have something the loop can't argue with, and the ratchet only turns one way.
The instinct on day one is to open a socket and start answering. The better first move is to find and pin the anchor, declare which dialect you are before you send a byte, and stand up a ledger for every question the sources can't settle. Those three decide whether every round afterward is real. An endpoint that answers nothing but is honestly anchored gets there. One that answers everything against its own model produces twelve days of green tests and no protocol.
I'm writing the method up as a checklist, eight preconditions and five gates, with the argument behind each one, because a checklist without the argument reads as content marketing. The figures all come from one reconstruction. The failure modes are measured. The claim that this carries over to other protocols is reasoned from properties and not yet shown on a second target. Treat it as a hypothesis worth testing cheaply, which is the most honest thing I can say about anything I built this summer.
-- Justin Higgins. Software Engineer, Midwest. Rebuilt a protocol from its client, and learned the client was never the oracle.
Companion pieces: Everything Looks Like Everything - the second time an instrument measured itself. The Rules Are Downstream of the Work - where the provenance tagging came from.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com