Twelve Days of Green Tests
In August I rebuilt a proprietary database wire protocol I am not allowed to name, for a platform I cannot license, using nothing but the real client drivers that already speak it. Point a driver at a server that implements nothing. Read what the driver does with the answer. Fix the one thing it complained about. Repeat. Seventeen days, thirty-two working sessions, 153 tests, 222 catalogued protocol elements, about 20,700 lines of Go, all of it written by an agent with me in the chair.
I can talk about the shape of it. I cannot talk about the names, which is a strange constraint for an essay and one I am going to honor anyway. What I want to talk about is the third time one of my own instruments told me something flattering that turned out to be a statement about the instrument.
I have said before that I stopped treating that as bad luck. I now think it is a law, and I think I can state it in one question.
The Suite That Could Not Fail
Roughly two weeks into the build, the agent found a bug in a test. Not in the code the test was checking. In the test.
The handshake at the start of every connection involves four request-and-reply pairs. The conformance test asserted the reply type for all four. It had been asserting them since the endpoint existed. It was green the whole time.
It was also wrong about two of the four. The test had been written by reading the endpoint's own implementation and asserting that the endpoint did what the endpoint did. Nobody had checked that against anything outside the codebase. The comment the agent left in the corrected test says it plainly: this previously asserted the same type for all four, which was this codebase's own model asserted back at itself.
Twelve days. Every run green. Caught nothing, because nothing outside was being checked.
It was not broken. It was circular.
The thing I keep coming back to is how it presented. Not as an error. As success. Green tests, a growing decision log, confident forward motion. Every signal I would normally read as "this is going well" was present. Every one of them was generated by the same model that generated the code. If the real driver had not eventually refused a connection outright, I do not know when I would have noticed. Possibly at a demo.
The Diff That Passed and the Client That Said No
A few days later the opposite thing happened, and it took me a while to see that it was the opposite.
The endpoint has to send a descriptor that tells the client the shape of a result: how many columns, what types, what names. I had a reference corpus for this, recorded from a real server, independent of anything the agent had written. The descriptor was checked against it byte for byte. Eighty-nine of eighty-nine objects. Sixty-four of sixty-four columns. Exact.
The real client rejected it anyway.
Matching a recorded reference proves what some server once sent, under some conditions, to some client. It does not prove what this client needs now. Versions move. Negotiated options move. The recording was true and it was not enough.
So here are the two oracles I had, and they fail in opposite directions. The live client alone confirms whatever you already believe, and it looks like success. The recorded reference alone confirms what was once true and shows up as a passing diff. Neither one ever says "that was correct." A client only says "I accepted that" or "I rejected that." A recording only says "this happened once." Only something you did not write, and did not get from the same source you are building from, tells the two apart.
The rule that fell out is short. A test may never check an implementation only against the model that generated it. At least one layer has to anchor to material the implementer did not write. And where no such material exists, do not make one up. File the gap and leave the claim unmade. A fixture you invented turns an open question into a permanent wrong answer that passes every time.
This is also, I think, the ceiling on every record-and-replay approach anyone has ever sold me. Coverage stops at what was recorded. I did not fully believe that until the eighty-nine of eighty-nine.
Belief Has No Version Control
The most expensive failure was the one with no technical warning sign at all.
Most protocols negotiate. The server announces what it is, and the client adapts: byte order, feature set, encoding, error vocabulary. The endpoint had a placeholder identity string, an internal name, never meant to mean anything. It happened to look like a different platform's naming convention. Every client that read it silently switched byte order and started speaking the wrong dialect of the protocol back at us. Every measurement after that was a correct reading of the wrong system. For weeks. With no error anywhere.
Two forces pushed toward that mistake and neither was technical. The freely available documentation and sample traffic were for the edition without a license gate, which was the wrong edition. And the client I could actually run would only speak the variant I cared about if I paid for an entitlement I did not have. The cheap path was the wrong-platform path. The oracle was bent by what it cost to get at it.
When I finally said it out loud, the sentence in the transcript is not elegant. "We need to cleanse the project of references to that platform potentially cause I think we may have poisoned the well." Typo and all. That sentence is the most important decision in the project and it was made in a side conversation about an environment variable.
Here is the part that changed how I think about agent-run work. When the contamination was found, the fix touched almost nothing in the source. One default identity string. The endpoint had been sending the correct type declarations the entire time. On the wire it was already clean. The poisoning was a nametag and prose.
The wrong premise had spread through piled-up notes, decisions, and working context, not through files. A diff could not find it. A revert could not undo it. The cleanup was a sweep through twenty documents for claims that had gone false, which is not a thing git knows how to do.
In an agent-run build, the durable artifact is belief, and belief has no version control.
The One Question
Everything above comes down to one pre-flight question, and it is the cheapest check I know of for this kind of work.
What is the anchor I did not author?
If the answer is "the specification I am also implementing from," the loop will confirm itself and the cost will not surface for weeks. If the answer is "the recording I made of the real thing," you will match the recording and the real thing will still refuse you. If the answer is a second, stricter client, an open implementation of the same protocol, a published set of test vectors, a rival product, then you have something the loop cannot argue with, and the ratchet only turns one way.
The instinct on day one is to open a socket and start answering. The higher-value first move is to find and pin the anchor. Declare which dialect you are before you send a byte, and stand up a ledger for every question the sources cannot settle. Those three decide whether every round afterward is real. An endpoint that answers nothing but is honestly anchored gets there. An endpoint that answers everything against its own model produces twelve days of green tests and no protocol.
I am writing the method up as a checklist, eight preconditions and five gates, with the argument behind each one, because a checklist without the argument reads as content marketing. The figures all come from one reconstruction. The failure modes are measured. The claim that this carries over to other protocols is reasoned from properties and not yet shown on a second target. Treat it as a hypothesis worth testing cheaply, which is the most honest thing I can say about anything I built this summer.
-- Justin Higgins. Software Engineer, Midwest. Rebuilt a protocol from its client, and learned the client was never the oracle.
Companion pieces: Everything Looks Like Everything - the second time an instrument measured itself. The Rules Are Downstream of the Work - where the provenance tagging came from.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com