1,591 Green Checks Never Looked at the Truck

On Saturday, November 7, I gave an agent session a brief for a 3D model of a modular flatbed. It's a business idea I want to pitch to my brother: truck-bed modules that drop onto docking cones and lock themselves, no straps, released by hand. I typed five lines in the first hour, and then I left.

The session ran 5 hours 43 minutes, with 6 workflows and 34 subagents. When it stopped, the test suite reported 1,591 passing checks and zero console errors. Draw calls were 105 against a budget of 400.

I didn't see a single picture until the very end. When I did, it was a gray box truck on a black deck covered in a grid of small squares. The latch, which is the whole idea, read as a cylinder and a green rectangle. In 5 of the 6 standard camera views it was a square or nothing. The hero shot showed a gantry crane lowering the module onto the deck, which says "needs a hoist". The pitch is that you don't need one.

On November 11 I had a postmortem done on the session: the commits, the QA notes and all three versions of the spec. This is what it found, and what I'd missed.

What the gates checked

The test file runs 16 sections. They include schema, derivations, axle math, rules, preset goldens, a hash round-trip, the lock sequence, and two checks named platform-equals-data and mesh-equals-data. That last one confirms each mesh's bounds land within 0.6 in of the numbers that built it.

The browser check looked for console errors, draw calls, sideways scroll and minimum font size. A contact-sheet script rendered PNGs of the model from fixed cameras, then recorded a sha256 hash of each one, so the next run could prove it was byte-identical.

In August I wrote that you should verify the artifact, not the process. This session did render the artifact. Then it hashed it. Every check that touched a render compared it to an earlier render or to the data it came from. Nothing compared it to what a truck looks like.

No picture to check against

There wasn't a reference image anywhere in the repo. The audit at the start of the session had one on its list: the critic's prompt named a "reference-photo blockout". The answer that came back swapped that for elevation drawings generated from the parameter file, so "proportions are checked against numbers rather than by eye". They're the same numbers the mesh was built from.

The spec's own "Done when" section said: "Executable checks first; then two named humans. Another model is never the check." The two humans were me and my brother. Both sign-offs were still marked "pending" in the QA notes when the postmortem ran. The contact sheets were "reviewed tile by tile by Fable", a model from the same session, with no rubric. So the model graded its own renders, and I wasn't there to say otherwise.

All three milestone reviews failed something at least once. Each failure was about logic or a false claim. They didn't fail anything for how it looked. At the end of the second milestone, the session reported that "the truck, the flatbed with its docking points, and the kitchen all read clearly." There wasn't a gate that could disagree with that sentence.

Where the time went

The first geometry builder started 2 hours 51 minutes in, about half the clock. Before that came an audit, two competing spec rewrites with judges, an amendment to support any truck, contracts, and the QA tooling. Builders ran for about 42 of the 343 minutes. That's 12 percent.

The truck got one real pass. It was written once, in a single 26-minute run, then got a 3-line change. The repo ended with 11,184 lines of fixtures and 2,722 lines in the geometry folders, tests included.

The spec grew from 7,917 words to 14,886 across two rewrites. The section on visual direction shrank by more than a third in the same rewrites. The truck heading lost a word: "Truck (generic, detailed, no branding)" became "Truck (generic, no branding)". The docking hardware went from "(high detail; this is the hero part of the model)" to "(the hero part)".

The box truck didn't break the spec, either. It ranked the truck fourth of four in detail priority, and the milestone prompt said "blockout quality is fine where the spec allows". The hero hardware was spec'd at 3 in on a 102 in deck, and its detail work went in at 0.1 in, which is below a pixel at truck-wide framing. It ended at milestone 1.5 of 7, with 1 of 31 module definitions built.

What this adds to the August post

The August rule was that the guard that prints green isn't a sensor and the artifact is. I still think that holds. What I missed is that checking the artifact against your own inputs can't tell you it's ugly. The mesh matched the data to within 0.6 in. The data described a box, and the mesh was a very accurate box.

How something looks didn't have an executable gate. The only checks that could have caught it were a reference photo, a reviewer that didn't build it, and me looking at an early render. I didn't have any of them in place before the spend.

What the rerun does differently

The prompt for the rerun names the failure: "Every gate checked data and none checked appearance. No reference image existed, and the model approved its own renders." And: "I saw nothing until the end."

It starts with 8 to 12 reference photos, before any geometry, and says to judge proportions and appearance "against these, never against your own parameters." It doesn't allow a spec phase: "The first hour produces a model and a render, not documents."

Within about 90 minutes it has to write a checkpoint page with one hero render, the exploded docking diagram, and its truck render side by side with a reference photo. Then: "Print its path and end your turn. Don't poll, sleep or continue unattended."

After that, a fresh reviewer with no build history puts each render beside its reference photo and lists every way it reads as a toy. It gets at most two fix rounds, and anything left comes to me "as open, not passed." A second fresh reviewer sees the hero and the attach and release strips with the captions removed. It isn't told the answer, and it has to explain the mechanism in one sentence. Model reviewers are a second line. I approve the checkpoint and the final page, and they don't.

The rerun started on November 11. Whether its truck looks like a truck is still open, and I'm the one who has to say so.

-- Justin Higgins. Software Engineer, Midwest. Had 1,591 checks agree with each other about a gray box.


Companion piece: Twelve Days of Green Tests - a core test that passed for twelve days by checking code against itself.

Reactions, disagreements, war stories: jchigg2000.dev@gmail.com