Survived Is Not Endorsed
Twice this year I pointed a swarm of agents at a pile of free government data and asked it to find something worth building. Both times the interesting output wasn't the findings. It was what the swarm said about its own process when I made it check.
The Skeptics
The second sweep ran 76 subagents in one sitting over public health-system data. Every number it produced got a label. Found meant published and cited with its date. Estimated meant arithmetic on found figures, with the arithmetic shown. Modeled meant it rested on assumptions, with the assumptions listed. Every candidate opportunity was attacked by three independent skeptics before it was sized: one on data reality, one on prior art, one on economics and regulation. The corrections they forced are stored next to the finding instead of folded into it.
That sounds rigorous, but the first version of the schema had one column for the skeptic verdict, and it ran two different things together.
Across the nine top picks, there was exactly one confirmed vote. Everything else that survived came through weakened: the skeptics forced corrections and the finding stood, in a smaller form. So the verdict column got split, because survived is not endorsed. A finding that made it through three attacks has three sets of scars, and the scars are the useful part. Storing it as a clean pass throws them away.
There's a related rule in the database README that I want to quote, because I've never seen it in a research pipeline before. Mine the workflow journal for every agent result, including candidates the skeptics knocked down, whose verdicts never reach the workflow's return value. The losers are evidence too. A pipeline that only keeps winners can't tell you why the winners won.
The Critic That Checked the Process
Two critics ran after the sweep. One checked rigor: are the numbers right? It pulled a landing page the sweep had cited and found it contained three percentages, none of which were the two the sweep had quoted. A load-bearing input had failed to check out three separate times and was being carried entirely on an upstream verifier's say-so. That's the ordinary kind of catch.
The other critic didn't check any numbers at all. It checked coverage: whether the review process had been applied evenly. Of sixteen candidates, eleven got three skeptic lenses, one got one, and four got no verification votes at all. One of those four survived into the final nine while the other three were cut, so the cut line between them was drawn on something other than evidence.
I hadn't thought to ask that. The numbers could all be right and the process could still be uneven, and an uneven process means the ranking is partly a product of which candidates happened to get attacked. The coverage critic also had a structural read that stung. All nine survivors were cost-side, retrospective, and drawn from the same family of files. Not one touched revenue or looked forward. It called the result a repricing shop's research agenda, not a health plan's, and named the growth-side opportunity the sweep had walked past twice.
I don't have a name yet for a critic that checks how evenly the rigor was spread instead of the rigor itself. It's the most useful agent I ran all year, and it cost less than lunch.
Forget Your Own Answer
The first sweep, in July, was the one that taught me to build the second one this way.
I asked a swarm to evaluate 66 free datasets and pick four a lean builder could turn into a business. It did. Then I ran the identical evaluation two days later with one change: ignore the final four and any rankings from before, and start from the full dataset yourself.
It landed on a completely different final four. None of the first run's picks came back as top choices. Three orchestrated workflows, about 95 subagents, about 3.9 million tokens, 67 minutes of wall clock, and a different company at the end.
The second run also wrote down that it had failed its own stopping rule. The fourth seat was supposed to be stable across two consecutive rounds, and it wasn't. Round one picked one candidate, round two a different one, round three the first one again. Every contender for that seat scored two out of five on seat-worthiness. The field had no confident fourth pick, and more rounds would thrash instead of settling, so it stopped and said so. It shipped "the fourth seat doesn't exist" as the answer instead of filling the seat.
What the anchoring comparison showed, in one sentence from the second run: when you don't anchor on the prior answer, the strongest opportunities shift toward fast, self-serve, verified-demand report tools on separate buyers, and away from the slower, procurement-gated diligence plays the narrower run favored. The first run had been pulled toward its own earlier conclusion, and I couldn't have seen that without running it blind.
The Uncomfortable Version
Everything above is about agents, and the reflex is to file it as an agent problem. I don't think it is.
Every research process I've watched inside a large company has the same shape. Some findings get attacked hard and survive weakened. Some never get attacked at all and survive clean, and the clean ones look stronger. Nobody stores the losers. Nobody re-runs the analysis with the prior conclusion hidden, because the prior conclusion is in the deck the executive already saw.
The swarm didn't invent any of that. It just ran fast enough, and logged thoroughly enough, that I could see it. If you want to know whether you anchored, you have to run it again without the anchor and be willing to get a different answer.
-- Justin Higgins. Software Engineer, Midwest. Ran the same evaluation twice, blind the second time, and got a different company.
Companion pieces: Twelve Days of Green Tests - what is the anchor I did not author? The Decision Log Ate the Project - three documents echoing one origin is not three confirmations.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com