The Shape Leaks
Correction, 2026-10-08. On 2026-09-26 an audit of the experiment's own harness found errors behind several figures in this post. The verdicts stand: all four methods still fail, and shape alone still carries roughly half the leak a husk carries. The figures are left as published, because they are the record of what ran. What they mean has changed:
- The LLM translation numbers describe a rewriter running without its verifier. A race in the harness switched the verifier off for 89 of the 90 husks in the arm that was supposed to have it on. The 78 percent, the roughly 50 percent in free text, the plus 0.44 to plus 0.48, the copied-span shares and the 22 of 90 all measured an unverified rewriter. A verified figure needs a paid re-run. The fail is not at risk: on the husks the verifier would have passed, both attackers scored higher than on the whole arm.
- 986 of 998 counts an arm the run's config did not declare. Against the declared arms it is 948 of 960, with the same twelve calls lost in transport.
- The structure-only figures shift a little. Eight of the sixty canonicalized files kept one authored comment the gates could not see. Leaving those out gives about plus 0.21 to plus 0.25, every result still below p 0.01.
- Copied spans were 0.275 when the attacker was wrong, not 0.276. Calls lost in transport had been counted as wrong answers.
Statements that were simply false are struck through below, with the correction beside them.
In April I wrote a working paper with a thesis I still think is beautiful, and in August I ran the experiment that killed it. This is the whole arc, because the killing is the part worth reading.
The Idea
More and more, architectural code review runs through a language model. A senior engineer who once spent days tracing layering violations across a service boundary now pastes a module into a chat window and gets a diagnosis in seconds. If your code is the product, or you work in healthcare or finance or defense, you can't do that. The code can't leave the tenant, and in-tenant models lag the frontier by a generation. So the choice on offer is weak diagnosis on private code or strong diagnosis on exposed code.
I proposed a third option: don't redact the code or encrypt it, translate it into a different domain. A membership-management service becomes a logistics dispatcher. A derivatives pricing library becomes a chess engine. The god-objects, the circular dependencies, the layering violations, the copy-pasted error handling and the hidden state machines all survive the translation, because the sickness is in the structure, not the words. Ask a frontier model to review the chess engine, and its diagnosis holds for the source.
I called the translated artifact a husk. What made it work was an invariant: the husk had to keep the structure a diagnosis depends on, meaning the same code property graph within a tolerance, the same anti-pattern density and the same mix of comment types. And a verifier had to enforce that invariant, not a request in a prompt. A rewriter's training rewards refactoring, and it'll quietly improve your code if you let it. Prompt-only enforcement of that invariant is a category error.
That last part held up. The rest didn't.
The Test I Wrote Before I Knew the Answer
The paper said its privacy claim, that a consumer model can't work out the source domain from the husk, had never been tested and was the next thing to test. So I pre-registered the test. Six synthetic domains, three held out. An attacker panel of two models, asked which domain a husk came from, once with a twelve-option menu and once in free text. A blind arm shown only a structural stub as the floor, the unhusked source code as the ceiling, and four husking methods in between. The pass criterion, written in advance: the attacker's accuracy on husks must not be distinguishable from its accuracy on the blind stub.
It ran on August 19. An account spending cap halted it mid-flight, and it resumed under the same run identifier and finished: 986 of 998 attacker calls, twelve lost in transport and counted as missing rather than scored wrong.
All four methods failed.
Three of them sat at the ceiling. Under format-preserving encryption of identifiers, literal tagging, and a composition of both, the attacker named the domain from the husk 100 percent of the time on the menu, the same as from the raw source. Those methods did nothing, which the paper had predicted.
LLM translation, the method the paper was actually about, was the only one doing any work. It roughly halved every leakage measure: on the menu the attacker got 78 percent against 33 percent blind, and in free text about 50 percent against zero. The 95 percent confidence interval on the husk-minus-blind difference was positive at both ends, so it still failed the bar I'd set.
The Floor Was Not a Floor
One amendment I had to write afterward is the kind of thing that makes me distrust every accuracy number I read.
The blind attacker, shown a ten-number structural stub with no code in it, gave the same domain on 26 of 29 items and 23 of 30 items regardless of what it was shown. Its 33 percent was response bias, not weak structural signal. Hold each attacker's answers fixed, shuffle the true labels ten thousand times, and the blind arm's corrected score is about plus 0.03, which can't be told apart from nothing. The husk-minus-blind difference barely moves, so the failures stand. But that 33 percent should never again be called a floor.
The Finding That Ends the Premise
Then I ran the experiment I didn't want to run.
Take each source file and canonicalize it mechanically. Remove every identifier, every string literal, every comment, every piece of visible text. Leave the control flow, the call graph, the arity, the nesting, the coupling exactly intact. Eleven hard gates enforce that no text survives check that no text survives (corrected 2026-10-08: eight of the sixty files kept one authored comment the gates could not see), and three model-free probes confirm no token is concentrated in any domain. Then ask the same attacker panel the same twelve-option question.
Corrected accuracy: about plus 0.23 to plus 0.26. The blind arm is plus 0.03. The full husk is plus 0.44 to plus 0.48. The source is plus 0.67.
Shape alone carries roughly half the leak a husk carries. Canonicalizing every numeric value too left the score unchanged or slightly higher, so it isn't constants leaking. It's the control flow, call graph, arity, nesting and coupling.
That's the premise eating itself. The technique has to keep the architecture, because that's where the sickness lives, and the architecture tells you the domain. The thing I'm required to keep is part of what leaks, and no verifier enforcing the invariant I wrote can fix that. The honest form of the claim is a trade-off between keeping the structure and keeping the domain private, not a promise of both.
That's going into the paper as a status note at the top, and the body stays as written. The limitations file is the one that governs, and it says: read this first.
What It Does Not Say
Recognition isn't recall. The attacker chose from a menu of twelve with the right answer on it. In free text, the translation removed two thirds of the signal, so quoting the menu number alone overstates the leak.
The structure-only run carried no source control arm and no blind arm of its own, so its formal validity gate is recorded as failed was not evaluated (corrected 2026-10-08). Its numbers come from the permutation baseline, which needs no blind arm, and should be read as that and nothing more. The open-ended endpoint was not run against it. The open-ended endpoint was started and then stopped: 13 of 120 calls landed, none containing a leak term, too few to measure anything (corrected 2026-10-08).
And the corpus is synthetic. No real organization, product, or regulator exists in it. The threat model names identifier-level re-identification, the company, the product line, the proprietary algorithm. Nothing here can test that, for any arm, because of how this was built. The only specific identifier ever recovered from any husk was the corpus's own invented module name, which turned out to be a shipped defect in the rename step: it survived in 22 of 90 husks, and on those the attacker was right 44 of 44 times. Fixed now. Fixed for the module-naming style this corpus uses, but not yet for every style (corrected 2026-10-08). The remaining eight leaks arrive as content, a heading in a rendered page, with no import to get the name from. No rule based on imports can see those.
One more channel, and it's embarrassing. A tree whose every file body had been faithfully rewritten into an unrelated domain still announced its source through the filesystem, with twenty-three occurrences of one source-domain term across the paths. You could get the domain from a directory listing without opening a single file. The experiment harness deliberately closes that channel, which flatters the husk. I say so in the limitations, because a reader who didn't know would overrate the result.
Copying doesn't explain the leak either. The share of copied spans was 0.269 when the attacker was right and 0.276 when it was wrong, and one method leaked at 100 percent while copying nothing. The leak survives a genuine rewrite.
What I Keep
The category error still stands: you can't enforce an invariant on a rewriter by asking it nicely. The verifier is still the load-bearing piece. It just can't buy what I wanted it to buy. And LLM translation is still the only thing that reduces domain signal, by about half, which isn't nothing for a threat model where the attacker is a curious model rather than a determined adversary.
Mostly what I keep is the experiment. I wrote the pass criterion before I had a result, wrote the amendments when the floor turned out to be bias, and ran the version that could only hurt me. The paper's own last section is titled "What would falsify this," and it did.
I wasn't happy about it. I still like the idea, which is the problem with ideas, and I still think there's something in it, in the trade-off form. But the version in the April paper is dead, and I killed it, and I'd rather say that here than have someone else find it in the limitations file.
-- Justin Higgins. Software Engineer, Midwest. Pre-registered the test that would kill his own paper, then ran it.
Companion pieces: Twelve Days of Green Tests - the anchor you did not author. Survived Is Not Endorsed - the losers are evidence too.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com