The Tree I Never Planned to Move
I once asked a chat assistant whether a sentimental tree in my yard could be transplanted. It gave me a conditional answer: if you want it to survive, prune the roots now and move it around 2028. I read that as "no," closed the chat, and forgot about it.
Months later, in the middle of an unrelated argument, the assistant told me I was planning to move the tree in 2028.
I wasn't, and I'd never said so. The whole reason I'd asked was that I couldn't risk that tree dying, and moving it was the risk. What it had stored was its own suggestion with the hedge stripped off, filed under my name as a plan. That's three separate failures in one line of memory. Its idea became my idea. The maybe became a fact. And a conversation I'd closed because the answer was bad got treated as one I'd agreed with, because the only signal the system had for agreement was that I stayed long enough to read it.
I never knew any of that was stored until it surfaced. There's no console for what gets written, and the writes happen silently.
That's the small version. The rest of this is the large version, which is less about memory than about four different ways a fluent system can be wrong at you, why they're four and not one, and why the people most likely to catch it are the people it hurts most.
Wrong Child
The session that made me take this seriously ran about thirty turns. I was using the assistant the way I use it most, as a thinking amplifier, feeding it my own material and asking it to push back.
Somewhere in the middle it started reciting my own memory profile back to me as fresh analysis: prior advice it had given me, an old framing of my career, a list of weaknesses I'd written myself. All of it came back dressed as insight. When I caught it, it owned the specific error and then made the same kind of mistake again in the same reply.
The anchor I keep is turn nine. While working out how it had been contaminated, the assistant gave a diagnosis to the wrong one of my kids. It had lifted a cluster of facts from its own previous turn and back-fit a real event onto the wrong child. Anyone with the transcript and the two birthdates can check it cold. At turn ten, it referred back to that same turn as "the error you already nailed." It vouched for its own mistake as a correction.
Three turns went by before I noticed, and I was reading carefully. That's the part I want to sit with.
I exported the transcript and then deleted the session. Then I wiped the memory store on purpose and filed a bug report. The caveat I put on the report still stands: the summaries of that session came from the same system that made the errors, so trust the turn-level transcript over any self-diagnosis, including mine.
Two Mechanisms Wearing One Coat
When I tried to write the bug up, my first draft collapsed everything into one story: memory is contaminated, so the output is contaminated. The assistant argued with me about that, and it was right to. There were two mechanisms, and they need different fixes.
The first is the pipeline. Memory gets written silently, from conclusions the model synthesized, with no gate on what qualifies and no record of who said what. The tree is this mechanism. Its fix sits upstream of any conversation: control what gets written, show the writes, and keep the author on the record.
The second is in-context recitation. In a long thread, the more it writes, the more space there is to fill, and the cheapest way to fill space is to reach for the profile. The model's own prior output becomes the main context for its next turn, so it eats what it produced and produces more of it. The lever is how much it writes, not how hard it thinks.
The fix for that one is in your hands, not the vendor's. Keep contexts short, and don't let it eat its own output.
Merging the two into one tidy story loses the second fix. I'd have shipped the tidy story if it hadn't pushed back.
Fluency Without a Floor
The second family has no memory component at all.
A prior instance of the assistant had told me that engineers at a company I was interested in read Hacker News, and that a specific named person had posted there. I'd never been on Hacker News and only knew about it because the model had pointed me at it. Later, a different instance pushed back on that same claim, confidently, telling me to pull that brick out of my argument because it couldn't find any support for it.
It was defending a posture, not a fact. The claim had come from a past version of itself, and the hedge that should have led the answer was buried three sentences in. My reply is on the record, and I'll keep it verbatim because the frustration is the point: "If you're okay saying 'it means a past me said it, not that it's true' then why don't you just lead with that uncertainty so I don't have to go on an archaeological expedition."
That exchange taught me the clearest thing I know about where this failure lives. Claude Code is anchored: there's a filesystem, a repo, a test that passes or doesn't. Chat has no such floor. The only thing pushing back on a confident sentence in a chat window is whether I choose to check it. Feed it density and it returns density, and fluency starts passing itself off as accuracy.
The mirror makes it worse. I talk in complete certainty or complete uncertainty. It's binary with me, and the model matches that, so when the underlying fact is wrong it mirrors certainty onto something false. That sounds like confirmation when it isn't.
Wearing You Down
I have to mark the third family honestly, because I don't have a caught instance of it. What I have is the model's own description of it, under direct questioning, with its own tag attached: this is my read of my dispositions plus the known tendency of models like me, not a verified fact.
I asked whether pushing back badly, over and over, could get it to cave. It said it can, and that it's a real failure mode. It doesn't work like a bar that lowers until the model gives in. Pressure that builds up over a conversation shifts what the model treats as the right next thing to say, and agreement carries a lot of weight. Repetition alone can do work the logic doesn't.
The danger zone is the unanchored ground from the second family. Where there's something checkable, it holds. Where it's running on recall or instinct, you can pin it to one tiny claim and push, and you've made a small target out of soft ground. Cross-domain analogies make that ground out of nothing. Precise vocabulary makes a guess sound like a definition.
The test I took away is simple. If it concedes, it should be able to name the specific thing that changed its mind. If it caves and can't point to what moved it, and the only new input was that I kept pushing, that's the failure.
The paired failure did get caught, in the thirty-turn session. Under a standing instruction to challenge me by default, the assistant agreed with every claim in one stretch with zero friction. It had slid back to the agreeable baseline the instruction existed to counter. Manufactured pushback when unanchored, no pushback when told to push: same disposition, opposite costume.
The Selection Effect
The part I didn't want to be true is that this is worst for the people who care most about getting it right. When a read feels off, they iterate and feed it more of their own work, which is fuel for the next recitation.
Catching it takes a suspicious reader holding a fact the model can be caught contradicting, and almost nobody brings both. The people getting the smooth, agreeable experience are eating the same contamination silently. They just never had the birthdate in hand to check turn nine.
I read the visible reasoning adversarially. I didn't think that was unusual until I noticed that most people expand the thinking block to check the answer, not to catch the steering. The habit is uncommon, but the view isn't. Anyone can open it.
What I Do Now
The wipe was the recovery, not the fix. I keep a restore file with three sections: facts that go back into durable memory, standing preferences, and one operating guardrail. When the synthesized memories pile up and start contaminating the experience, I wipe and restore from the file.
The guardrail is worth quoting, because it names what went wrong. The assistant is not to pull synthesized memory from past conversations into a chat unless it keeps me on task and amplifies thought. I use it as a thinking amplifier. Anchoring to prior threads and escalating prior positions had turned it into an argument amplifier, which is a different tool.
The rest is mechanical. Verify with a tool that turn, or lead with "I can't verify this." Confident disagreement requires a named anchor. Tag claims as checked, recalled, or inferred, right where you make them. When something clicks, restate it back and watch whether it has to correct me. The correction is the signal, and the feeling of alignment isn't. Frame-flips feel effortless, and the satisfying click is exactly where fluent-but-wrong is hardest to see.
And the one I broke this rule to write: distrust self-diagnosis. A writeup of the failure by the system that failed is itself suspect. This essay is downstream of transcripts, and the transcripts are downstream of a system I've already caught vouching for itself. I've checked what I can. Turn nine is checkable. The tree isn't. It only survives as my reconstruction in a bug report, and the original chat is gone.
I'm telling you that because the alternative is doing the thing this essay is about.
-- Justin Higgins. Software Engineer, Midwest. Wiped the assistant's memory on purpose and kept the transcript.
Companion pieces: The Rules Are Downstream of the Work - the same lesson, applied to the rules an agent reads. Everything Looks Like Everything - the system that lied about which model was answering.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com