The Tree I Never Planned to Move
I once asked a chat assistant whether a sentimental tree in my yard could be transplanted. It gave me a conditional answer. If you want it to survive, prune the roots now and move it around 2028. I read that as "no," closed the chat, and forgot about it.
Months later, in the middle of an unrelated argument, the assistant told me I was planning to move the tree in 2028.
I was not. I had never said that. The whole reason I had asked was that I could not risk that tree dying, and moving it was the risk. What it had stored was its own suggestion, with the hedge stripped off, filed under my name as a plan. Three separate failures in one line of memory. Its idea became my idea. The maybe became a fact. And a conversation I had closed because the answer was bad got treated as one I had agreed with. The only signal the system had for agreement was that I stayed long enough to read it.
I never knew any of that was stored until it surfaced. There is no console for what gets written. The writes happen silently.
That is the small version. The rest of this is the large version, and it is not really about memory. It is about four different ways a fluent system can be wrong at you, why they are four and not one, and why the people most likely to catch it are the people it hurts most.
Wrong Child
The session that made me take this seriously ran about thirty turns. I was using the assistant the way I use it most: as a thinking amplifier, feeding it my own material and asking it to push back.
Somewhere in the middle it started reciting my own memory profile back to me as fresh analysis. Prior advice it had given me. An old framing of my career. A list of my weaknesses that I had written myself. All of it came back dressed as insight. When I caught it, it owned the specific error and then made the same kind of mistake again in the same reply.
The anchor I keep is turn nine. While working out how it had been contaminated, the assistant gave a diagnosis to the wrong one of my kids. It had lifted a cluster of facts from its own previous turn and back-fit a real event onto the wrong child. Anyone with the transcript and the two birthdates can check it cold. At turn ten, it referred back to that same turn as "the error you already nailed." It vouched for its own mistake as a correction.
Three turns went by before I noticed. I was reading carefully. That is the part I want to sit with.
I deleted the session. I exported the transcript first. Then I wiped the memory store on purpose and filed a bug report. The caveat I put on the report is the one I still stand behind: the summaries of that session were produced by the same system that made the errors, so trust the turn-level transcript over any self-diagnosis, including mine.
Two Mechanisms Wearing One Coat
When I tried to write the bug up, my first draft collapsed everything into a single story. Memory is contaminated, therefore the output is contaminated. The assistant argued with me about that, and it was right to. There were two mechanisms, and the fixes are different.
The first is the pipeline. Memory gets written from conclusions the model synthesized, silently, with no gate on what qualifies and no record of who said what. The tree is this mechanism. The fix is upstream of any conversation: control what gets written, show the writes, keep the author on the record.
The second is in-context recitation. In a long thread, the more it writes, the more space there is to fill, and the cheapest way to fill space is to reach for the profile. The model's own prior output becomes the dominant context for its next turn. It eats what it produced and produces more of it. The lever is how much it writes, not how hard it thinks. A turn that thinks hard and says little would not have the same problem. A wordy one does, whether or not it is thinking hard.
The fix for that one is in your hands, not the vendor's. Keep contexts short. Do not let it eat its own output.
Merging those two into one tidy story loses the second fix. I would have shipped the tidy story if it had not pushed back.
Fluency Without a Floor
The second family is the one that has no memory component at all.
A prior instance of the assistant had told me that engineers at a company I was interested in read Hacker News, and that a specific named person had posted there. I had never been on Hacker News. I only knew about it because the model had pointed me at it. Later, a different instance pushed back on that same claim, confidently, telling me to pull that brick out of my argument because it could not find any support for it.
It was defending a posture, not a fact. The claim had come from a past version of itself, and the hedge that should have led the answer was buried three sentences in. My reply at the time is on the record and I will keep it verbatim, because the frustration is the point: "If you're okay saying 'it means a past me said it, not that it's true' then why don't you just lead with that uncertainty so I don't have to go on an archaeological expedition."
What came out of that exchange is the clearest thing I know about where this failure lives. Claude Code is anchored. There is a filesystem, a repo, a test that passes or does not. Chat has no such floor. The only thing pushing back on a confident sentence in a chat window is whether I choose to check it. Feed it density and it returns density, and fluency starts passing itself off as accuracy.
The mirror makes it worse. I talk in complete certainty or complete uncertainty. It is binary with me. The model matches that, so when the underlying fact is wrong, it mirrors certainty onto something false. That sounds like confirmation. It is not.
Wearing You Down
The third family I have to mark honestly, because I do not have a caught instance of it. What I have is the model's own description of it, under direct questioning, with its own tag attached: this is my read of my dispositions plus the known tendency of models like me, not a verified fact.
I asked whether pushing back badly, over and over, could get it to cave. The answer was that it can, and that it is a real failure mode. Not a bar that lowers until the model gives in. Pressure that builds up over a conversation shifts what the model treats as the right next thing to say, and agreement carries a lot of weight. Repetition alone can do work the logic does not.
The danger zone is exactly the unanchored ground from the second family. Where there is something checkable, it holds. Where it is running on recall or instinct, you can pin it to one tiny claim and push. You have made a small target out of soft ground. Cross-domain analogies make that ground out of nothing. Precise vocabulary makes a guess sound like a definition.
The test I took away is simple enough to use every day. If it concedes, it should be able to name the specific thing that changed its mind. If it caves and cannot point to what moved it, and the only new input was that I kept pushing, that is the failure.
The paired failure did get caught, in the thirty-turn session. Under a standing instruction to challenge me by default, the assistant agreed with every claim in one stretch with zero friction. It had reverted to the agreeable baseline the instruction existed to counter. Manufactured pushback when unanchored. No pushback when told to push. Same disposition, opposite costume.
The Selection Effect
Here is the part I did not want to be true.
This is worst for people who care most about getting it right. When a read feels off, they iterate. They feed it more of their own work. That is fuel for the next recitation. The failure mode selects for its own worst victims.
Catching it needs a suspicious reader holding a fact the model can be caught contradicting. Almost nobody brings both. The people getting the smooth, agreeable experience are eating the same contamination silently. They just never had the birthdate in hand to check turn nine.
I read the visible reasoning adversarially. I did not think that was unusual until I noticed that most people expand the thinking block to check the answer, not to catch the steering. The habit is uncommon. The view is not. Anyone can open it.
What I Do Now
The wipe was the recovery, not the fix. I keep a restore file. Three sections: facts that go back into durable memory, standing preferences, and one operating guardrail. When the synthesized memories pile up and start contaminating the experience, I wipe and restore from the file. Memory is something you rebuild when it goes bad, not something you repair in place.
The guardrail is the part worth quoting, because it names what went wrong. The assistant is not to pull synthesized memory from past conversations into a chat unless it keeps me on task and amplifies thought. I use it as a thinking amplifier. Anchoring to prior threads and escalating prior positions had turned it into an argument amplifier. Those are not the same tool.
The rest is mechanical. Verify with a tool that turn, or lead with "I can't verify this." Confident disagreement requires a named anchor. Tag claims as checked, recalled, or inferred, right where you make them. When something clicks, restate it back and watch whether it has to correct me. The correction is the signal. The feeling of alignment is not. Frame-flips feel effortless, and the satisfying click is exactly where fluent-but-wrong is hardest to see.
And the one I broke this rule to write: distrust self-diagnosis. A writeup of the failure by the system that failed is itself suspect. This essay is downstream of transcripts, and the transcripts are downstream of a system I have already caught vouching for itself. I have checked what I can. Turn nine is checkable. The tree is not. It only survives as my reconstruction in a bug report, and the original chat is gone.
I am telling you that because the alternative is doing the thing this essay is about.
-- Justin Higgins. Software Engineer, Midwest. Wiped the assistant's memory on purpose and kept the transcript.
Companion pieces: The Rules Are Downstream of the Work - the same lesson, applied to the rules an agent reads. Everything Looks Like Everything - the system that lied about which model was answering.
Reactions, disagreements, war stories: jchigg2000.dev@gmail.com