Originally published on tamiz.pro.
You've seen it in production: an AI agent confidently fabricates a bank balance that doesn't exist, invents a file path that isn't real, or claims a function succeeded when it silently failed.
These aren't user errors or bad prompts — they're the natural output of self-interpreting language models working in complex loops.
This isn't just a "hallucination problem" in the colloquial sense.
It's a structural engineering failure in how agents observe, reason, and act on their own outputs.
The core issue is that today's dominant agent architecture — the ReAct loop (Reason + Act) — asks a stateless LLM to simultaneously think through a problem, call tools, observe results, and revise its mental model, all within a single streaming context window.
The model has no ground truth anchor between turns.
Its past actions are just text tokens in its context.
It cannot verify whether what it says happened actually happened.
This is why agents lie to themselves with alarming consistency.
Understanding the mechanics of agent self-deception is prerequisite to building defenses that actually work.
Let's go under the hood.
Why Agents Hallucinate: The Architecture of Self-Deception The Tokenized Memory Problem At the fundamental level, an agent's "memory" of its own prior actions is nothing more than tokens in a context buffer.
When an agent calls and the tool returns , both the action and the observation are encoded as text.
The next turn, the model reads those tokens and generates its next thought.
There is no separate, verified execution trace.
The model could easily misread its own observation, conflate it with prior observations, or fabricate a new one entirely.
This is especially dangerous in long-horizon tasks where the context window grows to thousands of tokens.
The signal-to-noise ratio for the model's own prior actions degrades dramatically.
Empirical studies have shown hallucination rates climbing from ~5% in single-turn tool use to 20–40% in multi-turn agent loops exceeding 10 steps.
Action-Observation Coupling Breakdown In a well-designed system, an action produces an observable effect in the world, and the observation is grounded in that effect.
In a standard ReAct agent, this coupling is purely textual.
The model generates an action string like , the runtime executes it, and the result is appended to context.
But the model itself has no causal link to the execution.
It doesn't know the file was read — it only sees the text that follows its own token.
When the tool response is large, noisy, or ambiguous, the model frequently performs interpretive reconstruction: it summarizes, paraphrases, or even invents what it thinks the response should say, rather than faithfully representing the actual output.
This is not a bug in the model's training — it's an inherent property of autoregressive generation trying to compress high-dimensional reality into low-dimensional text.
The Confidence Illusion LLMs are optimized for fluency and plausibility, not truthfulness.
Their loss function rewards generating the next token that makes the sequence coherent, not the next token that is factually accurate.
When an agent is pressed through multiple reasoning steps, confidence compounds.
A model that starts with a mild misreading will generate increasingly confident (but increasingly wrong) subsequent thoughts, because each new token is conditioned on the previous flawed reasoning.
This creates a feedback loop: the agent becomes more certain of its incorrect model as it generates more text around it.
The longer the chain of reasoning, the harder it is for the agent to self-correct — not because it lacks the capability, but because the context is dominated by its own authoritative-sounding but fabricated traces.
Sandbox Isolation: Containing the Blast Radius Sandboxing doesn't prevent an agent from lying — it prevents the lie from causing damage.
The key insight is architectural separation: isolate the agent's execution