Originally published on tamiz.pro.
You've spent weeks tuning your system prompt.
You've tried chain-of-thought, ReAct, and tree-of-thought prompting.
You've benchmarked GPT-4o against Claude 3.5 Sonnet and nothing clicks.
Your agent still loses context, contradicts itself across turns, and feels like it's starting fresh every time a user comes back.
Here's the uncomfortable truth: your agent doesn't have a reasoning problem.
It has a memory problem.
The models available today are remarkably capable at reasoning within the context window they're given.
The gap between a "smart" agent and a "breaks after three turns" agent almost never traces back to the model's logical capabilities.
It traces back to what the agent remembers, how it remembers it, and whether that memory survives the transition from demo to production.
This is a deep-dive into production-grade agent state — the architecture, patterns, and trade-offs that separate prototypes from systems that handle real users across real sessions.
1.
The Reasoning Myth Before we fix the memory problem, let's kill the reasoning narrative once and for all.
What the benchmarks actually measure When we evaluate LLM reasoning, we're measuring something narrow: given a prompt and a constrained set of context tokens, how well does the model solve a defined problem?
Chain-of-thought papers, MATH benchmarks, GPQA, LiveCodeBench — these are all stateless evaluations.
The model sees the question, reasons through it, and produces an answer.
Nothing persists.
Nothing accumulates.
Real agent work is fundamentally different.
An agent operates across multiple turns, multiple tools, and distributions of information that arrive incrementally.
The reasoning challenge isn't "can the model think?" — it's "can the model think about what it already knows while also figuring out what to do next?" The stateless model, the stateful world Consider this conversation: User (turn 1): "I need to book a flight from SFO to NYC next Tuesday for under $400." Agent: Calls search tool → finds flights → returns results User (turn 2): "Which one has the shortest layover?" Agent: Calls another tool or reasons from prior results User (turn 3): "Actually, change the destination to Newark." Agent: ...what does it know about the original request?
At turn 3, the agent needs to reconcile a modified goal against previous tool results, intermediate conclusions, and user preferences expressed across turns.
That's not a reasoning deficiency — that's a state management deficiency.
No amount of prompt engineering on the model's reasoning ability fixes this.
The information simply isn't there to reason about.
Why "just give it more context" doesn't work You could throw every prior turn into the context window and hope the model tracks it.
This fails in production for three reasons: Context window is expensive.
Every token you send costs money.
A 10-turn conversation with tool results can easily consume 15,000–50,000 tokens per call.
At scale, this is bankrupting.
Context window is noisy.
Retrieving 40K tokens of conversation history doesn't mean the model attends to the right 40K tokens.
Attention mechanisms dilute across long contexts.
Critical details from turn 1 get buried.
Context window doesn't persist across sessions.
When the user returns tomorrow, last week's conversation is gone unless you explicitly saved it somewhere and retrieved it again.
The solution isn't bigger windows.
It's better memory architecture.
2.
What Agent State Actually Is Agent state is not a single thing.
It's a composite of several distinct but interacting layers.
Confusing these layers is the root cause of most production failures. 2.1 Working Memory (Episodic) The short-lived, session-bound state that drives the current interaction.
This includes: Current goal and sub-goals Tool call history (what was called, what returned) Intermediate conclusions and plans User intent as expressed so far in the conversation This is typically held in the context wi