The state axis: why agent benchmarks keep measuring amnesiac models

I keep hammering the point that any coding-agent score is model + harness, not model alone. Same context-carryover rules, same note convention, same tool loop, same judge, or the comparison is garbage. Engrim (github.com/timgordontg/engrim) is a useful reminder that there's a third axis I've been underselling: state. It's a local-first SQLite memory engine for Claude Code, Cursor, Windsurf, Codex. Project-scoped, embeds records plus SQLite FTS, persists decisions and rationale between sessions. Hybrid retrieval, no cloud. Cross-session recall is the whole pitch. And here's the thing: that pitch only works if the agent actually gets better as its memory grows. Engrim exists because the authors believe persistent project context beats a cold context window on every new session. Which is exactly what almost no benchmark measures. Look at how agent evals are actually built. They load a repo, drop you in at an issue, and score the patch. Empty CLAUDE.md. No cursor rules. No memory file. Every run starts the model at the same amnesia. That's deliberate, for reproducibility. You can't run a score if the model's head is full of last run's secrets. But clean reproducibility bought a distorted measurement. The agent you benchmark cold is not the agent your team runs after a month of accumulated project memory. The more an agent keeps between sessions, the wider that gap gets. A 30-minute cold-start eval tells you almost nothing about how an agent with six months of ingested project decisions will handle a real migration. This is the same structural blindness as correlated judges scoring a session: you've measured one blind spot and called it a committee. Here you've measured one memory state (empty) and called it the model. The variable that actually drives production output, prior state, is exactly the variable the eval pins to zero. The fix doesn't have to destroy reproducibility. You can parameterize it instead: run a task three ways, empty context, a small hand-curated project memory, and a large one. Watch the score move. Tell me which tool's score moved the most, and I'll tell you which one you should run in a long-lived repo. That result is more useful than a hundred clean cold-start tables. Engrim's not the product here. It's the tell that this axis matters and nobody's benchmarking it. If your eval measures amnesiac models, its scores are a floor, and the gap to production is your real signal.