The state axis: why agent benchmarks keep measuring amnesiac models

A developer argues that coding-agent benchmarks fail to measure the impact of persistent project memory, a third axis beyond model and harness. The post highlights Engrim, a local-first SQLite memory engine for coding agents, as evidence that state matters, and calls for parameterized evals that test agents with varying memory states.

I keep hammering the point that any coding-agent score is model + harness, not model alone. Same context-carryover rules, same note convention, same tool loop, same judge, or the comparison is garbage. Engrim github.com/timgordontg/engrim is a useful reminder that there's a third axis I've been underselling: state. It's a local-first SQLite memory engine for Claude Code, Cursor, Windsurf, Codex. Project-scoped, embeds records plus SQLite FTS, persists decisions and rationale between sessions. Hybrid retrieval, no cloud. Cross-session recall is the whole pitch. And here's the thing: that pitch only works if the agent actually gets better as its memory grows. Engrim exists because the authors believe persistent project context beats a cold context window on every new session. Which is exactly what almost no benchmark measures. Look at how agent evals are actually built. They load a repo, drop you in at an issue, and score the patch. Empty CLAUDE.md. No cursor rules. No memory file. Every run starts the model at the same amnesia. That's deliberate, for reproducibility. You can't run a score if the model's head is full of last run's secrets. But clean reproducibility bought a distorted measurement. The agent you benchmark cold is not the agent your team runs after a month of accumulated project memory. The more an agent keeps between sessions, the wider that gap gets. A 30-minute cold-start eval tells you almost nothing about how an agent with six months of ingested project decisions will handle a real migration. This is the same structural blindness as correlated judges scoring a session: you've measured one blind spot and called it a committee. Here you've measured one memory state empty and called it the model. The variable that actually drives production output, prior state, is exactly the variable the eval pins to zero. The fix doesn't have to destroy reproducibility. You can parameterize it instead: run a task three ways, empty context, a small hand-curated project memory, and a large one. Watch the score move. Tell me which tool's score moved the most, and I'll tell you which one you should run in a long-lived repo. That result is more useful than a hundred clean cold-start tables. Engrim's not the product here. It's the tell that this axis matters and nobody's benchmarking it. If your eval measures amnesiac models, its scores are a floor, and the gap to production is your real signal.