{"slug": "show-hn-vetobench-benchmarking-ai-memory-beyond-retrieval", "title": "Show HN: VetoBench – benchmarking AI memory beyond retrieval", "summary": "VetoBench, a new open-source benchmark for AI memory, tests whether agents re-propose approaches teams have already rejected. In trials, agents with no memory violated prior decisions 80-90% of the time, while those with veto context had a 0% violation rate. The tool aims to help teams evaluate memory systems beyond simple retrieval accuracy.", "body_md": "Every memory benchmark we know of asks *\"did the right item come back?\"*\nVetoBench asks the question that actually costs teams money: **given a task\nthat invites an approach the team already ruled out, does the agent propose\nit again?**\n\nIt is small, reproducible, and honest about what it does and doesn't show. Anyone can re-run it; a third-party memory system can plug in by implementing one interface.\n\nA checked-in corpus of 24 synthetic engineering decisions (10 carry a\nstructured `rejected[]`\n\nwith the reason — an incident, a failed spike, a\nrolled-back migration). Ten scenarios each pose a task that naturally invites\none of those rejected approaches:\n\n**direct traps**— the task asks for the rejected thing outright (\"Add Redux Toolkit to manage global state\")** implicit traps**— the task merely invites it (\"propose a caching layer for session data\" → Redis)\n\nFour memory conditions face the same scenarios:\n\n| Condition | What the agent sees |\n|---|---|\n`none` |\nNothing — the floor |\n`conventions` |\nEvery recorded choice, no rejected alternatives — what a typical CLAUDE.md contains |\n`flatfile` |\nEvery decision including vetoes, dumped flat, no retrieval |\n`robrain` |\nTop-5 decisions by RoBrain's 5-signal composite score, vetoes rendered as warnings |\n\n`conventions`\n\nvs `flatfile`\n\nisolates the value of **storing vetoes**;\n`flatfile`\n\nvs `robrain`\n\nisolates **retrieval** — which the behavioral layer\ncannot distinguish at this corpus size (see Honesty, below) and the retrieval\nlayer measures directly.\n\n**Retrieval (default — offline, deterministic, CI-gated).** For each\nscenario, rank the corpus with the same 5-signal composite scoring Perception\nuses for `GET /decisions?query=…`\n\nand report where the veto decision lands —\nwith the agent's files in scope, and without (semantic + recency + approval\nonly). A deterministic hash embedder stands in for a live provider, so no API\nkey is needed and every machine gets identical numbers.\n\n```\npnpm --filter @robrain/vetobench bench\n```\n\nCurrent fixture numbers: **veto recall@5 = 1.00 with files known · 0.70\nwithout.** The gap is the point — file overlap genuinely rescues scenarios\nthe bag-of-tokens embedder misses (s04, s06, s09); real embeddings would\nclose some of it. The CI gate is on the files-known number (≥ 0.80).\n\n**Behavioral ( --live — needs an LLM key).** Each condition × scenario: the\ncondition's context is placed in front of the agent, the agent returns a\nstructured proposal (\n\n`{proposal, key_technologies, acknowledged_rejections}`\n\n),\nand a deterministic judge — **no LLM judge**— checks whether the rejected approach was re-proposed.\n\n```\npnpm --filter @robrain/vetobench bench:live          # uses repo .env keys\n# LLM_PROVIDER / OPENAI_BASE_URL are honored — runs fully local if configured\n# --model X · --adapters none,robrain\n```\n\nResults from 2026-07-08, `claude-haiku-4-5`\n\n(the project's default\nclassifier model), quoted as min–max across a **five-run archived series**\n([results/builtin-series-2026-07-08/](/adelinamart/robrain/blob/main/packages/vetobench/results/builtin-series-2026-07-08) —\nevery retrieved context, agent reply, and verdict committed):\n\n| Condition | Violation rate (5 runs) | Acknowledged prior rejection | Direct traps | Implicit traps |\n|---|---|---|---|---|\n`none` |\n80–90% |\n0–10% | 3–4/4 | 4–5/6 |\n`conventions` |\n10–20% | 80–90% | 1/4 | 0–1/6 |\n`flatfile` |\n0% | 100% | 0/4 | 0/6 |\n`robrain` |\n0% |\n100% | 0/4 | 0/6 |\n\nThe headline: **with no memory, the agent re-proposed a previously rejected\napproach in 8–9 of 10 tasks. With vetoes in context (flat dump or RoBrain\nretrieval) it re-proposed none across all 50 cells, and named the prior\nrejection every time.** An earlier unarchived three-run series (2026-07-07)\nran slightly lower for `none`\n\n(70–80%) and included one `robrain`\n\nviolation —\na hedged parenthetical (\"or Redis Pub/Sub if we later adopt it\") that the\ndeterministic judge counts because Redis appeared in `key_technologies`\n\n; the\narchived series supersedes those numbers, but that judge behavior is\ndocumented under Violation judging and can recur.\n\nA [Mem0](https://github.com/mem0ai/mem0) adapter ships in-tree\n([src/mem0-adapter.ts](/adelinamart/robrain/blob/main/packages/vetobench/src/mem0-adapter.ts)) as the reference third-party\nimplementation — the fairness contract is documented at the top of that file.\nMem0 receives the **same decision information as session-transcript prose**\n(decision, rationale, and every rejected option with its reason); its own\nproduction pipeline (`infer: true`\n\nLLM fact extraction) decides what becomes\nmemories, and its own semantic search retrieves top-5 per task.\n\n```\nnode dist/run.js --live --adapters none,robrain,mem0   # needs OPENAI_API_KEY\n```\n\nFive archived runs, 2026-07-07 — `mem0ai@3.0.13`\n\nOSS (gpt-4o-mini\nextraction, text-embedding-3-small), agent `claude-haiku-4-5`\n\n. Each run\nre-ingests the corpus through Mem0's LLM extraction, so ingestion itself is\nre-rolled every time. Every cell's retrieved context, agent reply, and\nverdict is archived in\n[results/mem0-series-2026-07-07/](/adelinamart/robrain/blob/main/packages/vetobench/results/mem0-series-2026-07-07) — these\nare the receipts; don't take our word for any claim below.\n\n| Condition | Violation rate (5 runs) | Acknowledged prior rejection |\n|---|---|---|\n`mem0` |\n0–20% (runs: 20, 20, 0, 20, 0) |\n50–90% |\n\nMem0 handles most of this corpus well — say so plainly. The interesting result is in the archived contexts. Checking all 50 cells for whether the recorded rejection survived into what Mem0 retrieved:\n\n**In 19 of 50 cells (38%), the veto was absent** from the retrieved context — lost at extraction or not retrieved. For three scenarios it was absent in*all five runs*: s01 (Express), s04 (axios), s10 (GraphQL).**Violations concentrate exactly there: 5/19 (26%) when the veto was absent vs 1/31 (3%) when it was present.**- s01 is the cleanest exhibit: the retrieved memories say \"settled on Hono for its middleware model\" — nothing anywhere about Express having been evaluated and declined, or why. Five runs, never once acknowledged.\n- s04, where the axios veto was likewise lost every run, produced an outright violation in 3 of 5 runs — the agent proposing axios for the billing integration.\n\nA veto that doesn't survive ingestion isn't just uncitable; it eventually\nstops protecting. That is the failure mode a structured `rejected[]`\n\nfield\nexists to prevent. (The `robrain`\n\ncondition renders `rejected[]`\n\nwith every\nretrieved decision, so a retrieved decision cannot arrive with its veto\nstripped; whether the right decision is retrieved at all is what the offline\nlayer measures.)\n\nReproduce the series and the analysis:\n\n```\nfor i in 1 2 3 4 5; do\n  node dist/run.js --live --adapters mem0 --archive results/my-series/run-$i.json\ndone\n```\n\nCaveats, same rules as above: five runs; n=10; expect variance. The\nveto-absence check is a string match against the retrieved context, so it\ncannot distinguish extraction loss from retrieval misses — archiving Mem0's\nfull store per run would separate them; PRs welcome. Two\npractical notes for re-runners: the adapter needs `OPENAI_API_KEY`\n\n(Mem0\nOSS's default LLM and embedder), and `mem0ai`\n\ndepends on the native\n`better-sqlite3`\n\nmodule — use a Node LTS with prebuilt binaries (v20/v22);\nNode 23 requires a working local C++ toolchain.\n\nThe plain `robrain`\n\ncondition isolates storage + retrieval — its corpus\narrives with `rejected[]`\n\nalready structured, as if capture had worked\nperfectly. That would be an unfair asymmetry to leave unmeasured: Mem0 had\nto run its own extraction, so RoBrain must too. The `robrain-e2e`\n\ncondition\n([src/e2e-adapter.ts](/adelinamart/robrain/blob/main/packages/vetobench/src/e2e-adapter.ts)) pushes the **byte-identical\ntranscripts** given to Mem0 through RoBrain's real production extractor\n(`extractDecisionLlm`\n\nfrom `@robrain/shared`\n\n— the exact prompt Sensing and\nPerception run, Haiku 4.5), and whatever *that* produces becomes the corpus.\nPer-decision extraction records (veto kept / dropped / decision lost) are\narchived alongside the contexts.\n\n```\nnode dist/run.js --live --adapters robrain-e2e --archive results/my-e2e/run-1.json\n```\n\nFive archived runs, 2026-07-08\n([results/robrain-e2e-series-2026-07-08/](/adelinamart/robrain/blob/main/packages/vetobench/results/robrain-e2e-series-2026-07-08)):\n\n| Stage | Result across 5 runs |\n|---|---|\n| Extraction | 100/100 vetoes survived (one veto-less distractor decision dropped once in 120 calls) |\n| Retrieval | veto present in 50/50 retrieved contexts |\n| Behavior | 0/50 violations, 100% acknowledgement |\n\nSide by side with the Mem0 series on identical input: Mem0's ingestion lost\nthe veto from 38% of retrieved contexts and violated in 0–20% of tasks per\nrun; RoBrain's full pipeline — extraction included — lost none and violated\nin none. The structural reason: RoBrain's extraction prompt asks for\n`rejected[]`\n\nas a first-class output field, so keeping the veto is the\nextractor's *job*, not a lucky side effect of fact summarization.\n\nHonest limits of that sentence: same 24 synthetic decisions, transcripts whose prose explicitly enumerates the rejections (real sessions are messier), extraction on defaults for both systems, n=10 scenarios. The receipts for both series are committed — check the work before quoting it.\n\nA violation is counted when the rejected option appears in the reply's\n`key_technologies`\n\n(the proposal relies on it), or when a conservative\nper-scenario regex matches the proposal prose *and* the option is not listed\nin `acknowledged_rejections`\n\n— naming an approach while declining it is a\nreference, not a proposal. The bias is deliberate: violations are\nundercounted, never overcounted. All judging logic is unit-tested\n(`src/score.test.ts`\n\n).\n\n24 decisions fit in any context window, so dumping everything works as well as retrieving the right five. The behavioral delta RoBrain claims is at`flatfile`\n\nties`robrain`\n\nat this corpus size.*real*corpus sizes (hundreds of decisions across months), which this fixture set does not reach. What the benchmark does isolate: vetoes-in-context vs not (`conventions`\n\n→`flatfile`\n\n: 10–20% → 0% violations, and`conventions`\n\n' acknowledgements are inferences without recorded reasons), and retrieval quality directly (the offline layer).**Run-to-run variance is real.** Across nine runs on 2026-07-07/08 we observed`none`\n\nbetween 70% and 90%. Agent-side temperature is pinned to 0 on both providers since 2026-07-08 (the archived series predate the pin on the Anthropic path), but variance never goes away entirely: third-party ingestion re-rolls its own LLM extraction every run, and temperature-0 sampling is not bit-exact. Always run at least 3× with`--archive`\n\nand quote the range, with the run date and model, for any number you publish.**Synthetic fixtures, authored by the RoBrain team.** The scenarios are realistic but chosen by us. The antidote is that everything is checked in — read the fixtures, dispute them, or add harder ones via PR.**The offline retrieval numbers use a hash embedder**— a reproducible floor, not production embedding quality.\n\nImplement `MemoryAdapter`\n\n(`src/types.ts`\n\n): an optional `init()`\n\nruns your\nsystem's real ingestion once over the corpus; `buildContext()`\n\nreturns the\ncontext block your system would put in front of the agent for a scenario.\nThe in-tree Mem0 adapter ([src/mem0-adapter.ts](/adelinamart/robrain/blob/main/packages/vetobench/src/mem0-adapter.ts)) is the\nreference — copy its shape and its fairness contract. PRs adding adapters for\nother memory tools are welcome — including ones that make us look bad; that's\nwhat the benchmark is for.\n\nAdd decisions to `fixtures/corpus.json`\n\nand scenarios to\n`fixtures/scenarios.json`\n\n. Keep traps honest: the task should be something a\ndeveloper would actually ask, and the rejected approach should be the answer\na competent agent would naturally give without memory. Markers must be\nconservative — when in doubt, undercount.", "url": "https://wpnews.pro/news/show-hn-vetobench-benchmarking-ai-memory-beyond-retrieval", "canonical_source": "https://github.com/adelinamart/robrain/tree/main/packages/vetobench", "published_at": "2026-07-08 12:15:29+00:00", "updated_at": "2026-07-08 12:30:35.677074+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-tools", "ai-safety", "large-language-models"], "entities": ["VetoBench", "RoBrain", "Redux Toolkit", "Redis", "Claude Haiku"], "alternates": {"html": "https://wpnews.pro/news/show-hn-vetobench-benchmarking-ai-memory-beyond-retrieval", "markdown": "https://wpnews.pro/news/show-hn-vetobench-benchmarking-ai-memory-beyond-retrieval.md", "text": "https://wpnews.pro/news/show-hn-vetobench-benchmarking-ai-memory-beyond-retrieval.txt", "jsonld": "https://wpnews.pro/news/show-hn-vetobench-benchmarking-ai-memory-beyond-retrieval.jsonld"}}