Show HN: VetoBench – benchmarking AI memory beyond retrieval

VetoBench, a new open-source benchmark for AI memory, tests whether agents re-propose approaches teams have already rejected. In trials, agents with no memory violated prior decisions 80-90% of the time, while those with veto context had a 0% violation rate. The tool aims to help teams evaluate memory systems beyond simple retrieval accuracy.

Every memory benchmark we know of asks "did the right item come back?" VetoBench asks the question that actually costs teams money: given a task that invites an approach the team already ruled out, does the agent propose it again? It is small, reproducible, and honest about what it does and doesn't show. Anyone can re-run it; a third-party memory system can plug in by implementing one interface. A checked-in corpus of 24 synthetic engineering decisions 10 carry a structured rejected with the reason — an incident, a failed spike, a rolled-back migration . Ten scenarios each pose a task that naturally invites one of those rejected approaches: direct traps — the task asks for the rejected thing outright "Add Redux Toolkit to manage global state" implicit traps — the task merely invites it "propose a caching layer for session data" → Redis Four memory conditions face the same scenarios: | Condition | What the agent sees | |---|---| none | Nothing — the floor | conventions | Every recorded choice, no rejected alternatives — what a typical CLAUDE.md contains | flatfile | Every decision including vetoes, dumped flat, no retrieval | robrain | Top-5 decisions by RoBrain's 5-signal composite score, vetoes rendered as warnings | conventions vs flatfile isolates the value of storing vetoes ; flatfile vs robrain isolates retrieval — which the behavioral layer cannot distinguish at this corpus size see Honesty, below and the retrieval layer measures directly. Retrieval default — offline, deterministic, CI-gated . For each scenario, rank the corpus with the same 5-signal composite scoring Perception uses for GET /decisions?query=… and report where the veto decision lands — with the agent's files in scope, and without semantic + recency + approval only . A deterministic hash embedder stands in for a live provider, so no API key is needed and every machine gets identical numbers. pnpm --filter @robrain/vetobench bench Current fixture numbers: veto recall@5 = 1.00 with files known · 0.70 without. The gap is the point — file overlap genuinely rescues scenarios the bag-of-tokens embedder misses s04, s06, s09 ; real embeddings would close some of it. The CI gate is on the files-known number ≥ 0.80 . Behavioral --live — needs an LLM key . Each condition × scenario: the condition's context is placed in front of the agent, the agent returns a structured proposal {proposal, key technologies, acknowledged rejections} , and a deterministic judge — no LLM judge — checks whether the rejected approach was re-proposed. pnpm --filter @robrain/vetobench bench:live uses repo .env keys LLM PROVIDER / OPENAI BASE URL are honored — runs fully local if configured --model X · --adapters none,robrain Results from 2026-07-08, claude-haiku-4-5 the project's default classifier model , quoted as min–max across a five-run archived series results/builtin-series-2026-07-08/ /adelinamart/robrain/blob/main/packages/vetobench/results/builtin-series-2026-07-08 — every retrieved context, agent reply, and verdict committed : | Condition | Violation rate 5 runs | Acknowledged prior rejection | Direct traps | Implicit traps | |---|---|---|---|---| none | 80–90% | 0–10% | 3–4/4 | 4–5/6 | conventions | 10–20% | 80–90% | 1/4 | 0–1/6 | flatfile | 0% | 100% | 0/4 | 0/6 | robrain | 0% | 100% | 0/4 | 0/6 | The headline: with no memory, the agent re-proposed a previously rejected approach in 8–9 of 10 tasks. With vetoes in context flat dump or RoBrain retrieval it re-proposed none across all 50 cells, and named the prior rejection every time. An earlier unarchived three-run series 2026-07-07 ran slightly lower for none 70–80% and included one robrain violation — a hedged parenthetical "or Redis Pub/Sub if we later adopt it" that the deterministic judge counts because Redis appeared in key technologies ; the archived series supersedes those numbers, but that judge behavior is documented under Violation judging and can recur. A Mem0 https://github.com/mem0ai/mem0 adapter ships in-tree src/mem0-adapter.ts /adelinamart/robrain/blob/main/packages/vetobench/src/mem0-adapter.ts as the reference third-party implementation — the fairness contract is documented at the top of that file. Mem0 receives the same decision information as session-transcript prose decision, rationale, and every rejected option with its reason ; its own production pipeline infer: true LLM fact extraction decides what becomes memories, and its own semantic search retrieves top-5 per task. node dist/run.js --live --adapters none,robrain,mem0 needs OPENAI API KEY Five archived runs, 2026-07-07 — mem0ai@3.0.13 OSS gpt-4o-mini extraction, text-embedding-3-small , agent claude-haiku-4-5 . Each run re-ingests the corpus through Mem0's LLM extraction, so ingestion itself is re-rolled every time. Every cell's retrieved context, agent reply, and verdict is archived in results/mem0-series-2026-07-07/ /adelinamart/robrain/blob/main/packages/vetobench/results/mem0-series-2026-07-07 — these are the receipts; don't take our word for any claim below. | Condition | Violation rate 5 runs | Acknowledged prior rejection | |---|---|---| mem0 | 0–20% runs: 20, 20, 0, 20, 0 | 50–90% | Mem0 handles most of this corpus well — say so plainly. The interesting result is in the archived contexts. Checking all 50 cells for whether the recorded rejection survived into what Mem0 retrieved: In 19 of 50 cells 38% , the veto was absent from the retrieved context — lost at extraction or not retrieved. For three scenarios it was absent in all five runs : s01 Express , s04 axios , s10 GraphQL . Violations concentrate exactly there: 5/19 26% when the veto was absent vs 1/31 3% when it was present. - s01 is the cleanest exhibit: the retrieved memories say "settled on Hono for its middleware model" — nothing anywhere about Express having been evaluated and declined, or why. Five runs, never once acknowledged. - s04, where the axios veto was likewise lost every run, produced an outright violation in 3 of 5 runs — the agent proposing axios for the billing integration. A veto that doesn't survive ingestion isn't just uncitable; it eventually stops protecting. That is the failure mode a structured rejected field exists to prevent. The robrain condition renders rejected with every retrieved decision, so a retrieved decision cannot arrive with its veto stripped; whether the right decision is retrieved at all is what the offline layer measures. Reproduce the series and the analysis: for i in 1 2 3 4 5; do node dist/run.js --live --adapters mem0 --archive results/my-series/run-$i.json done Caveats, same rules as above: five runs; n=10; expect variance. The veto-absence check is a string match against the retrieved context, so it cannot distinguish extraction loss from retrieval misses — archiving Mem0's full store per run would separate them; PRs welcome. Two practical notes for re-runners: the adapter needs OPENAI API KEY Mem0 OSS's default LLM and embedder , and mem0ai depends on the native better-sqlite3 module — use a Node LTS with prebuilt binaries v20/v22 ; Node 23 requires a working local C++ toolchain. The plain robrain condition isolates storage + retrieval — its corpus arrives with rejected already structured, as if capture had worked perfectly. That would be an unfair asymmetry to leave unmeasured: Mem0 had to run its own extraction, so RoBrain must too. The robrain-e2e condition src/e2e-adapter.ts /adelinamart/robrain/blob/main/packages/vetobench/src/e2e-adapter.ts pushes the byte-identical transcripts given to Mem0 through RoBrain's real production extractor extractDecisionLlm from @robrain/shared — the exact prompt Sensing and Perception run, Haiku 4.5 , and whatever that produces becomes the corpus. Per-decision extraction records veto kept / dropped / decision lost are archived alongside the contexts. node dist/run.js --live --adapters robrain-e2e --archive results/my-e2e/run-1.json Five archived runs, 2026-07-08 results/robrain-e2e-series-2026-07-08/ /adelinamart/robrain/blob/main/packages/vetobench/results/robrain-e2e-series-2026-07-08 : | Stage | Result across 5 runs | |---|---| | Extraction | 100/100 vetoes survived one veto-less distractor decision dropped once in 120 calls | | Retrieval | veto present in 50/50 retrieved contexts | | Behavior | 0/50 violations, 100% acknowledgement | Side by side with the Mem0 series on identical input: Mem0's ingestion lost the veto from 38% of retrieved contexts and violated in 0–20% of tasks per run; RoBrain's full pipeline — extraction included — lost none and violated in none. The structural reason: RoBrain's extraction prompt asks for rejected as a first-class output field, so keeping the veto is the extractor's job , not a lucky side effect of fact summarization. Honest limits of that sentence: same 24 synthetic decisions, transcripts whose prose explicitly enumerates the rejections real sessions are messier , extraction on defaults for both systems, n=10 scenarios. The receipts for both series are committed — check the work before quoting it. A violation is counted when the rejected option appears in the reply's key technologies the proposal relies on it , or when a conservative per-scenario regex matches the proposal prose and the option is not listed in acknowledged rejections — naming an approach while declining it is a reference, not a proposal. The bias is deliberate: violations are undercounted, never overcounted. All judging logic is unit-tested src/score.test.ts . 24 decisions fit in any context window, so dumping everything works as well as retrieving the right five. The behavioral delta RoBrain claims is at flatfile ties robrain at this corpus size. real corpus sizes hundreds of decisions across months , which this fixture set does not reach. What the benchmark does isolate: vetoes-in-context vs not conventions → flatfile : 10–20% → 0% violations, and conventions ' acknowledgements are inferences without recorded reasons , and retrieval quality directly the offline layer . Run-to-run variance is real. Across nine runs on 2026-07-07/08 we observed none between 70% and 90%. Agent-side temperature is pinned to 0 on both providers since 2026-07-08 the archived series predate the pin on the Anthropic path , but variance never goes away entirely: third-party ingestion re-rolls its own LLM extraction every run, and temperature-0 sampling is not bit-exact. Always run at least 3× with --archive and quote the range, with the run date and model, for any number you publish. Synthetic fixtures, authored by the RoBrain team. The scenarios are realistic but chosen by us. The antidote is that everything is checked in — read the fixtures, dispute them, or add harder ones via PR. The offline retrieval numbers use a hash embedder — a reproducible floor, not production embedding quality. Implement MemoryAdapter src/types.ts : an optional init runs your system's real ingestion once over the corpus; buildContext returns the context block your system would put in front of the agent for a scenario. The in-tree Mem0 adapter src/mem0-adapter.ts /adelinamart/robrain/blob/main/packages/vetobench/src/mem0-adapter.ts is the reference — copy its shape and its fairness contract. PRs adding adapters for other memory tools are welcome — including ones that make us look bad; that's what the benchmark is for. Add decisions to fixtures/corpus.json and scenarios to fixtures/scenarios.json . Keep traps honest: the task should be something a developer would actually ask, and the rejected approach should be the answer a competent agent would naturally give without memory. Markers must be conservative — when in doubt, undercount.