{"slug": "open-call-test-your-agent-memory-layer-on-an-adversarial-coding-benchmark", "title": "Open call: test your agent memory layer on an adversarial coding benchmark", "summary": "A developer has released AMB, an open and preregistered benchmark that tests whether memory layers in coding agents improve real coding work, using executable tests to grade artifacts in a sandboxed repository. The public feed contains 195 session transcripts, and a hard corpus scales to 4,900 documents with roughly 143,000 chunks. The benchmark is adversarially constructed with absent, stale, superseded, contradictory, adjacent, and irrelevant memories, and small memory vendors are invited to test their systems at no cost.", "body_md": "I built AMB, an open and preregistered benchmark for memory layers in coding agents.\n\nMost memory benchmarks test whether a system retrieves a relevant chunk. AMB tests whether that memory actually helps an agent complete real coding work, or causes a worse change.\n\nClaude Code runs inside a sandboxed repository, and executable tests grade the resulting artifact. The benchmark is adversarially constructed with absent, stale, superseded, contradictory, adjacent, and irrelevant memories.\n\nThe current public feed contains 195 pre-authored session transcripts. A separate hard corpus record scales to 4,900 documents, including 196 real documents and 4,704 synthetic distractors, with roughly 143,000 chunks.\n\nI am inviting small memory vendors to test their systems. There is no participation fee. Vendors can review the adapter contract, run with their own credentials, and publish their configuration and results. Negative results are welcome.\n\nThe current suite mainly measures the read path. It does not yet measure whether a memory layer learns from the agent’s own work across sessions, and no multi product ranking has been published.\n\nRepository:\n\nWhat would you want this benchmark to measure before trusting its results?", "url": "https://wpnews.pro/news/open-call-test-your-agent-memory-layer-on-an-adversarial-coding-benchmark", "canonical_source": "https://discuss.huggingface.co/t/open-call-test-your-agent-memory-layer-on-an-adversarial-coding-benchmark/179762#post_1", "published_at": "2026-09-02 14:01:04+00:00", "updated_at": "2026-09-02 14:27:14.352609+00:00", "lang": "en", "topics": ["ai-research", "ai-tools", "developer-tools", "machine-learning"], "entities": ["AMB", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/open-call-test-your-agent-memory-layer-on-an-adversarial-coding-benchmark", "markdown": "https://wpnews.pro/news/open-call-test-your-agent-memory-layer-on-an-adversarial-coding-benchmark.md", "text": "https://wpnews.pro/news/open-call-test-your-agent-memory-layer-on-an-adversarial-coding-benchmark.txt", "jsonld": "https://wpnews.pro/news/open-call-test-your-agent-memory-layer-on-an-adversarial-coding-benchmark.jsonld"}}