{"slug": "when-does-memory-help-a-cost-aware-evaluation-of-long-term-memory-in-tool-using", "title": "When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents", "summary": "A new benchmark called MERIT (Memory Evaluation for Realistic Instrumented Tasks) shows that long-term memory lifts dependent-task success for tool-using LLM agents from a leak-verified floor of 0.00 to 0.55-1.00, according to an arXiv paper (2609.05441v1). Across 23,440 scored episodes costing $42.57, embedding retrieval on updated facts collapsed unpredictably (0.30-0.95 across models, max seed gap 0.45) and agents acted on a correctly retrieved value only 55% of the time, while update-on-write stores including LLM summarization held at 0.70-1.00. The authors report that swapping a memory's implementation moves task success by up to 60 points and that full replay is never economical, with the best condition per domain delivering 2.7-3.9x its marginal utility per dollar.", "body_md": "arXiv:2609.05441v1 Announce Type: new \nAbstract: Long-term memory for LLM agents is evaluated today by conversational recall benchmarks (LoCoMo, LongMemEval), which measure question answering over dialogue history, not whether remembered facts change what a tool-using agent does. We present MERIT (Memory Evaluation for Realistic Instrumented Tasks), a benchmark and harness that measures the marginal utility of memory for task-executing agents under explicit cost accounting. MERIT provides episodic tool-use tasks in three domains whose dependence on earlier-episode facts is verified by an automated leak check; a difficulty ladder ending in updated-fact recall; controlled memory corruption; and full token and dollar metering of every memory operation. Across 23,440 scored episodes ($42.57), a two-generation pilot on gpt-4.1-mini and a preregistered 3-model x 3-seed grid (GPT-4.1, Claude Haiku 4.5; memory side held fixed), memory lifts dependent-task success from a leak-verified floor of 0.00 to 0.55-1.00. On updated facts, embedding retrieval collapses unpredictably (0.30-0.95 across models; max seed gap 0.45), and agents act on a correctly retrieved value only 55% of the time, while update-on-write stores (a structured fact store and, notably, LLM summarization) remain at 0.70-1.00; the hybrid is worse than the fact store alone. A latest-generation spot-check (Claude Sonnet 5, gated on a clean full-replay control) reproduces the pattern. Swapping a memory's implementation moves task success by up to 60 points, and full replay is never economical: the best condition per domain delivers 2.7-3.9x its marginal utility per dollar. We release the benchmark, harness, and all traces.", "url": "https://wpnews.pro/news/when-does-memory-help-a-cost-aware-evaluation-of-long-term-memory-in-tool-using", "canonical_source": "https://www.machinebrief.com/news/when-does-memory-help-a-cost-aware-evaluation-of-long-term-m-1wm7", "published_at": "2026-09-10 04:00:00+00:00", "updated_at": "2026-09-10 05:52:01.367048+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "ai-tools"], "entities": ["MERIT", "GPT-4.1", "GPT-4.1-mini", "Claude Haiku 4.5", "Claude Sonnet 5", "arXiv", "LoCoMo", "LongMemEval"], "alternates": {"html": "https://wpnews.pro/news/when-does-memory-help-a-cost-aware-evaluation-of-long-term-memory-in-tool-using", "markdown": "https://wpnews.pro/news/when-does-memory-help-a-cost-aware-evaluation-of-long-term-memory-in-tool-using.md", "text": "https://wpnews.pro/news/when-does-memory-help-a-cost-aware-evaluation-of-long-term-memory-in-tool-using.txt", "jsonld": "https://wpnews.pro/news/when-does-memory-help-a-cost-aware-evaluation-of-long-term-memory-in-tool-using.jsonld"}}