{"slug": "the-benchmark-said-no-barkup-bench-studies-l-and-m", "title": "The Benchmark Said No | barkup-bench Studies L and M", "summary": "The barkup-bench series found that LLM agents editing structured trees fail when conversation history is removed and when models must locate nodes without stable IDs. Study M showed stateless sessions increased corrupted trees from 1 in 20 to 7 in 20, while Study L revealed that self-guided tree exploration is a trap, costing more tokens than full-tree context and failing on cheaper models.", "body_md": "By last week, the barkup-bench series had assembled a pleasingly cheap recipe for letting LLM agents edit structured trees: address nodes by stable id, reply with anchored patches, show the model a focused view instead of the whole tree, refresh that view every turn. Each step was pre-registered, measured, and shipped. Input costs stopped scaling with document size. Sessions stopped drifting.\n\nTwo hard parts remained, and both looked deletable. The model still needed to be told *which* nodes an edit concerns (our tasks named their target ids, an oracle assumption we flagged in every post). And sessions still carried full conversation history, which grows without bound and is exactly where the nastiest bug of this series once lived. So we pre-registered two more studies to delete them: let the model find its own targets, and let sessions forget.\n\nBoth gates failed. This post is about how they failed, because the failure shapes are more useful than the successes would have been.\n\n## Study M: the view carries the state, but memory still matters\n\nThe setup: our session studies showed that attaching a fresh 1.5k-token view of the current tree to every turn eliminates drift. If the view carries the state, what is the history for? We ran the same twelve-edit sessions with no conversation memory at all (every turn a fresh, single-turn conversation: view plus instruction) and with a two-exchange sliding window.\n\nThe economics worked exactly as predicted. Stateless input is flat: about 1,300 tokens at step one and at step twelve, forever, no context ceiling, and structurally immune to history-construction bugs because there is no history to construct.\n\nThe accuracy did not. Sonnet lost 7 steps to 0 against full history (p = 0.016), and sessions ending with a byte-perfect tree dropped from 19 of 20 to 13 of 20. Every stateless-only failure was a late-session placement edit: a legal patch, validated and applied, with the new node in the wrong position. The model saw the same child lists in the same view either way. History was providing redundancy for positional reasoning that the view alone does not, and the two-exchange window recovered only part of it.\n\nSo the guidance is the boring version: keep the history and the per-turn view. At twelve-edit lengths, history costs about 55k input tokens per session, and deleting it buys you three times the corrupted sessions. They cover different failure modes; you want both.\n\n## Study L: finding the node is the actual job\n\nThe setup: we regenerated our large-tree editing tasks with the ids stripped out of the instructions. Instead of \"the node with id n819\", the model gets what a person would say: \"the image named maple-ember\", or \"the 3rd block inside the section named atlas\". Every description was programmatically verified to match exactly one node, so ambiguity was never the model's problem. Then we compared three ways of supplying context: the whole tree in the prompt, a skeleton view plus an expand tool the model drives itself, and a deliberately dumb keyword retriever feeding the focused view.\n\nFirst result: models are decent grounders when they can see everything. With the full tree in context, sonnet solved 86.7% and gemini 84.4%, against oracle bounds of 95.6% and 91.1%. Call it a 7 to 9 point tax for describing instead of naming. Real, but far from disqualifying.\n\nSecond result, and the one we would most like agent builders to read: **the explore-the-tree-yourself pattern is a trap.** On the frontier model it actually works, 43 of 45, matching the oracle bound. But it took a median of 54 expand calls per task, and all those expansions plus their accumulating history cost more input tokens than just pasting the entire tree would have. And on the cheap model it does not work at all: 51%, with most failures being the model wandering the tree until its budget ran out without ever producing a patch. Navigation joins whole-tree rewrite in the category of techniques that are accurate exactly where you least need the savings.\n\nThird result: the keyword retriever landed at 60%, with nearly every failure a clean miss (the target region never made it into the view). That was the floor by design. The distance between it and the oracle is the value your application's retrieval layer has to add.\n\n## What the pair of failures buys\n\nA benchmark series that only ever confirms its own thesis is advertising. These two studies are why we trust the rest of ours: the same pre-registration discipline that produced \"views are free\" and \"sessions need fresh views\" also produced \"views cannot find themselves\" and \"sessions cannot forget\". We published the gates before running, and the gates failed, and the honest boundary is more useful than the result we wanted:\n\n- Stable ids make edits cheap and reliable at any size we tested.\n- Views make the input cost of\n*known*edits nearly free. - History makes sessions hold together; keep it.\n- And the one part you cannot outsource to the codec or the prompt:\n\nknowing which nodes the user means. That is retrieval, it is your application's job, and naive keyword matching is not enough.\n\nEverything is reproducible from the [benchmark repo](https://github.com/kevinpeckham/barkup-bench): pre-registrations (BRIEF-L, BRIEF-M, including one disclosed pre-run amendment), corpora, unit-tested graders, raw analyses. Combined cost of both studies: about fifty dollars, and two hypotheses we were fond of.", "url": "https://wpnews.pro/news/the-benchmark-said-no-barkup-bench-studies-l-and-m", "canonical_source": "https://www.lightningjar.com/blog/the-benchmark-said-no", "published_at": "2026-07-08 18:00:00+00:00", "updated_at": "2026-07-08 19:16:08.797465+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research", "ai-tools"], "entities": ["barkup-bench", "Sonnet", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/the-benchmark-said-no-barkup-bench-studies-l-and-m", "markdown": "https://wpnews.pro/news/the-benchmark-said-no-barkup-bench-studies-l-and-m.md", "text": "https://wpnews.pro/news/the-benchmark-said-no-barkup-bench-studies-l-and-m.txt", "jsonld": "https://wpnews.pro/news/the-benchmark-said-no-barkup-bench-studies-l-and-m.jsonld"}}