A study from barkup-bench found that returning structured error messages to AI models during patch validation does not improve their ability to correct failures. Across three models—Opus, Gemini, and …
A study by barkup-bench found that requiring an LLM to restate a goal before rewriting it does not improve performance: every model complied perfectly 90 out of 90 times but lost every decisive compar…
Barkup-bench's Study AD confirms that the core LLM editing stack validated on claude-sonnet-4.5 and gemini-3.5-flash transfers to claude-opus-4.8, passing every pre-registered gate with 194/200 on the…
A new pre-registered benchmark series, barkup-bench, found that large language model agents silently invent missing information rather than asking for it, but a simple one-sentence permission to ask e…
Barkup-bench Study Z found that LLM agents (claude-sonnet-4.5, gemini-3.5-flash, and claude-opus-4.8) correctly retrieved planted facts and applied styleguide rules from full 3,300-token context packs…
A new study in the barkup-bench series found that AI models can follow qualitative goals embedded in application memos as effectively as explicit instructions, but models that read the goal's document…
Barkup-bench Studies T and U reveal that stateless LLM agents fail at two common real-world editing tasks: recalling facts stated earlier in a conversation and reading content outside the visible docu…
A pre-registered benchmark series called barkup-bench found that both a 'keep-history' and a 'stateless plus worked examples' recipe for letting an LLM edit structured documents remain reliable across…
A new benchmark series, barkup-bench, finds that LLM agents editing structured data do not need conversation history; two worked examples in the system prompt match or exceed the performance of full s…
Researchers at barkup-bench published two new studies showing that replacing tree-navigation with a search tool dramatically improves LLM agent accuracy and cost when editing structured data. The sear…
The barkup-bench series found that LLM agents editing structured trees fail when conversation history is removed and when models must locate nodes without stable IDs. Study M showed stateless sessions…
Researchers at barkup-bench ran seven pre-registered studies with over 13,000 model runs to determine the most reliable way for LLM agents to edit structured data. They found that giving every node a …
A new study from the barkup-bench series reveals that LLM agents suffer from session drift when serializing tree state only once, with per-step success dropping from 98.8% to 83.8% for Claude Sonnet 4…
A benchmark comparing whole-tree HTML rewrite against granular JSON tools for LLM-driven editing found that a defect in the benchmark harness had hidden small-model multi-turn tool reliability. After …