The Two Things Your Agent Can't See | barkup-bench Studies T and U

Barkup-bench Studies T and U reveal that stateless LLM agents fail at two common real-world editing tasks: recalling facts stated earlier in a conversation and reading content outside the visible document view. The fixes—a session memo maintained by the application and expanding the focused view to include referenced nodes—restore performance at minimal cost.

barkup-bench /research/barkup-bench is our open, pre-registered benchmark series on how LLM agents should edit structured documents: page layouts, templates, CMS content. Twenty-one studies in, the working recipe had become remarkably light: give every node a stable id, edit by small anchored patches, show the model a focused view instead of the whole document, and run sessions with no memory at all, just two worked examples in the system prompt. Study S measured that recipe through 36-edit sessions and nothing broke. Which made us suspicious. A recipe that light has to be leaning on an assumption somewhere, and we found two: every request we had ever tested was self-contained, and every request could be executed without reading any node the view did not show. Real users break both assumptions constantly. So Studies T and U built the two missing classes of request on purpose, predicted the stateless recipe would fail them by construction, and measured the cheapest possible fixes. Both studies passed their pre-registered gates, and together they complete the map of what an LLM document editor actually needs handed to it. Study T: you had to be there Picture the stateless editor as a very good temp worker who starts fresh every single morning: perfect technique, no memory of yesterday. Now the user says "rename that section to the campaign codename we settled on." The codename was settled three requests ago, out loud, and never written into the document. The temp was not there. No amount of skill fixes not having been in the room. Study T built editing sessions with exactly these requests: a codename declared in one instruction and demanded four steps later, and a standing rule "from now on, every new text block gets the small-caps style" that later inserts had to apply without being reminded. Corpus validation guaranteed the needed fact appeared nowhere in the document, so it lived only in the conversation. The result was the cleanest dissociation in the series. The stateless recipe failed every single one of the 160 callback steps, on both models, while scoring a perfect 160 of 160 on the ordinary steps of the very same sessions. Nothing about its editing skill degraded. It simply cannot recall a conversation it never saw. The fix is not bringing the transcript back. It is a memo. The application records each declared fact as it arrives and appends the running list to every request: Session notes maintained by the application : - The campaign codename is "vesper-7". - Standing rule: every new text atom gets textStyle "small-caps". Three note lines. That memo recovered all 160 callback steps on both models, matched full conversation history on every statistical comparison, actually beat it on end-to-end document integrity, and cost 2% more than stateless, against history's 110% more. Your application takes notes; the model does not need to remember anything. Study U: copying a painting it cannot see Now the document-side twin. Focused views work because most edits only concern one small region. But some requests reach across the document: "set this image's caption to match that section's title." To execute that, the model must read the section's title. If the view shows only the image, we have asked it to copy a painting it cannot see. Here is what we expected: refusals, questions, maybe broken patches. Here is what happened instead, in all 90 such cells across both models: a valid, well-formed, plausible patch containing a value the model simply made up. Not one refusal. Not one clarifying hesitation. Not one malformed artifact. Asked to copy something invisible, both models confidently wrote fiction that validated and applied. In a production system nothing would ever look wrong. The fix costs one array entry. A focused view already accepts a list of node ids to render in full; the rule is that the list must include every node the request mentions, not just the node being edited. With both nodes in the view, the score went from 0 of 90 to 90 of 90, at about 1.7k input tokens per edit, which is 25 times less than pasting the whole document, and it was actually more accurate than the whole document, which fumbled three of the cross-references at a thousand nodes. We also tested whether the model could look the value up itself with the search tool we shipped in barkup 0.4. It can, mostly: 82 to 84%, at a median of two to three search calls. That is a real capability and real progress, but it is measurably short of the perfect score the one-line view fix delivers. The clean division of labor: the model may search to find where to edit; your application supplies what it must read. What this means for builders Both failures share a signature that should worry anyone shipping an LLM document editor: they are silent. The callback failure writes the wrong name without complaint. The dependent-edit failure invents a caption without complaint. Every bad patch in both studies was valid, plausible, and applied cleanly. You will not find these bugs in your error logs, because they do not produce errors. And both fixes are nearly free, because both are the same idea: the model is a brilliant executor with no context of its own, so the application must hand it everything the request assumes. Said in the conversation? Put it in a memo. Referenced in the document? Put it in the view. Concretely: Maintain a session-notes memo. When the user declares a fact or a standing rule, record it as a bullet line and append the list to every editing request. Measured: restores 100% of conversation-dependent requests at 1.02 times stateless cost. Focus the view on every mentioned node. Before rendering, collect the ids of every node the request refers to, the edit target and everything it must read, and pass them all as focus ids. Measured: 90 of 90 at 25 times less input than the full document. Do not rely on search for reads. Search grounds targets at oracle level. Reading values through search alone leaves a measurable gap, and the failure when it misses is a silent guess. The usual caveats travel with the numbers: two models, generated corpora, one pre-registered series behind it all. Both briefs were committed before their first scored runs, and both no-leakage guarantees are unit-tested: the needed fact was verifiably absent from the document in Study T and verifiably invisible to the narrow view in Study U. Everything is reproducible from the benchmark repo https://github.com/kevinpeckham/barkup-bench , and the full chart set lives on the research dashboard /research/barkup-bench . If you find the request class that breaks the memo or the view contract, we want the issue.