{"slug": "when-an-ai-agent-goes-off-the-rails-file-a-bug-to-fix-the-documentation-then-the", "title": "When an AI agent goes off the rails, file a bug to fix the documentation – then test the fix", "summary": "XMLUI co-maintainer filed a bug and added a new How To document after Claude failed to find a pattern for converting a select widget to a radio group, with the new doc scoring 4.2 in MCP search versus 0.7 for previous top hits. The project now uses MCP query logs and a portfolio of five apps to systematically identify and fix documentation gaps for AI agents.", "body_md": "Here is a mundane question that came up in an agentic coding session: How do I convert a select widget to a radio group? I was working on [Bram](https://url.usb.m.mimecastprotect.com/s/dfaeCrgGYGToE07VT7f6Zh4J9QI?domain=github.com), a desktop app that harnesses coding agents to a GitHub-backed workflow. Bram’s UI is made with [XMLUI](https://url.usb.m.mimecastprotect.com/s/DOhdCvmMjMuKxjqNcXhlxhQHqYX?domain=github.com), a framework that wraps a simple declarative syntax around React and CSS. Agents using the XMLUI Model Context Protocol (MCP) server are instructed to prefer [How To](https://url.usb.m.mimecastprotect.com/s/peyKCwnNkNs9Kzk8h9iG6hJ1lWq?domain=xmlui.org) examples that provide verified working patterns. In this case Claude didn’t find one, fell back to the documentation for the [RadioGroup](https://url.usb.m.mimecastprotect.com/s/MIGfCxo0l0Uwk4DMcwsEGhyICST?domain=xmlui.org) component, and after a bit of fumbling figured it out. All good, right?\n\nNope. Claude should have found a How To and solved the problem immediately. The absence of that document was a bug, and [this new How To document](https://url.usb.m.mimecastprotect.com/s/87ExCyp4m4hkZzKXFNtm2hxqy-Z?domain=xmlui.org) is the fix. This particular bug didn’t bite too hard, but the How To catalog is full of complex and hard-won patterns that agents can struggle to rediscover. To ensure that XMLUI is reliably learnable by agents, we continually expand that catalog.\n\nWe’ve always said documentation is integral to software products. Now that we write for both people and agents, the need to make good on that promise is more obvious. What’s less obvious is that we can now bring systematic rigor to documentation. We needn’t only write what we imagine people want to read; we can also write what agents provably need to read.\n\nAs a co-maintainer of XMLUI I am acutely sensitive to agents fumbling as they build XMLUI apps. When something that should be a no-brainer isn’t, because the MCP search didn’t find the answer it should have, I file an issue and then direct an agent to write the missing document.\n\nThat new doc is a testable fix. For example, I asked Claude to find its failed searches for the select-to-radio conversion in the MCP log. The response:\n\nTwo how-to searches, both at 21:12, both misses:\n\n“radio group mutually exclusive choice change value form input” — top hits: use-a-custom-formitem, prefill-a-form-from-an-api-response, use-regex-validation-in-formitem. Form-adjacent noise, zero radio content.\n\n“choose one of two options horizontal selector” — the rephrase moments later, attacking from the UX side instead of the component side. Also nothing.\n\nThe failing searches had returned only weak matches with a top relevance score around 0.7. After adding the new doc I asked it to repeat the search and compare. The new document surfaced at #1 with a 4.2 score, six times higher than anything else. (Because the MCP server can pin the version of the docs it consults, direct A/B comparison is possible.) I also tried a handful of synthetic queries, like “radio buttons for a small set of options”. These confirmed the result.\n\nWhen I’m in the thick of the action and see an agent miss finding a document that should exist, I can respond with focused precision. But that’s just me, working in an app that only exercises a subset of XMLUI’s capabilities. To automate the discovery of missing documents across a portfolio of apps, three sources of information come into play.\n\nYou can’t improve what you can’t measure, so the MCP server logs the queries that agents make plus the results they see. That has driven several rounds of [iterative improvement](https://url.usb.m.mimecastprotect.com/s/XnH5Czq8n8H96pEghwu7Wh9uX28?domain=xmlui.org).\n\nA portfolio of real apps is an inventory of patterns that someone needed to discover and use. So reverse-engineer the process: what search would lead to a working recipe? Here’s a powerful prompt:\n\nFan out subagents across these five apps. Return the top patterns that a developer would search a How To catalog to find. Then run the MCP searches that should find those patterns.\n\nThe MCP logs directly measure documentation gaps.\n\nA missing doc is only a problem if somebody needed it and didn’t find it. Session logs correlated by timestamp with MCP logs can surface moments where a developer/agent team flailed and hit dead ends doing things that should have been easy.\n\nDoing that correlation reliably is easier said than done. Although it helps to make full-text search available to agents, as Bram does, I haven’t yet gotten agents to reliably surface episodes of struggle. The signals they find in the logs can be useful, though, and may improve.\n\nLike other humanities majors, I found my way into tech as a writer of software documentation, then moved on to other roles as soon as I could. But after decades as a developer and tech journalist/editor I’ve returned to the role, first as a contributor to [Turbot](https://url.usb.m.mimecastprotect.com/s/HRhWCA8LmLh243B6C2C9mhGhS2t?domain=turbot.com)’s docs, then XMLUI’s.\n\nMy identity as a writer is not bound to these docs. They are not literary works. They are what we always said the docs should be: integral components of software products. As with all software components now, we direct agents to build them. In this case the work requires both editorial and engineering chops. When you bring both disciplines to bear, doc writing becomes more interesting, fun, and impactful than it ever has been.", "url": "https://wpnews.pro/news/when-an-ai-agent-goes-off-the-rails-file-a-bug-to-fix-the-documentation-then-the", "canonical_source": "https://www.infoworld.com/article/4211198/when-an-ai-agent-goes-off-the-rails-file-a-bug-to-fix-the-documentation-then-test-the-fix.html", "published_at": "2026-08-26 09:00:00+00:00", "updated_at": "2026-08-26 09:13:58.069183+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "artificial-intelligence"], "entities": ["XMLUI", "Claude", "Bram", "GitHub", "React", "CSS", "Model Context Protocol"], "alternates": {"html": "https://wpnews.pro/news/when-an-ai-agent-goes-off-the-rails-file-a-bug-to-fix-the-documentation-then-the", "markdown": "https://wpnews.pro/news/when-an-ai-agent-goes-off-the-rails-file-a-bug-to-fix-the-documentation-then-the.md", "text": "https://wpnews.pro/news/when-an-ai-agent-goes-off-the-rails-file-a-bug-to-fix-the-documentation-then-the.txt", "jsonld": "https://wpnews.pro/news/when-an-ai-agent-goes-off-the-rails-file-a-bug-to-fix-the-documentation-then-the.jsonld"}}