In pursuit of the docs feedback loop Technical writer Tom Johnson reported that sanitized chat logs from Google's AI Studio app builder failed to reveal documentation gaps because coding agents rarely consulted the Google Maps Platform docs at all, usually building from training memory or choosing an open-source mapping library instead. Johnson cited Vercel's agent evals, which found agents did not invoke a Next.js docs skill in 56% of cases and passed only 53% of the time — no better than with no docs — while explicit instructions to use the skill raised the pass rate to 79%, though small wording changes made the approach fragile. Vercel found that embedding a compressed docs index of about 8 KB in the project's AGENTS.md file worked better than the skill approach, a workaround Johnson said proved more useful than the logs themselves. In pursuit of the docs feedback loop A couple of weeks ago, I wrote about a documentation strategy I’m pursuing: mining chat logs from AI app builders to find doc problems and fix them in a continuous loop. See My AI strategy to fix docs based on chat logs https://idratherbewriting.com/blog/ai-strategy-to-fix-docs-based-logs . Since you might not have read that post, let me briefly set up the larger goal. For years, technical writers have written documentation largely in the dark, with little visibility into where developers get stuck when trying to use an API. Now that developers are using AI coding agents and app builders to build apps with our products, those chat sessions should be a goldmine. If we can look at the logs of how users are trying to build with our APIs, identify the frictions and failure points where the agent or user goes off the rails, turn those failures into actionable doc bug tickets, and feed them into an automated doc-bug-fixing skill, we can create a feedback loop that ratchets up documentation quality in impressive ways. That was the idea, anyway. In this post, I’ll go deeper into what got in the way when I tried to put this loop into motion, and describe a workaround that turned out to be more useful than the logs themselves. The logs don’t show what the agent did with your docs To test this workflow, I first got access to sanitized logs from an AI app builder where developers describe an app they want and an agent writes the code to build it AI Studio https://aistudio.google.com/apps . I expected the logs to show where the agent read our Google Maps Platform documentation, misunderstood an instruction or code sample, and produced broken code that we could trace back to a doc gap. However, even if you have access to sanitized logs from people using agents to build apps with your APIs, those logs won’t give you a clear picture of what’s wrong with your docs. There’s no neat log entry that says the agent read this page, misused this parameter, and the build failed as a result. The logs I had were sanitized summaries of each conversation. They didn’t record which docs or skills the agent loaded, which tools it called, or which errors it hit. A bigger problem showed up first, though. In most sessions, the agent never consulted our docs at all: - Many sessions had nothing to do with the product I document Maps . - In sessions that did involve the product, our product skill rarely fired. It loaded in only a small fraction of them. - The agent usually built from memory. It relied on what it already knew from training, and it often chose an open-source mapping library instead of ours. Agents are confident some would say arrogant . They assume they know how to build whatever the user asks for, so they rarely stop to look anything up. What Vercel found about skills that don’t fire Vercel ran into the same problem and wrote about it in AGENTS.md outperforms skills in our agent evals https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals . Vercel makes the Next.js framework and runs a hosting platform , but the study wasn’t about building on Vercel. They tested whether coding agents could correctly use Next.js 16 APIs that aren’t in the models’ training data. When they gave agents a Next.js docs skill, the agents didn’t invoke it in 56% of the eval cases. The pass rate 53% was no better than having no docs at all. Adding explicit instructions to use the skill raised the pass rate to 79%, but the wording mattered a lot. When the instruction told the agent it must invoke the skill first, the agent read the docs right away and anchored on the doc patterns, missing details in the project itself. An instruction to first explore the project structure and then invoke the skill worked better. Vercel took that sensitivity to small wording changes as a sign that the approach was fragile. What worked was putting a compressed docs index about 8 KB into the project’s AGENTS.md file, which the agent loads into its context on every turn. The index points to version-matched doc files in the project and includes this line: IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning for any Next.js tasks. With that in place, the pass rate went to 100%. The fix wasn’t a better-targeted skill. Vercel moved the pointer to the docs into context the agent always reads, so the agent no longer had to decide whether to look something up. Vercel can’t edit its users’ AGENTS.md files, but it owns the project scaffolding. According to the Next.js docs, create-next-app adds an AGENTS.md file to new projects starting with Next.js 16.2 , and a codemod adds the docs index to existing projects. An API provider usually doesn’t create the developer’s project, so it doesn’t have that channel. The same idea applies to the system prompt of an AI app builder. If you control that system prompt, you can add something like this: Your training data about insert product APIs is out of date. Before you build a solution that uses maps or location data, load the insert product skill and use its docs instead of relying on memory. I tried a version of this. I can’t change the builder’s system prompt, so I put a two-sentence hint like this at the start of the user prompt instead. Every prompt in my test set more on that below started with the hint, and every build that got past setup loaded our skill. That sounds convincing, but those runs had no comparison without the hint, and short prompts that clearly describe a map app loaded the skill with or without it. The cleaner test used a long prompt that didn’t mention our product by name. Without the hint, the agent loaded our skill in 0 of 2 runs. With the hint, it loaded the skill in 2 of 2 runs. A prompt for an app that didn’t need a map didn’t load the skill, so the hint didn’t make the agent load it everywhere. These are small samples, but they point the same direction as Vercel’s results. Sponsored A system prompt reaches only one platform Even if a hint like this works, it fixes the problem only on the platform whose system prompt you control. People build apps with Claude Code, ChatGPT, Cursor, Replit, and many other tools, and each one has its own system prompt that you can’t change. On those platforms, the agent’s choice depends on what the model learned in training, what it finds when it searches the web, and which skills or MCP servers the developer installed. And skills and MCP servers help only the developers who already chose your product. Still, fixing one platform is better than nothing. Once your skill fires and the agent starts building, you get to the second part of the problem: figuring out what the agent struggles with in your docs. Recreating user sessions with a build log If your logs record what the agent struggled with, great. Mine didn’t. What I did have were sanitized, high-level descriptions of the kinds of apps people were trying to build, for example a site for a bakery that shows its store locations on a map. So I had my analyze-chat-logs skill go through a week’s worth of these descriptions a sample, not the full set and write 10 prompts that roughly simulated the most common scenarios. Then I ran those prompts myself in the same AI app builder tool AI Studio https://aistudio.google.com/apps . To each prompt, I added a block of instruction asking the agent to keep a build log file. For each step, the log records what the agent did, which source it used, why, and the result or error. The block also asks the agent to note where the docs were hard to use. The agent’s own log isn’t enough, because agents are unreliable narrators. In one build, the log said the agent used a skill that it never downloaded. In others, the agent downloaded doc files and never opened them. Because I ran these sessions myself, I could also check each one against the platform’s action history. In Google AI Studio’s Build mode, the person running a session can see its action history. Every session and action history I describe in this post is one I ran, not a user’s. Even if you don’t have detailed logs of user sessions, you can recreate the scenarios yourself and ask the agent to log what it does. The log from a recreated session shows you which problems the agent runs into, and where it got the information that led to each one. What the recreated sessions turned up Across the 10 scenarios, I ran about 15 builds some failed because of quota limits or platform errors, so I reran them , and 9 apps were built. I fed the built apps and build logs into AI for analysis. It identified 8 doc-related issues. Seven of them were in the product skills the agent loaded, not in the docs on our site. The issues fell into a few broad patterns, which I suspect show up with many APIs: - Setup steps the agent had to discover by trial and error. A missing configuration step cost a fix cycle in almost every build. - Cost-blind code. The agents requested more data, and more expensive data, than the app needed, and none of the apps told the user what running them might cost. - Example code treated as a spec. Agents copied examples line for line, shortcuts and placeholders included. - Instructions that didn’t hold everywhere. A rule that worked for some endpoints caused errors on others. - Thinner guidance for frameworks. Guidance for building with a popular framework like React was weaker than guidance for the core API, partly because framework samples often live outside the main docs. - Policy rules in the wrong place. Requirements like attribution lived on pages the agent didn’t open while it was writing code. Fresh docs help only if the agent fetches them I expected agents to pull fresh docs on demand. An agent can get information about an API in a few ways: from its training data, by fetching web pages, or by calling a docs tool, such as an MCP server that searches official docs. Google’s Developer Knowledge API https://developers.google.com/knowledge/api is one of these docs sources. Agents can call it through an MCP server, and it re-indexes published pages frequently. A lot of people including me assumed agents would call tools like that to get current docs. In the builder I tested, the Developer Knowledge MCP server wasn’t one of the agent’s tools. The agent got docs in three ways: - Product skills most of the time . The agent downloaded static Markdown files that a pipeline generates from the published docs. - A docs retrieval tool occasionally . The agent called a Maps-specific docs retrieval tool retrieve-google-maps-platform-docs , from the Maps Code Assist MCP server https://developers.google.com/maps/ai/code-assist in only a small fraction of sessions. - Doc pages rarely . Once in a while, the agent fetched a doc page directly. A docs API can still play a part, because a skills pipeline can use it to pull the pages that an LLM then rewrites into skills. In that case, though, the API feeds the pipeline, not the agent. Having generated skill files sit between the docs and the agent creates two problems: - Slower doc updates. A skill changes only when someone reruns the pipeline and publishes the output, and I’m not sure when the skill files my test agents loaded were last regenerated. A doc fix I publish today might reach a docs API within a couple of days, but it won’t reach agents in this builder until the next skill regeneration. - Harder debugging when the agent makes a mistake. Tracing a build failure back to a specific doc page takes detective work, because the problem could come from several places: - The source doc page has an error or a gap. - The LLM introduced an error when rewriting the doc page into a skill. - The skill’s example code has a flaw or shortcut that the agent copied. - The model ignored a rule that was right in front of it. This setup applies to the one builder I tested. In other tools, such as an IDE agent like Claude Code, Cursor, or Gemini CLI, a developer can connect a docs MCP server so the agent queries live docs directly. For Google Maps, for example, a developer can configure the Maps Code Assist MCP server https://developers.google.com/maps/ai/code-assist or the broader Developer Knowledge MCP server https://developers.google.com/knowledge/reference/mcp , and the agent can call tools like retrieve-google-maps-platform-docs or search documents to pull current pages on demand. Even in those tools, though, developers have to install the MCP server first, and wherever skills sit between the docs and the agent, the skills are the docs the agent reads. That’s a strong argument for tech writers to take a bigger role in product skills instead of treating them as someone else’s generated artifact. Writers can review what the generator produces especially the example code, which agents copy line for line , have a say in how often skills are regenerated, and test whether a doc fix makes it all the way to the agent. One actionable doc bug When I sent the 8 bugs through my doc bug-fixing workflow, only one became a doc change. A couple of API reference pages didn’t explain how the fields you request in a call affect the pricing tier for that request, so the fix adds that explanation to those pages. The other seven issues were in the skills, which a different team owns, so they became handoffs with proposed changes. So my plan of mining chat logs for a steady stream of doc fixes was perhaps too dreamlike. Accessing, curating, and parsing the data to find doc problems is like looking for special seashells on the ocean floor. And most of what I found pointed to the skills, not the docs. A live example: embedding a biking map on my own blog To show what a build log looks like in practice without sharing anything from internal user sessions , I ran the same experiment on my own blog using Gemini in Antigravity IDE. I asked the Gemini coding agent to embed a Google Map of a 17.5-mile bike ride into an older post of mine, The most perfect 17-mile biking loop in Santa Clara https://idratherbewriting.com/blog/biking-the-santa-clara-loop/ , and to keep a build log.txt file describing every step it took to retrieve information, whether from the Google Maps Platform plugin, remote skill files, a docs API, or memory. Across 8 logged steps, the agent found the post, extracted the route coordinates from an older KML link, prompted me to install the official google maps platform plugin from the Agent Marketplace, loaded the remote skills index index.json along with 4 sub-skills and 9 capability reference files from gstatic.com , fell back to the Maps Code Assist API mapscodeassist.googleapis.com when a routing reference file contradicted the main skill, and embedded a