Self-Improving Docs, Part 1: Automatically Turning Documentation Gaps Into Pull Requests

A developer built a self-improving documentation system that automatically converts documentation gaps and reader feedback into draft pull requests for human review. The system captures failed assistant answers and reader feedback signals, then opens automated documentation jobs that draft changes and create PRs, keeping a human in the loop to decide what to merge. The developer cites Mintlify survey findings showing most teams take a week or longer to reflect product changes in docs, while most use AI agents to draft updates but few allow them to publish without human review.

This blog is Part 1 of my series, Self-Improving Docs : Turning Gaps Into Pull Requests, on how I run documentation as a self-updating system. In Part 2 , I will go deeper into the docs assistant itself: how readers ask questions, how we handle feedback, and what we learned while building this experience. In Part 3 , I will talk about the other half of the same problem, which is making the docs usable for coding agents through things like indexes, skills, and packaging. This post is about how I close the docs gap when readers fail to get a right answer. Let’s start with the old process: before I built automation, the feedback loop was slow because it involved a lot of dependencies. Some readers hit an outdated page, which usually opened a ticket to a support engineer in Slack, and eventually the product team sent that message back to me. I used to fix the wrong doc page, publish the new one, and move on. That still happens, but what changed is the speed and the audience. Now, AI agents read documentation a lot and differently from humans. A developer who opens an outdated page can notice the version looks wrong, ask a colleague, or check a changelog. An agent that lands on the same page does not usually pause and ask whether the page is correct; instead, it treats the docs as instructions and keeps going from that incorrect information. That is why outdated documentation stopped feeling like only an editorial problem and started feeling like a system problem. Mintlify https://www.mintlify.com/ recently published survey findings that matched what I was already seeing in my own work. A large share of teams still take a week or longer for a product change to show up in docs. Only about a quarter say it usually lands the same day. At the same time, most teams say AI agents now draft documentation updates for them, but only a small group lets those agents publish without a human in the middle. The interesting part is not the exact percentages. Here, the interesting part is the pattern: people know documentation cannot keep up with product changes by writing and updating alone, and they are using agents to draft, while still wanting a human to decide what is true. That is the same place I ended up. The problem was not “we need a chatbot” . A lot of conversations about AI and docs start with chat in my chat assistant. I built a docs assistant on our developer site, and readers use it every day. But chat by itself does not close the loop of missing and incorrect information. If the assistant cannot answer, and that failure disappears into a log nobody reads, you only built a nicer search box; that’s it. Well, I wanted it to know what’s wrong and self-update the docs. I wanted to understand: when the docs assistant cannot answer, or when a reader says the docs need an update, does that become work in the documentation repo? If the answer is no, the bot is only an answer-giving machine and is not helping me close the docs gap loop. If the answer is yes, the bot becomes part of how documentation self-improves. So I built a system that self-improves. When I say closing the feedback loop, I mean this: This captured data can be used to open an automated documentation job that drafts a change and creates a pull request for me. The idea is that a human still reviews the PR and decides what to merge, but the hunting and getting feedback directly from the doc site is an insanely fast way to improve documentation. I also capture feedback that never reaches the “not found” path. Readers can tag answers as helpful or not helpful, mark that something needs a doc update, report a bug, or send free-form feedback into Slack. Those signals matter because not every gap looks like a wrong answer. Sometimes the assistant answers confidently, and the reader still knows something is off. So I ended up with two kinds of input into the same system: Automatic detection catches problems left out, and human feedback catches “the answer was there, but it was not good enough.” If I draw the whole thing, it will look like this: The important line in that diagram is the last one. The draft is not published. The PR is where I decide what is true and what needs to get merged. The idea is simple. After the docs assistant answers, I ask one question: did the docs actually cover what the readers asked? I do not call a separate success API for that; instead, I read the answer itself. If it sounds like the topic is missing from the generated answer, or the model is saying it only has partial context, I treat that as a documentation gap. When that happens, the system automatically sends the gap details to my backend with some information such as the reader’s question, the page they were on, and why I think the docs failed. From there, the backend starts a Mintlify agent job with that context. The agent’s job is not only to refresh the live site. Its job is to draft the missing documentation, open a branch in the docs repo, and create a pull request. That PR is what I review. If the draft is good, I merge it, and if it needs edits, I edit and fix the missing part of it. If I feel that the gap is not a real docs gap, I close the PR and move on. So the mechanism is: Answer comes back → Detect gap from the answer text → Send question + page + reason to backend → Mintlify agent drafts a docs change → PR opens in the docs repo → Human reviews and merges In code, the important part is the agent job call. Once any doc gap is detected, the backend asks Mintlify to draft from the failed question: js const res = await fetch https://api.mintlify.com/v1/agent/${projectId}/job , { method: "POST", headers: { Authorization: Bearer ${adminApiKey} , "Content-Type": "application/json", }, body: JSON.stringify { branch: doc-gap-${Date.now } , asDraft: true, messages: { role: "user", content: User asked: ${question} , Page: ${pageUrl} , reason === "not found" ? "This was not found in the docs. Please add or expand documentation." : "The assistant had only partial context. Please add or expand documentation where relevant.", .join "\n" , }, , } , } ; const jobId = res.headers.get "X-Message-Id" ; That is the whole auto-update mechanism. A failed answer becomes context, and that context becomes a draft PR. Tech writers can still decide what ships. What I have noticed is that most teams are fine letting agents draft documentation updates, but they still want a human deciding what gets published. That matches how I run it. The agent can open the PR, but the PR is not live docs yet. Someone still has to read the change, check whether it is accurate, and only then merge it. On a multi-product developer site like mine, that review step matters a lot, because one wrong sentence in the wrong product section is enough to break an integration for developers who use our docs. If I only talk about auto triggers, then this story is incomplete. On my docs site, readers also send feedback deliberately. Sometimes they use tag chips under an answer, or they escalate to support when the assistant cannot help. They can also open the feedback option and write what is missing in their own words. Those messages go to Slack and into logging, so that I can see the exact wording instead of a vague “someone was unhappy.” If I put the feedback mechanism next to the automatic one, it looks like this: This matters because documentation gaps rarely arrive as one clean ticket. Most of the time it shows up as the same question asked again, an answer that only covers half of what the readers needed, or a short note like “this page assumes I already know X.” If that signal stays buried in chat history, nothing happens with it. But when the same note reaches Slack or a sheet with the page URL and the original question attached, it becomes useful data that a docs team can actually use for better documentation updates. Think of feedback and auto triggers as two doors into the same room. One door is opened by the system when the answer looks empty. The other door is opened by the reader when the answer looks wrong or incomplete, and both must lead to documentation work. In many companies and even in the company I work for, the older feedback mechanism depended on someone deciding the docs were worth a ticket. That still happens, and it is still valuable. What changed with a docs assistant is that the failure can be observed at the moment of asking and can be fixed automatically without pulling in a lot of people. That is closer to how product teams ship. You do not wait only for support tickets to learn that a flow is broken. You look at where users drop. With this process, documentation can work the same way. Mintlify’s report also talks about companies pointing agents at feedback and turning those signals into documentation patches. Anthropic’s public story in that report is a good example of the same direction: collect signals, triage them, and turn the real documentation gaps into PRs while people still decide what ships. Our version is built for our docs stack, but the idea is similar. Read the failure then draft a fix, review it and then merge it to publish. I want to be honest about the limits when it comes to auto updating the documentation, because otherwise this starts sounding like this is so easy so we all need to follow this process. An agent job is only as good as the context you give it and the documentation truth underneath it. If the product behavior is unclear, the draft will be unclear too. Suppose, the question was actually a support issue or a configuration problem, opening a docs PR is the wrong way to update your docs. So, somebody still has to look at the PR and ask: is this a real documentation gap , or are we documenting around a product problem? There is also noise. Automatic “not found” detection based on phrasing is useful, but it is not perfect. Sometimes the model hedges or the docs do contain the answer and RAG based retrieval missed it. It could also be that the reader asked something outside the product which is not documented at all. So the loop has to stay reviewable and easy to change. And closing the loop on gaps is only one part of keeping docs updated. Product changes still need writers and engineers who notice what shipped. Version drift, naming consistency, and release notes still need their own updating systems. I will discuss this more in Part 3 of this series. Before this mechanism existed, a weak docs answer mostly meant I would find out later, if at all. Now a wrong answer can show up as a concrete artifact: a logged question, a Slack escalation, a feedback note, or a draft PR waiting for me to review. This changed how I prioritize my work. Instead of only asking “which page should I rewrite or update this week,” I can also ask “which unanswered questions keep repeating” and “which draft PRs are actually fixing the same gap.” It does not remove writing but it changes where the writing starts. Sometimes it starts from a blank outline or from a reader's question that already proved the gap exists. For me, that is the useful definition of docs systems work. You still own clarity and correctness and you also own the path from failure back into the repo. Documentation used to be something we published and hoped people say all is correct and we never needs to update it. But, in real world that is not the case. Now that most of the modern docs systems also uses an AI assistant to answer reader's questions, if those systems hit a gap and nothing happens afterward, the gap stays invisible until someone catches and flag it. In this post I only covered the loop itself: detect the failure, capture feedback, draft the docs gap, and keep a human on the merge. In Part 2 I will write about the docs assistant side in more detail, including how we collect information without turning every chat into noise. In Part 3 I will write about preparing the documentation for agents in the first place, so fewer questions fail before the gap even starts. If you maintain developer docs and you already have an assistant on the site, I would start with one question: When the bot cannot answer, where does that failure go? If the answer is nowhere, that is the first system you need to build.