The Harness and Memory Architecture Behind Self-Updating Docs Mintlify rebuilt its self-updating documentation architecture around a background agent and a runner agent sharing git-backed memory, raising its internal docs PR accept rate from 70% to 90%, software engineer Patrick Foster wrote in a company blog post. The background agent runs daily per site with an optimization function to maximize merged docs PRs while keeping the accept rate above 80%, and can create, read, update and delete runner triggers, skills and shared memory. Mintlify built its harness in-house on Vercel's AI SDK after finding OpenCode and Pi abstracted away too many engineering decisions; benchmarked against Pi on an offline docs eval suite, the two performed similarly but Pi used 50% more tokens. All articles https://www.mintlify.com/blog The Harness and Memory Architecture Behind Self-Updating Docs 7 minutes read Patrick Foster Software Engineer Share this article Patrick Foster Software Engineer Share this article Before Mintlify was a platform for writing documentation, we started off trying to make docs self-updating https://techcrunch.com/2022/05/30/mintlify-taps-ai-to-automatically-generate-documentation-from-code/ . It's the holy grail of problems for a documentation company. It's also a hard problem. Docs changes take many forms, context is scattered across tools, and every team works differently. We recently rebuilt our self-updating docs architecture from the ground up. Internally our docs PR accept rate has gone from 70% to 90%. Getting there meant rethinking the harness, the infrastructure, and memory. High-level architecture high-level-architecture We split the system into a background agent and a runner agent. Both run as an agent loop with access to a sandbox, the ability to clone repos, read access to third-party integrations, and a shared memory. Memory is stored as a git-backed repo containing different memory files, with every agent run log stored as JSON. We store a set of triggers, such as cron schedules, merges in a repo, or events in third-party integrations e.g., a project being marked as done in Jira . Each trigger is associated with a skill. When a trigger fires, a job is queued for a runner agent to execute that skill. Most often, the runner's output is a documentation PR, followed by a notification asking the right people to review the change. Every day we run the background agent for each site. Its system prompt tells it to take ownership of the documentation's success. Its optimization function is to maximize the number of merged docs PRs while keeping the accept rate above 80%. To do this, the background agent can perform CRUD operations on the runner agent's triggers, their skills, and the shared memory. It decides what to change by reviewing past run logs, information in connected integrations, analytics from the docs platform, and which PRs are getting accepted. For instance, one trigger listened for PR merges in a code repo. Its CLI docs PRs kept getting closed because the CLI ships on a rolling release cycle. The background agent noticed, removed CLI changes from that trigger's skill, and created a monthly cron trigger to write CLI release notes. Harness harness We decided to build the harness in-house. We had previously tried using OpenCode and Pi but found they abstracted away too many of the underlying engineering decisions for it to perform well in our environment and match our data models. Building a harness is fairly straightforward. We use Vercel's AI SDK to loop LLM calls and expose a set of tools. We built custom tools for subagents, compaction, following up on other runs, forking the current session, using skills, and running shell commands that we pipe to a sandbox. We benchmarked ours against Pi on an offline docs eval suite and found they had similar performance, but Pi used 50% more tokens. Fixed schema memory fixed-schema-memory The infrastructure above is table stakes. The real challenge is tastefully designing the memory scaffolding, custom tools, and preset skills. Take the PR-merge trigger example from earlier. Memory has to answer a lot: Which team owns this change, and where do they discuss it in Slack or Notion? Should the docs build on a rolling release? Who should review it, and how should we notify them? The list goes on. To avoid being noisy and to write at a high quality, our agent needs to know all of this reliably. We do this by storing a set of fixed schema JSON memory files. For instance, here's a simplified version of what we store for each large feature release: type Release = { // Code and docs PRs that make up the release codePRs: string ; // People driving or reviewing it stakeholders: UserID ; // Specs and planning docs in Google Docs or Notion docs: string ; commitDirectlyOrCreateOtherPRs: | 'commitDirectly' | 'other-prs'; // Don't start drafting docs before this date startDraftingChangesDate: string | null; communicationMessaging: { relevantChannels: string ; issueTrackingProjects: string ; relevantThreads: string ; } | null; }; We apply the same level of granularity to memory for users, style guides, site-wide settings, and every integration. The memory repo file structure looks roughly like this: site-wide.json style-guide.json users/