{"slug": "building-a-software-factory-for-ai-sdk", "title": "Building a software factory for AI SDK", "summary": "Vercel's AI SDK, an open-source project with over 20 million weekly npm downloads and 26,000 GitHub stars, faced a backlog of over 1,000 open issues and nearly 800 pull requests by late June. To address this, Vercel built a software factory called ai-sdk-factory, which within four weeks authors 25-35% of merged PRs and closes 70-80% of issues, while keeping human review for high-risk changes.", "body_md": "The [AI SDK](http://github.com/vercel/ai) is one of the most popular open-source AI projects in the world. It serves over 20 million npm downloads a week and the repo has over 26,000 stars. Maintaining the codebase means tracking four moving targets at once:\n\nModel providers: new providers, new capabilities, and new bugs\n\nUI frameworks: bindings for React, Next.js, Svelte, Vue, and others\n\nSandboxes: the execution environments agents run code in\n\nHarnesses: adapters for Codex, Claude Code, Pi, and others\n\nAfter multiple years of growth, the repo was getting 100+ new issues every month, and when Anthropic's Opus 4.6 model was released, PRs hit an inflection point. By late June, that compounding had accumulated over 1,000 open issues and almost 800 pull requests.\n\nThat backlog is not a discipline problem. No maintainer, however good, can close that gap by working harder, and because generating code is cheap, it will only grow.\n\nInstead of trying to scale ourselves, we built a software factory. Four weeks in, it authors between 25 and 35% of PRs we merge and closes 70-80% of issues.\n\nBefore we built anything, we had to answer three questions:\n\nWhy our existing approach using agents wasn't enough\n\nWhat level of automation fit a project like the AI SDK\n\nHow to align automation and human effort to risk\n\nThe best maintainers are already using agents aggressively. Mitchell Hashimoto runs Ghostty with the goal of an agent always working, and encodes every agent failure in AGENTS.md so it never repeats. Simon Willison runs four coding agents in parallel, and review bots like Vercel Agent and CodeRabbit sit on millions of repos. Other maintainers like Daniel Stenberg have opted to block AI-generated submissions to curl.\n\nAll of it helps, but none of it solves for the core constraint: every one of these solutions still routes every change through one human's attention. We believe that human accountability is still the [core of trust](https://vercel.com/blog/agent-responsibly) in agentic engineering, so we knew our factory needed to solve for reviewer efficiency as a first principle.\n\nSoftware factories sit on a spectrum. At one end is full automation, where agents write, ship, and deploy without a human ever reading the code. In the middle are harnesses like Codex and Claude Code, where a person steers an agent or fleet of agents. At the far end is software you barely want to automate because the risk is too high, like firmware in a pacemaker or self-driving vehicle.\n\nThe AI SDK needs to operate closer to the careful end. It is foundational AI infrastructure with millions of applications built on top, so quality and security are non-negotiable, and a human needs to have control over what ships. We needed our factory to heavily automate the lifecycle around the human, without removing them.\n\nFor a given change, the depth of human review needed scales with risk.\n\nCentralized roadmaps with detailed feature specs can define risk up front and directly shape the work going into a software factory. But open-source projects like AI SDK also get issues and pull requests from the community, and there is no guarantee they align with the goals of the project, or that changes are safe to make. Higher risk means human judgment is an even more critical part of the system.\n\nTo optimize our factory for that judgment, we knew agents had to go beyond generating code based on a request; they needed to evaluate full units of work in the context of the entire project.\n\nOur goal was for the factory generate a comprehensive assessment of each change for both fit and risk, including a full chain of documented evidence, making it easy for reviewers to apply the right amount of effort:\n\nDocs fixes get a quick glance for verification\n\nWell-defined provider changes get focused validation\n\nA new public API gets deep review\n\n`ai-sdk-factory`\n\nis a software factory that autonomously processes incoming issues and pull requests for AI SDK. Agents in the factory perform specific, reviewable tasks, like reproducing bugs, implementing features, and creating backports for older SDK versions. A human stays in control throughout the entire process, including merging every change.\n\nWe didn't ship the factory in one swing. We built it incrementally, starting at the beginning of the process with classification of issues as bugs, features, or documentation updates. That first step not only gave us more visibility into the shape backlog, but also passed helpful context to the other specialized agents we built later.\n\nWe prototyped multiple ways to build each new type of functionality, and in the process developed a set of guiding principles that shaped the architecture of the factory that went into production.\n\nOnce our classification agent reached a high level of accuracy, we focused on automating bug reproduction, fixes, and review of those fixes. We explored building a single agent equipped with skills for each step, but quickly realized that would come with a higher maintenance and troubleshooting burden over time.\n\nInstead, we built a single agent for each specific task, making every new capability easier to reason about, test in isolation, and debug. Each one is scoped to a specific job, with its own prompts, context, and evals.\n\nToday the factory has dedicated agents for every step in the flow:\n\nBug reproduction\n\nBug fixes\n\nPR reviews\n\nBackports\n\nDocumentation updates\n\nFeature analysis\n\nFeature implementation\n\nWe implemented security with the second agent, because bug reproduction was the first step where the factory executed code based on content it didn't control.\n\nA factory operating on a public repository has to assume attacker-controlled input: every issue, pull request, comment, and the links inside them are untrusted. Because successful open-source projects are high-value targets, threats range from malicious code changes and supply chain attacks to resource exhaustion, API key exfiltration, and prompt exfiltration.\n\nSandboxes are the foundation of our defense. Every agent in `ai-sdk-factory`\n\nruns inside an isolated Vercel Sandbox containing its code, its runtime, and only the secrets the agent's specific task needs. With those guardrails, untrusted content can shape what an agent proposes, but the damage any one task can do is contained to the sandbox.\n\nWe also built a shielding layer around the sandbox that controls what agents can reach over the network, blocking the paths an attacker would use to pull secrets out of the isolated environment.\n\nThe last line of defense is human review: nothing is merged without approval from a human on the AI SDK team.\n\nThe first several agents we built for the factory ran through a local CLI. That enabled our team to iterate quickly as we noticed inaccuracies, felt friction, and prototyped different ideas.\n\nOnce multiple steps were running reliably through the CLI, we were ready to move the system onto managed infrastructure. `ai-sdk-factory`\n\nuses:\n\nVercel Functions for the API, workers, and webhook ingress\n\nVercel Queues for task execution\n\nVercel Blob for logs\n\nVercel Sandbox for the agent workspaces\n\nNeon Postgres for factory data\n\nToday, GitHub webhooks feed the issue queue, and as soon as they arrive workers automatically pull them and kick off agent runs in sandboxes. We also built a monitoring UI that tracks every run in parallel, and visualizes the queue for the team of reviewers.\n\nOn July 24, a community member asked for blocked-domain support in OpenAI web search, and the request became [issue #17898](https://github.com/vercel/ai/issues/17898). The following section explains every step the software factory went through to process the issue, open a pull request with an implementation of the feature, and ultimately backport the merged feature to v5 and v6 of the SDK.\n\nThe first agent that runs in the factory classifies issues and pull requests. In this case, the `ai-sdk-factory`\n\nbot [commented](https://github.com/vercel/ai/issues/17898#issuecomment-5101009222) on the issue and applied a label, identifying the type as Feature with high confidence. The agent included its rationale for the classification in the comment.\n\nAfter classification, the analysis agent runs. Agents in the factory don't make any assumptions about the technical validity of a feature request or bug, so the analysis agent wrote a probe to confirm the absence of support for blocked domains. `issue-17898-type-probe.ts`\n\nwas generated and run, and failed with an error when it looked for blocked-domains and didn't find it.\n\nThe failing probe proved the feature was missing on `main`\n\n, and the agent included it in [the analysis](https://github.com/vercel/ai/issues/17898#issuecomment-5101058789) as evidence.\n\nThe analysis agent then used the results of its investigation to build out a spec for the feature: add an optional `blockedDomains`\n\nfilter to the existing web-search tool and map it to the provider's `blocked_domains`\n\nfield.\n\nThe agent also confirmed the spec fit the SDK's provider-adapter architecture and was backward compatible, and even scoped documentation changes.\n\nAnother agent then [implemented the spec](https://github.com/vercel/ai/issues/17898#issuecomment-5101199946) and opened [pull request](https://github.com/vercel/ai/pull/18033). The implementation agent ran a live end-to-end test, executing an OpenAI web search with wikipedia.org blocked and confirming the domain wassn't reachable. The test was included as additional evidence on the pull request.\n\nNext, a review agent [scored the change](https://github.com/vercel/ai/pull/18033#issuecomment-5101303161), and when it didn't find any concerns, approved it. The agent rated the feature as fully implemented, with:\n\nSide-effect risk: low\n\nPerformance risk: none\n\nBackwards-compatibility risk: low\n\nFinally, Lars read the chain of evidence from the agents, reviewed the code changes, and merged [PR #18033](https://github.com/vercel/ai/pull/18033) into main.\n\nOnce Lars merged the initial PR, `ai-sdk-factory`\n\nopened additional PRs for two backports, [#18035](https://github.com/vercel/ai/pull/18035) for v6 and [#18036](https://github.com/vercel/ai/pull/18036) for v5. The v5 backport did not apply cleanly, so the factory agent labeled and committed the conflicted state, identified and validated a fix, and pushed it seventeen minutes later. After review, Lars merged both backport PRs.\n\nWe are just over four weeks into running the software factory in production. Here are the results:\n\n**PRs merged to main**\n\nBetween 25-35% of the PRs we merge on a weekly basis are now authored by `ai-sdk-factory`\n\nagents.\n\n**Backports**\n\nFactory PRs are above 50% of weekly merges to the v6 release line, and v5 looks similar. Backports used to be work we skipped because dealing with merge conflicts wasn't worth the effort. v5 and v6 get far better support now.\n\n**Issues**\n\nIn July, over 75% of closed issues were closed by the factory.\n\nOpen issues fell from a peak of 1,022 in late June to 844 by early August, and open bugs are down roughly 25%.\n\nThe `ai-sdk-factory`\n\nruns in public, so you can see [every pull request](https://github.com/vercel/ai/pulls?q=is%3Apr+author%3Aapp%2Fai-sdk-factory) it has authored on the repo.\n\nThe most interesting part of running the factory is what happens when it fails. Every run ends one of four ways: success, flawed, blocked, or manual. Only success ships, so the rest are signal that re-enters the system as feedback.\n\nA flawed run means an agent produced the wrong thing, and the fix is better prompts, better context, or a new eval case so the same mistake gets caught automatically next time\n\nA blocked run means the environment was missing something, like a credential, a service, or a dependency, and the fix is provisioning it\n\nA manual run marks a boundary we drew on purpose, and forces us to ask whether improvements in the factory justify removing it\n\nEach of those fixes expands the automation boundary, and every week the factory can handle work that it couldn't be trusted with before.\n\nRunning the factory is the same discipline teams already apply to their test suites and pipelines, but pointed at the system that does the work, instead of the system that checks it. In a world where agents define the SDLC, improving the factory will become the standard engineering job.", "url": "https://wpnews.pro/news/building-a-software-factory-for-ai-sdk", "canonical_source": "https://vercel.com/blog/building-a-software-factory-for-ai-sdk", "published_at": "2026-08-12 00:00:00+00:00", "updated_at": "2026-08-12 16:46:48.732382+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "ai-infrastructure"], "entities": ["Vercel", "AI SDK", "ai-sdk-factory", "Anthropic", "Opus 4.6", "Mitchell Hashimoto", "Ghostty", "Simon Willison"], "alternates": {"html": "https://wpnews.pro/news/building-a-software-factory-for-ai-sdk", "markdown": "https://wpnews.pro/news/building-a-software-factory-for-ai-sdk.md", "text": "https://wpnews.pro/news/building-a-software-factory-for-ai-sdk.txt", "jsonld": "https://wpnews.pro/news/building-a-software-factory-for-ai-sdk.jsonld"}}