The AI SDK is one of the most popular open-source AI projects in the world. It serves over 20 million npm downloads a week and the repo has over 26,000 stars. Maintaining the codebase means tracking four moving targets at once:
Model providers: new providers, new capabilities, and new bugs
UI frameworks: bindings for React, Next.js, Svelte, Vue, and others
Sandboxes: the execution environments agents run code in
Harnesses: adapters for Codex, Claude Code, Pi, and others
After multiple years of growth, the repo was getting 100+ new issues every month, and when Anthropic's Opus 4.6 model was released, PRs hit an inflection point. By late June, that compounding had accumulated over 1,000 open issues and almost 800 pull requests.
That backlog is not a discipline problem. No maintainer, however good, can close that gap by working harder, and because generating code is cheap, it will only grow.
Instead of trying to scale ourselves, we built a software factory. Four weeks in, it authors between 25 and 35% of PRs we merge and closes 70-80% of issues.
Before we built anything, we had to answer three questions:
Why our existing approach using agents wasn't enough
What level of automation fit a project like the AI SDK
How to align automation and human effort to risk
The best maintainers are already using agents aggressively. Mitchell Hashimoto runs Ghostty with the goal of an agent always working, and encodes every agent failure in AGENTS.md so it never repeats. Simon Willison runs four coding agents in parallel, and review bots like Vercel Agent and CodeRabbit sit on millions of repos. Other maintainers like Daniel Stenberg have opted to block AI-generated submissions to curl.
All of it helps, but none of it solves for the core constraint: every one of these solutions still routes every change through one human's attention. We believe that human accountability is still the core of trust in agentic engineering, so we knew our factory needed to solve for reviewer efficiency as a first principle.
Software factories sit on a spectrum. At one end is full automation, where agents write, ship, and deploy without a human ever reading the code. In the middle are harnesses like Codex and Claude Code, where a person steers an agent or fleet of agents. At the far end is software you barely want to automate because the risk is too high, like firmware in a pacemaker or self-driving vehicle.
The AI SDK needs to operate closer to the careful end. It is foundational AI infrastructure with millions of applications built on top, so quality and security are non-negotiable, and a human needs to have control over what ships. We needed our factory to heavily automate the lifecycle around the human, without removing them.
For a given change, the depth of human review needed scales with risk. Centralized roadmaps with detailed feature specs can define risk up front and directly shape the work going into a software factory. But open-source projects like AI SDK also get issues and pull requests from the community, and there is no guarantee they align with the goals of the project, or that changes are safe to make. Higher risk means human judgment is an even more critical part of the system.
To optimize our factory for that judgment, we knew agents had to go beyond generating code based on a request; they needed to evaluate full units of work in the context of the entire project.
Our goal was for the factory generate a comprehensive assessment of each change for both fit and risk, including a full chain of documented evidence, making it easy for reviewers to apply the right amount of effort:
Docs fixes get a quick glance for verification
Well-defined provider changes get focused validation
A new public API gets deep review
ai-sdk-factory
is a software factory that autonomously processes incoming issues and pull requests for AI SDK. Agents in the factory perform specific, reviewable tasks, like reproducing bugs, implementing features, and creating backports for older SDK versions. A human stays in control throughout the entire process, including merging every change.
We didn't ship the factory in one swing. We built it incrementally, starting at the beginning of the process with classification of issues as bugs, features, or documentation updates. That first step not only gave us more visibility into the shape backlog, but also passed helpful context to the other specialized agents we built later.
We prototyped multiple ways to build each new type of functionality, and in the process developed a set of guiding principles that shaped the architecture of the factory that went into production.
Once our classification agent reached a high level of accuracy, we focused on automating bug reproduction, fixes, and review of those fixes. We explored building a single agent equipped with skills for each step, but quickly realized that would come with a higher maintenance and troubleshooting burden over time.
Instead, we built a single agent for each specific task, making every new capability easier to reason about, test in isolation, and debug. Each one is scoped to a specific job, with its own prompts, context, and evals.
Today the factory has dedicated agents for every step in the flow:
Bug reproduction
Bug fixes
PR reviews
Backports
Documentation updates
Feature analysis
Feature implementation
We implemented security with the second agent, because bug reproduction was the first step where the factory executed code based on content it didn't control.
A factory operating on a public repository has to assume attacker-controlled input: every issue, pull request, comment, and the links inside them are untrusted. Because successful open-source projects are high-value targets, threats range from malicious code changes and supply chain attacks to resource exhaustion, API key exfiltration, and prompt exfiltration.
Sandboxes are the foundation of our defense. Every agent in ai-sdk-factory
runs inside an isolated Vercel Sandbox containing its code, its runtime, and only the secrets the agent's specific task needs. With those guardrails, untrusted content can shape what an agent proposes, but the damage any one task can do is contained to the sandbox.
We also built a shielding layer around the sandbox that controls what agents can reach over the network, blocking the paths an attacker would use to pull secrets out of the isolated environment.
The last line of defense is human review: nothing is merged without approval from a human on the AI SDK team.
The first several agents we built for the factory ran through a local CLI. That enabled our team to iterate quickly as we noticed inaccuracies, felt friction, and prototyped different ideas.
Once multiple steps were running reliably through the CLI, we were ready to move the system onto managed infrastructure. ai-sdk-factory
uses:
Vercel Functions for the API, workers, and webhook ingress
Vercel Queues for task execution
Vercel Blob for logs
Vercel Sandbox for the agent workspaces
Neon Postgres for factory data
Today, GitHub webhooks feed the issue queue, and as soon as they arrive workers automatically pull them and kick off agent runs in sandboxes. We also built a monitoring UI that tracks every run in parallel, and visualizes the queue for the team of reviewers.
On July 24, a community member asked for blocked-domain support in OpenAI web search, and the request became issue #17898. The following section explains every step the software factory went through to process the issue, open a pull request with an implementation of the feature, and ultimately backport the merged feature to v5 and v6 of the SDK.
The first agent that runs in the factory classifies issues and pull requests. In this case, the ai-sdk-factory
bot commented on the issue and applied a label, identifying the type as Feature with high confidence. The agent included its rationale for the classification in the comment.
After classification, the analysis agent runs. Agents in the factory don't make any assumptions about the technical validity of a feature request or bug, so the analysis agent wrote a probe to confirm the absence of support for blocked domains. issue-17898-type-probe.ts
was generated and run, and failed with an error when it looked for blocked-domains and didn't find it.
The failing probe proved the feature was missing on main
, and the agent included it in the analysis as evidence.
The analysis agent then used the results of its investigation to build out a spec for the feature: add an optional blockedDomains
filter to the existing web-search tool and map it to the provider's blocked_domains
field.
The agent also confirmed the spec fit the SDK's provider-adapter architecture and was backward compatible, and even scoped documentation changes.
Another agent then implemented the spec and opened pull request. The implementation agent ran a live end-to-end test, executing an OpenAI web search with wikipedia.org blocked and confirming the domain wassn't reachable. The test was included as additional evidence on the pull request.
Next, a review agent scored the change, and when it didn't find any concerns, approved it. The agent rated the feature as fully implemented, with:
Side-effect risk: low Performance risk: none
Backwards-compatibility risk: low Finally, Lars read the chain of evidence from the agents, reviewed the code changes, and merged PR #18033 into main.
Once Lars merged the initial PR, ai-sdk-factory
opened additional PRs for two backports, #18035 for v6 and #18036 for v5. The v5 backport did not apply cleanly, so the factory agent labeled and committed the conflicted state, identified and validated a fix, and pushed it seventeen minutes later. After review, Lars merged both backport PRs.
We are just over four weeks into running the software factory in production. Here are the results:
PRs merged to main
Between 25-35% of the PRs we merge on a weekly basis are now authored by ai-sdk-factory
agents.
Backports
Factory PRs are above 50% of weekly merges to the v6 release line, and v5 looks similar. Backports used to be work we skipped because dealing with merge conflicts wasn't worth the effort. v5 and v6 get far better support now.
Issues
In July, over 75% of closed issues were closed by the factory.
Open issues fell from a peak of 1,022 in late June to 844 by early August, and open bugs are down roughly 25%.
The `ai-sdk-factory`
runs in public, so you can see [every pull request](https://github.com/vercel/ai/pulls?q=is%3Apr+author%3Aapp%2Fai-sdk-factory) it has authored on the repo.
The most interesting part of running the factory is what happens when it fails. Every run ends one of four ways: success, flawed, blocked, or manual. Only success ships, so the rest are signal that re-enters the system as feedback.
A flawed run means an agent produced the wrong thing, and the fix is better prompts, better context, or a new eval case so the same mistake gets caught automatically next time
A blocked run means the environment was missing something, like a credential, a service, or a dependency, and the fix is provisioning it
A manual run marks a boundary we drew on purpose, and forces us to ask whether improvements in the factory justify removing it
Each of those fixes expands the automation boundary, and every week the factory can handle work that it couldn't be trusted with before.
Running the factory is the same discipline teams already apply to their test suites and pipelines, but pointed at the system that does the work, instead of the system that checks it. In a world where agents define the SDLC, improving the factory will become the standard engineering job.