Anthropic published a post about their CI infrastructure that I keep coming back to.
The headline number is that Claude writes 80% of their code. That's the part everyone quoted. The part that stuck with me was further down: their test suite grew 10x, and CI jobs went up 25x over six months.
Read quickly, that's an infrastructure scaling story. Buy more compute, move on.
I think it's something else. We spent years optimizing how fast developers could produce code, and AI is now removing that constraint faster than the systems around it can adapt. When code generation stops being the bottleneck, something else takes over. Going by Anthropic's numbers, that something is verification.
There's an assumption buried in most AI coding conversations: if developers build faster, the whole lifecycle gets faster.
It doesn't work that way.
Say a developer used to ship five meaningful changes a sprint, and an agent helps them ship twenty. You haven't just raised development throughput. You've raised pull requests, code paths, test executions, review load, deploys, and the number of failures somebody has to investigate. The generation side scaled. Everything downstream has to scale with it or it backs up.
That's what a 25x CI increase looks like from the inside.
Software delivery has always had multiple constraints, and development was one of the biggest. AI compresses that one hard, so the upstream pipe gets wider.
But code review, CI capacity, observability, and deployment controls don't speed up just because code generation did. AI doesn't eliminate bottlenecks. It relocates them.
When the cost of making a change drops, the cost of proving that change is safe is suddenly the number you notice.
The most useful detail in Anthropic's post is their test selection service. They don't run every test on every change. They run a deterministic test impact analysis that picks tests based on package relevance and historical performance.
That distinction matters more than it sounds.
If an agent makes a two-line change and you answer with a full regression suite every time, you're burning compute and the feedback window the agent needs to correct itself. The other extreme is worse, though. Asking an LLM "which tests should we run here?" isn't a verification strategy, it's a guess with good grammar. Their answer is deterministic, and I think that's the right call. This isn't traditional testing versus AI testing. The interesting systems will use both, with the deterministic layer carrying the decisions you can't afford to get wrong.
Here's a difference I didn't expect to matter as much as it does.
A human developer sees a red test and thinks "that's unrelated to my change." They're usually right, because they carry context the test output doesn't contain. An agent doesn't have that unless you hand it over explicitly.
Anthropic notes that when agents get a specific set of valid tests, they self-verify and iterate more effectively. So the test suite stops being a gate at the end of the process. It's the signal the agent steers by: write, test, read the result, correct, test again.
That changes what a slow test suite costs you. Before, it annoyed a developer. Now it throttles an autonomous loop that would otherwise run unattended. Feedback latency is a throughput number, not a comfort one.
QA conversations used to center on "can we automate this test?"
I think the question now is whether your quality system can keep pace with development that doesn't sleep. Those aren't the same problem, and having four thousand automated tests doesn't mean you've solved the second one.
What I'd actually want to know: which tests run and when, what signal goes back to the agent, and whether the agent can tell a real regression from a flaky failure. Then the harder one. Which decisions stay deterministic, and where does a human still have to sign off?
Anthropic's setup keeps humans driving and approving a significant share of PRs. That's worth noting, because more automation didn't mean less control. It meant the control system had to handle more volume.
AI writes tests fast too. Sounds like good news until you look at what you get.
Test count and test quality aren't the same thing. An agent will happily produce twenty tests covering variations of the same behavior while missing the failure mode that would actually page someone at 2am.
So "we generated 5,000 more tests" is a terrible metric. The question I'd rather answer is what risk those tests cover that wasn't covered before. Are the assertions meaningful, or are they asserting that a function returns something? Are they coupled so tightly to the implementation that the next refactor deletes all of them?
Without that, AI creates a loop that looks productive and isn't. More code, more generated tests, more CI jobs, a bigger bill, roughly the same confidence. Automation theatre, at a scale nobody has ever run it at before.
I don't think a quality engineer's value in two years comes from executing test cases by hand. I don't think writing automation scripts is enough either.
The work moves up the stack: designing the system that decides what's risky, what has to be tested, which checks stay deterministic, and when an agent gets to self-correct versus when a human has to look. That's a more interesting job than receiving a finished feature and testing it at the end.
Quality gets embedded in the environment the agent runs inside. It isn't the last checkpoint. It's part of what makes autonomous development trustworthy enough to leave running.
There's a version of this that goes badly.
AI makes development faster. Leadership sees throughput climb. The reasonable-sounding response is "great, we can ship more." But if verification hasn't changed at the same rate, what you've built is more code, more changes, faster merges, the same infrastructure, and less confidence per change than you had before.
The wrong lesson is that moving faster means fewer checks. If machines can produce changes faster than humans ever could, the verification systems need to be faster and smarter than anyone designed them to be. Speed upstream raises the value of confidence downstream.
The industry spent decades optimizing code production, and AI may have finally made it abundant. But abundant code means abundant surface area. More to review, more to trust.
That's why the number worth staring at in Anthropic's post isn't the 80%. It's the 25x that came after it.
The question stops being how fast we can generate software. It starts being how fast we can prove the software deserves to ship.
If you want somewhere concrete to start: work out what your CI bill and your merge queue look like if your team's output triples next quarter. The answer is usually uncomfortable enough to be useful. Reference: Agentic coding is straining CI. Here's how we scaled test impact analysis at Anthropic.