Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of five topics from the past issue of The Pulse*. Full subscribers received the article below two weeks ago. If you’ve been forwarded this email, you can* subscribe here*.*
incident.io (note: I’m an investor) is a software reliability platform founded in 2021, a year before ChatGPT, that’s obsessed with measuring everything. In that spirit, the company has shared details about AI spending, changes to PR frequency, and the ROI at the company from using AI. Thank you to Lawrence Jones and the incident.io team for sharing the data!
They have come up with the concept of the “effort point” as a way to analyse AI spend. One effort point is a medium-sized pull request written by a human or generated by an agent. The measurement is far from perfect, but it is one step more granular than looking at lines of code, and lets them compare the complexity of AI-generated PRs with those written by human devs in the past. Here’s what has changed in the last two years:
Pull request stats
At a glance:
AI agents started to generate production code in June 2025, which was around the same time as the first agent-generated got pushed at incident.io. The company has used AI-assisted IDEs since the beginning, such as GitHub Copilot and Cursor.
In June this year, a few things happened in quick succession, the incident.io team told me:
- Claude Agents launched, making it easier to coordinate several agents running at once
- Worktree tooling improved a lot, making it easier for agents to do parallel work on the same codebase with worktrees
- Also around this time, more engineers realized that they should be running multiple agent sessions using different worktrees, rather than driving just one session at once.
Today, nearly 80% of PRs that ship to production include code generated by an agent:
More:
- 2.7x headcount growth in product engineering (engineers who ship to production numbered 18 in 2024, and 50 today)
- 3.4x PR requests shipped to production in September compared to a year ago
- 3.6x increase in PRs with no human reviewer shipped to prod (12% to 43%)
- 4.8x increase in the size of PRs (from 22 to 111 lines)
- 6x “effort points” shipped to prod: “effort points” awarded by an AI model estimating the complexity of a PR, and ranking it from XS to XL.
- 80% decrease in human comments per “effort point”
- 60% decrease in incidents per 1,000 pull requests
Stats on release frequency:
- 47% rise in releases/eng/day when taking headcount into account: an average of 2.8 releases/day/eng, up from 1.9 releases/day/eng in 2024
- Oct 2026: 140 releases/day to production with 50 engineers. In Oct 2024, the figures were 35 releases/day to prod with 18 engineers
AI spend
This year, spending on AI tools is on track to be around $1.6M at the company – which is a lot! We’re talking around $2,700 per engineer each month, which puts the reliability startup among the big spenders of companies mentioned in this article. Based on that, they’ve attempted to calculate a rudimentary ROI for the spend. At present, incident.io sees AI tools as mostly positive.
AI spending peaked this summer and has been decreasing since. In July, the company spent more than $300K on AI (an annual projection of $3.6M, had it continued). Here’s the allocation:
Opus 5.5 brought a massive cost reduction. Before it, the company was spending around $4,500/engineer/month, mostly on Opus 5. But now, with the 5.5 version, the cost is down to circa $1,600. Here’s the impact of Opus 5.5 on monthly spend:
Also, the adoption of Opus 5.5 has been rapid:
Cost per “effort point” also peaked over the summer. The nice thing about the “effort point” metric is that it makes it possible to compare the cost of generating a PR to how much it cost a year ago, when the company started using AI agents. Here’s the change:
Infrastructure teams spend more tokens on the same “effort point.” A fascinating detail from incident.io comes from separating the cost of an “effort point.” The infrastructure team stands out, spending ~5x more on a medium-sized PR than almost every other team:
But why the big difference? Lawrence Jones, an engineer at incident.io, explains:
“This is a great example of why it’s hard to use a single target for cost per PR that applies universally.
Our AI team, for example, does work that can bevery token-intensive such as iterating on evals. Infrastructure teams might use Claude to explore production environments or do lots of analysis or run benchmarks etc which don’t lead to PRs, so their cost per-PR looks higher.
Different types of work will simply have different cost profiles.”
Measuring ROI
Today, average spend is down at ~$1,600/engineer/month, of which around 80% is on Claude, with 20% going to Cursor. The big question: has AI made engineers more productive? Unfortunately, measuring engineering productivity in only a single dimension is fraught with hazard, but a combination of insights could give a sense of where things are headed. These are imperfect measurements with their own biases, but are still interesting.
The number of PRs shipped per engineer weekly tracks with tenure. This is a very interesting finding from incident.io. Compared to two years ago:
- The median engineer shipped ~10% more PRs (10.1 PRs in Oct 2024 with no AI; 10.7 PRs per week in Oct 2026 with AI).An eye-catchingly small change!
- The mean engineer shipped ~30% more PRs (11.1 PRs in Oct 2024 vs 14.8 PRs in Oct 2026)
- Tenured engineers shipped ~70% more PRs (11.1 PRs in Oct 2024 vs 19 in Oct 2026)
New engineers are told to hold off on using AI for a month after they join. The AI SRE startup collected feedback from new engineers and found that new joiners using a lot of AI right after onboarding ended up performing worse in the medium/long term.
What incident.io found is that it’s worth ignoring expectations of shipping velocity during the first month. Instead, new devs learn how to navigate the codebase. This AI-abstinent approach skews the above numbers.
Harder to quantify: shipping faster & incidents
Compared to pre-AI, how important is it for PRs to be shipped faster to production? A benefit of AI agents and AI reviews is faster feedback than waiting on someone to review a PR. Here’s how quickly the median PR makes it to production at the reliability startup:
Let’s take an average day – a Tuesday last month – when 185 changes were shipped to production, 75 of which without human review:
There are fewer incidents per change, but more overall. It would be quite the omission to not bring up reliability when talking about a reliability startup! This is both good and bad news: there’s fewer incidents per PR, but more overall due to the existence of more PRs:
The good news is that the rate of incidents is lower than expected, given the PR volume:
The team looked into why incidents occurred, and found that most outages were because of incorrect code that could have been caught in review. The team took steps to address this:
- Improved the codebase for AI use
- Started leaning more on Cursor’s Bugbot, added stricter rules, and saw the tool start to give better feedback and catch more code-related bugs in review
- Newer models produced better code.
Today, most incidents are no longer due to faulty code, but because there’s information absent about the production environment. The bad news is that if this continues and PR volume grows, then the number of incidents will also rise unless the team avoids outages related to production behavior. They are working on building feedback loops where agents can observe and learn from production.
I found these data points educational, especially looking at how the “cost per medium-sized PR” does (and probably should!) vary based on the complexity of work that goes into it. Thanks a lot to the incident.io team for sharing such detailed data!
Read the full issue of The Pulse from two weeks ago**.** The latest The Pulse covers:
- What can we learn from the Railway hack? Cloud platform Railway has published an extremely detailed postmortem about their recent hack in which the attacker likely used ‘abliterated’ AI models (those with no safeguards). Railway has hardened its systems as a result. Also, a competitor tried to benefit from the incident by getting Railway’s customers to migrate – but then got hacked itself in the exact same way.
- Native mobile and frontend jobs are disappearing. New data shows native mobile and frontend job listings are down by 50% at leading startups and Big Tech, while forward deployed engineer (FDE) listings have quadrupled in the same timeframe.
- How an extremely efficient engineering team works: Superlogical. Mitchell Hashimoto’s new startup is “AI-native” and cares deeply about craft and developer tools. They skip code reviews and commit to main thanks to deep trust between team members.
- Industry pulse. Jensen Huang no longer the CEO of NVIIDA, OpenAI buys Pierre Computer, Carta starts selling a coding harness, Meta buys and shuts down Conductor (a T3Code competitor), and Binance hacked with $6.3B stolen after devs were tricked into down packages and attending fake interviews.
Subscribe to my weekly newsletter to get articles like this in your inbox. It's a pretty good read - and the #1 software engineering newsletter on Substack.