The Honeycomb engineering team set out to double our productivity in a year. Our most important takeaway? Nothing anyone tells you about AI will land if your starting substrate is unhealthy.
By: Liz Fong-Jones
The Second Edition of Observability Engineering Is Here
The second edition of Observability Engineering is available for download on our website.
Download In this two-part blog series, I give a detailed report-out on how our Honeycomb engineering team 2.5x-ed our throughput using AI without breaking everything or lowering our standards for quality. Part 1 explains how we did it and shows data about how that ramp-up happened. In this blog, I share what we learned.
The “platform engineering” frame and the “autonomy, ownership, feedback loops” frame are the same frame, spoken in two different vocabularies. AI amplifies your existing practices. It can make a dysfunctional org more dysfunctional, or it can bring out the best in an org that has high autonomy, ownership, and feedback loops.
Autonomy is the ability to ship without asking permission. Continuous delivery, hermetic builds, feature flagging, fast CI: this is the substrate that lets work flow. Skip it, and your build queue saturates the moment AI shortens the writing loop.
Ownership is accountability for what gets shipped, even when AI wrote it. CLAUDE.md and skills, code review, blameless incident analysis, the post-incident substrate updates that close the loop on whatever slipped through: these are how teams stay accountable to the system as a whole, regardless of the skill of whoever’s driving the tool. Skip these, and “the AI did it” becomes the default explanation for every production bug.
Feedback loops are how you learn whether the work actually landed. MCP-mediated dev-loop access, closed-loop production observability, SLOs, and a dissemination layer that surfaces what shipped to the humans who need to know: these are how the agent and the human both find out whether what they did worked. Without them, AI just gives you faster shipping of code whose effects nobody’s measuring.
Before any of this can help you, your underlying org has to be functional in the first place. *Nothing Honeycomb, or Intercom, or anyone else on a stage tells you about AI will land if your starting substrate is unhealthy. *That’s the most important sentence in this post.
New Engineering Realities, New Questions: A live AMA with the authors of Observability Engineering
July 29 | 10 a.m. PT / 1 p.m. ET
Continuous delivery: Bound the duration of any individual bad PR #
We run an hourly deploy train. Twelve to fourteen deployment events on a typical workday, packing about 70 PRs across them, with one to three reverts. Recovery is within the hour, because the next train is right behind. This bounds the duration of any individual bad PR’s impact in production. It doesn’t bound the radius; when something deploys, everyone gets it. But the duration of “everyone gets it” is short enough that we can absorb the cost.
Without this, none of the rest works. The agent feedback loop—i.e., Did this code do the thing in production?—doesn’t close. A merged PR that ships in six weeks effectively doesn’t exist as far as the agent is concerned.
Feature flags are a complementary lever. Train architecture bounds incident duration; feature flags bound blast radius at the level of the individual change, since one flagged change can be turned off without rolling back the whole train. Both reduce change failure rate when something does slip through. Feature flagging is an under-emphasized partner to CD, especially for AI-shipped work, now that the marginal cost of “let’s gate this behind a flag” has gone down.
What’s coming next is what we’re calling actually-continuous delivery: batches shipped as fast as they can be safely verified, rather than on an hourly schedule. We’d wanted this for two to three years already; the increased PR rate made it a necessity rather than a desire, and AI made it cheaper to build in the first place, since writing the code for it got cheaper too. Each platform investment unlocks the next one.
CLAUDE.md and skills: Code ownership reframed
DORA-style code ownership was about authorship: who wrote the code, who’s on the hook to maintain it. With agents writing the majority of new lines, authorship is no longer load-bearing. What survives is accountability for what humans allow AI to ship, even when humans no longer wrote the code themselves.
CLAUDE.md is “what we will ship and what we will not,” at codebase or team level. Skills are reusable, specialized capabilities the agent can pick up for specific tasks. Together, they make team standards legible to the substrate that’s actually writing the code.
When something slips through that both the human driving the AI and the human reviewer missed, the miss becomes a substrate update. It goes back into CLAUDE.md, into a skill file, or into a review-agent rule. Accountability here isn’t “we’re on the hook.” It’s “we close the loop when we miss.”
The blameless-incident reframe isn’t “blame the typewriter” or “blame the typist.” It’s that we’re accountable for the system as a whole, regardless of the skill of whoever’s driving the tool. The integration matters more than any of the individual components.
Auto PR review, with humans empowered to push back
Auto-review agents catching style violations and obvious holes are useful, and they don’t substitute for senior engineers willing to say, “This is slop, this is waste, this is over-verbose, this is missing tests,” out loud and in writing. Humans empowered to push back, including pushing back vigorously, is the load-bearing practice here.
Two worked examples from the public record. In one PR, closed unmerged, I’d pushed an AI-driven optimization. Ian Wilkes escalated through three rounds of review, ending with: “You need to test code like this thoroughly. Given the opportunities for problems here I would be fine with the original, slower sorted set implementation.”
The auto-review bot also weighed in. It claimed a “high-severity race condition” in the relevant cache. Ian: “This is false. That cache only stores directory names. I think it’s super confused. Definitely don’t take Claude’s word for it.” The bot’s claim was specific, confident, and wrong; the cache it flagged stores directory names, and the race it described couldn’t physically happen against that data structure. Without Ian’s codebase context, that false signal would have driven defensive changes nobody needed.
What did ship was a simpler version that took the general-purpose SortedSet
library type from the earlier commit as a drop-in replacement for the Set in one specific hot path. The bsearch-by-prefix optimization from the original was deferred. It was roughly 80% of the gain at 10% of the complexity. Ian had speculatively asked for the AI optimization (because how hard could it be to make Claude spit out the code?), then had regrets once he was reviewing it, and we landed on a simpler approach together. The experimentation gave us value. Just not the value we originally thought we’d get. Just because you can do something with AI doesn’t mean you should.
Finally, in a third PR that was ultimately merged after rework, the discipline was escalating attention. The first review was architecture-only; Ian wouldn’t engage with line-level detail until the structure was right. After the rework, he left 12 inline comments. The most pointed of them: “obvious AI slop.” On the third review pass: “OK, let’s try it.” Merged.
A senior engineer’s detailed attention is allocated, not assumed by default; that’s the inverse of typical reviewer behavior. And the practice has to reproduce all the way down. Both Ian and I are senior in our orgs. Two distinguished and principal engineers arguing is the easy case. The hard case is junior engineers blocking principal engineers’ PRs on the merits. Hierarchy shouldn’t insulate me from review, and seniority isn’t what makes the practice work.
MCP: Give the agent the tools the human had #
In a single Claude Code session or Slack thread (pointing autobot at the issue), an engineer can file a Linear ticket, fix the issue, and submit a PR pointing back at the ticket. Intent capture happens automatically, because the ticket lives in the same conversation as the fix. The cost of capturing intent goes to near-zero. It doesn’t stop mattering; the audit trail and the design rationale still matter, and now they’re recorded by default rather than skipped because nobody had time.
Closed-loop conversation needs two categories of MCP server, and most teams under-invest in the second one. Tooling-side servers (Honeycomb for production telemetry, Playwright for UI verification) give the agent runtime feedback. Documentation-and-ticket-side servers (Notion for design docs and RFCs, Linear for tickets, user stories, and acceptance criteria) give the agent the user story, design intent, and verification criteria the human originally had.
An agent without these tools is flying blind in a way a junior engineer wouldn’t be. A junior engineer would pull up the dashboard, click through the UI, read the doc, read the ticket. The principle generalizes well beyond engineering; revops connecting HubSpot, Salesforce, Gong, and Apollo via MCP for Claude is the same pattern, applied to a different function.
Fast and AI-legible CI #
Two things have to be true for CI to function as part of an agent’s feedback loop. It has to be fast, and it has to be AI-legible.
Mainline CI builds per day (top) and build-duration heatmap (bottom), September 2025 through June 28, 2026. The late-March step change is the CI investment landing: builds-per-day roughly doubled while duration halved. Spliced from two CI systems, CircleCI through March 23 and Mint (RWX) after, both counting end-to-end mainline pipeline runs.
Fast comes first, because it’s the half most rooms underweight. AI shortens the writing loop, so the testing loop has to shorten with it. A 60-minute coding session against a 15-minute CI run is a fine ratio. A 5-minute AI-generated change against that same 15-minute CI run means the build queue dominates both wall-clock time and agent iteration speed. As AI raises the rate at which changes get authored, build time has to come down to keep the loop closed. Fast CI is no longer a nice-to-have. It’s now a precondition for AI throughput to be worth anything at all.
AI-legible means the CLI is readable by agents, especially with skill files teaching them how to drive it. Agents run code analysis via the CLI, and fix their own build failures fast. We use RWX; the principle generalizes beyond the specific vendor. RWX hosts our Claude Code auto-review agents too. It isn’t itself an AI tool, and that’s rather the point: feedback infrastructure has to be AI-legible and fast, not necessarily AI-native. MCP servers are one way to get there; well-designed CLIs with skill files are another.
The flipside worth flagging: some CI vendors’ MCP wrappers add overhead that’s net-negative. Plain CLI is often just faster, so long as the agent can figure out how to drive it.
Closed-loop observability for agent work #
The trap to avoid is treating “agentic observability” as monitoring the agent itself. Tokens spent, sub-agent decisions, internal flow: these are interesting, and they’re not the point. The point is monitoring the consequences of what the agent shipped, in production, on real users.
Three things matter here.
**Instrument the shipped code, not just the agent. **Your production data should let you find what the agent shipped, later. Span attributes that link the running code back to the PR and the agent session that created it let you ask, “This PR shipped Tuesday; what’s it doing in production right now?” That’s the loop you actually need to close.
**Production outcomes matter more than agent activity. **Latency, error rate, cost, user behavior on agent-shipped features. Did it work? What happened to real users? An agent that took 47 seconds and made 12 sub-agent calls tells you nothing about whether the change improved or broke anything. Production telemetry tells you that.
**Then there’s the closed-loop pattern itself: production signal feeding back into the next change. **Cost-per-interaction SLOs firing back into the agent’s context for the next iteration. Anomalies in BubbleUp auto-converting into review-agent rules, so the same shape of mistake can’t slip through twice.
The worked example, from our own customer base: the Honeycomb-Intercom case study. A cost-per-interaction SLO on Fin, an AI-built product, caught eager-request waste that finance would otherwise have found a quarter later. That’s observability of agent-shipped code catching a real issue, not observability of the agent’s internal flow.
Speed without closed-loop signals is enshittification. Speed with them is engineering.
AI for slop cleanup, not just generation #
When you buy a table saw, you don’t keep doing your woodworking the way you did with a hand saw. You redesign the work around what the table saw does well: long straight cuts, repeated dimensions, less fatigue per cut. The hand saw still earns its keep on what it’s good at; you just stop reaching for it on the things the table saw handles better.
AI is the same type of tool. The question isn’t whether to use it. It’s what shape your work takes once you have it. Migrations. Dead-code deletion. Dependency upgrades. Doc rewrites. Deletion is productivity.
AI lets you do the side quests: things we ought to be doing, that deliver quick wins, but that historically cost too much of a context switch to fit into anyone’s week. Dependency merges are the canonical example. A human’s honest response to Dependabot is “rubber-stamp it” or “ignore it,” because the time to read every changelog isn’t there. An agent has the time. It can read the changelog, run the regression tests, check for deprecations.
Reclaiming work that humans were doing badly anyway is reallocation, not a surrender of autonomy.
The dissemination layer #
Code review does two jobs: it catches bugs, and it produces shared understanding among humans about what’s changing. Automate the first, and the second needs its own surface area, decoupled from approval gating.
Concrete examples from our own production environment: an Argo bot that surfaces what’s deploying via Argo, scoped to the humans who care about it; a Deploy Train bot that gives visibility into the hourly train (who’s on it, what’s pending, what shipped); Terraform Hush, a TFC notifier built as an INSTALL.md-driven internal PaaS that tags humans on infrastructure changes; and a sister team’s hound-changelog watcher, surfacing upstream changes from our shared monorepo into a separate codebase that has to keep pace. All four of these were themselves built with AI. We used AI capacity to build the tooling that lets humans stay aware of what AI is shipping.
That’s “AI amplifies your existing practices” operating at the meta level: it amplifies your ability to build the practices you never had time to build before. The marginal cost of internal tooling has dropped far enough that small categories of dissemination tooling, which were never worth a full project, are now worth the afternoon it takes to build them.
Feature flagging and SLOs #
Feature flagging sits at the intersection of autonomy and feedback loops. AI changes ride behind flags the same way risky human changes always have. The new angle is that agents can be given the ability to gate their own changes for gradual rollout; the flag itself becomes the feedback signal for whether the change is doing what it claimed.
SLOs sit at the intersection of ownership and feedback loops. AI features need SLOs on cost-per-interaction and latency, not just availability. An SLO is an ownership statement, since you’re committing to a level, and a feedback mechanism, since you’re measuring compliance against it. It’s a constraint on autonomy bounded from outside, which is why it sits at this intersection rather than at the center of the framework.
Where the skeptics are right #
Two real concessions belong here.
The first is that bank line-of-business (LOB) engineering won’t benefit as much from this new AI-driven 2x results world. You’re not going to radically realign Line of Business (LOB) engineering at a bank. Most of what makes this work at Honeycomb, or at Intercom, is downstream of cultural conditions that don’t exist at a typical five-thousand-person regulated enterprise. This shape of story is reproducible at companies like Honeycomb or Intercom. It is not reproducible at the average bank’s LOB engineering org. If your organization can’t make the structural changes we and Intercom both made, cross-team realignment, plugin marketplaces, internal PaaS substrate, blameless accountability, you won’t get the same shape of result. Regulation isn’t actually the barrier; willingness to change how you work is. Bank LOB engineering is just the clearest example of an org that structurally can’t or won’t make that change, and any org in that position hits the same wall, regulated or not. That’s the realistic shape of this story for most companies.
The second concession is on spot-fix versus systematic-fix tension; complexity has a cost. Tickets aren’t the failure mode anymore; we always file them, programmatically enforced. The real worry is that AI makes spot-fixes so cheap that you keep reaching for them when a systematic answer, a refactor, an abstraction, a root-cause fix, would actually be better. Cause-fix cost held steady; symptom-fix dropped to near-zero; the ratio shifted underneath us. And the write-cost of code dropping doesn’t drag the maintenance-cost down with it. New code still has to be read, debugged, paged on, and onboarded against.
The receipts in the performance optimization PRs cash both halves of that concession. Spot-versus-systematic: “I would be fine with the original, slower implementation.” Complexity-has-a-cost: “this is a lot of kludge for a pretty fringe optimization.” Both sides show up in the same PR comments. The judgment call stays human. The skeptics are right that this is harder than it looks.
Order of adoption #
This is the question we get asked most: which platform practices to adopt, in which order.
There isn’t a strict ordering. CD and observability co-evolve, and you need both to close the agent feedback loop. Trying to do one without the other won’t unlock AI throughput on its own.
CD plus observability and SLOs form the paired foundation. Each one without the other is half a story. CD without observability is fast deploys you can’t debug. Observability without CD is a tight loop that takes weeks to act on. You can’t safely add AI throughput on top of either alone.
Code ownership plus CLAUDE.md and skills come next. Without these, AI amplifies confusion as agent-emitted volume grows.
Hermetic, speedy builds plus feature flags scale with you. Reproducibility for verification, gradual rollout for safety. Both matter more under AI throughput than they did before it.
Chaos engineering is audit, not foundation. Do it once the rest is solid. Trying chaos engineering before the foundation is in place is just unstructured outage.
Structuring the platform team #
In an AI-amplified world, the platform team’s job grew. Classic platform work still matters: CI/CD, infrastructure, internal tooling, eng-prod foundations. The new work is maintaining the agent substrate: CLAUDE.md cultivation, MCP servers, AI-legible CI, dissemination tooling, auto-review agent rules. That’s platform work, not eng-prod tools work, and it belongs to the same team.
Intercom’s pattern is a dedicated team, team-2x, running a Claude Code plugin marketplace with 153 contributors (31% of their R&D org) and 267 skills. The substrate is platform infrastructure that ships its own product, and modernizing the factory is now everyone’s job rather than something a platform team hands down to others. The platform team doesn’t just maintain the platform. In an AI-amplified world, it maintains the agent’s relationship to the platform.
Our platform team is busy building autobots for coding and review, to ensure that we can best allocate human attention where it matters, and to abstract the humans away from needing to stare at a Claude Code window all day long. The higher-value work is to have real human conversations with your teammates and stakeholders, and break up the work into the right chunks. That is your competitive differentiation. Everyone has access to Claude Code and the models. It’s what you choose to do with them that matters.
What’s left if you read nothing else #
AI amplifies your existing practices. Going fast without autonomy, ownership, and feedback loops is enshittification. Going fast with them is engineering finally working closer to the way DevOps promised it would back in 2014. Fix your org’s dysfunction first.
Capacity is a choice. AI gives you new capacity; the highest-leverage thing you can do with it is reinvest in the platform substrate that lets you absorb more of it, not spend it on marginal features. Actually-continuous delivery, AI-legible CI, agent-accessible observability, the dissemination layer: each of those was itself built with the new capacity. The compounding is the point.
“We didn’t obviously break anything” isn’t success on its own. It’s a necessary condition for the rest to matter, but it’s a low bar to clear. Real success looks like sustainable team load, real user outcomes from the shipping rate, complexity managed rather than accumulating off-dashboard, and cultural practices reproducing all the way down rather than concentrated at the top. No one is done. Including us.
If you came here looking for permission to skip the platform work and let agents carry the velocity on their own, that isn’t a thing that exists. The work is still the work. AI just changed which parts of it you can finally get to. And you can’t skip past observability and CI.
What we’re watching #
The bottoms-up process for “what we’re going to measure next” is still landing inside the org. One candidate I’d push for is the tie to slop reduction and the quality of delivered results, not just throughput; how much of the new capacity goes into reinvestment versus marginal features, and how that ratio moves over time. Sustained-use metrics for AI-built features. Rework rate over rolling six-month windows.
Other questions we haven’t fully answered yet. April’s 75% AI-line floor turned out to be an inflection, not a high-water mark; June hit 82.6%, so we still haven’t found the ceiling. Whether the autonomous workflow’s 8.4% share of June merges keeps compounding the way human delegation did after February, or plateaus, and whether it turns into a lines-of-code story the way human-driven AI work already has, rather than staying a merge-count one. What happens to PR cycle time once auto-review runs at scale. Whether the platform substrate we’ve built is enough to safely scale auto-approval, and on what timeline, and whether that answer changes once part of the workload has no human in the commit loop at all. Whether we can keep incident severity trending down even as incident count keeps climbing with change volume. Holding the count flat looks like a losing game at this point; the target that matters is how cheap each individual failure is to contain, not whether failures happen at all.
We’ll have more in another six months. For now, this is our report-out, and our answer to Intercom’s 2x posts. They got there. We got there too. And so can you.
AI Influence Level disclosure #
This post: AIL-3.0 (substantial AI involvement, human steering on every load-bearing call). The data analysis, custom git-of-theseus extensions, commit-history trawling, calibration overrides, the merge-rate and incident charts, was AI-assisted. The July refresh, the May-June numbers and the autonomous-workflow analysis added above, was pulled with Claude Fable 5. The slide deck this post derives from was composed with AI assistance, and AI assistance was used to reformat the slide bullet points and speaker notes into essay form. The voice and tone polish was done in a separate Claude project tuned to my writing style, followed by a very extensive manual editing process where I further added or changed at least 20% of the words.
Strategic decisions, data interpretation, and judgment calls about what to keep and what to cut are mine. AI helped me move faster on a deadline; it didn’t supply the substance. The talk and this post are themselves an example of the same 2x story they describe: weeks of human work, AI-assisted, not 10x. The substance wouldn’t exist without me, and the level of polish wouldn’t exist without AI.
AIL framework: danielmiessler.com/blog/ai-influence-level-ail. Illustrations in the original talk: AIL-0, by bbghost.bsky.social. Art should be made by artists, not machines. Illustrations in the blog by our amazing design team.
Sources and context: Fin/Intercom 2x post; Fin/Intercom AI PR approval safety post; Honeycomb-Intercom case study; Emily Nakashima on AI-amplified engineering leadership. This post is adapted from the talk of the same name, delivered at Sydney Tech Leaders and LDX3 London in 2026.