# We Automated RCAs. The Real Value Was the Big Picture. > Source: > Published: 2026-08-14 05:00:00+00:00 We Made Bug Tickets Write Their Own Root Cause Analysis. The Value Wasn't the RCAs β€” It Was What They Became in Aggregate. πŸš€ **Every bugfix now ships with a structured, machine-written root cause analysis, authored at fix time by a coding agent. It took under two days to build. Reading the last hundred of them back is what changed how we plan quality work.** ## The Problem We wanted to answer a set of questions about our own codebase that we could not answer: - **Which areas break repeatedly?** Not "where are the bugs" β€” every team can guess that. Which modules keep producing *the same class of failure*, normalized for how much they change. - **Are we getting better?** We had no way to tell improvement from a quiet quarter. Bug count moves with traffic, headcount, and release cadence; it doesn't tell you whether the underlying quality changed. - **What kind of bugs are these, really?** Process failures and architectural failures need completely different responses, and we had no categorization to separate them. - **Where are our detection gaps?** Which gate *should* have caught each defect, and which missing gate would pay off most if we built it. - **What's actually worth refactoring?** We wanted a heatmap of problematic areas grounded in defect history, not in whoever complained loudest in planning. Every one of those is a query over a corpus. We had years of bug history and nothing queryable in it β€” because RCA lived in a free-text box, where it fills up with things like *"the dropdown click is triggering the click-outside handler."* True, useful to exactly one person for exactly one week, and impossible to aggregate. The blocker was never willingness. It was that writing a real RCA costs an hour of archaeology at the exact moment someone is trying to close a ticket. So the fix isn't asking people to try harder β€” it's removing the moment where trying is required, and making the output structured enough to answer the questions above. ## How we solved it - **Author at fix time, not at ticket close.** The agent runs when a dev *starts* a bugfix, while the spec, the git history, and the failing behavior are all still live context. By ticket close that context is gone, which is exactly why humans write one-liners. - **Seven fixed sections, not a text box.** Summary Β· Cause Chain Β· Category Β· Feature Β· Component Β· Detection Gap Β· Prevention. Structure *is* the feature β€” free text can't be aggregated by construction. - **Cause chains carry citations, not adjectives.** Each WHY-N links the PR and ticket that introduced that layer. This is the single design decision that mattered most: it turns an RCA from prose into a graph edge. - **Orthogonal categories.** People / Process / Technology / Architecture, primary + secondary, each with a written justification. Lifted from ODC (IBM Research, early '90s β€” still the reference model for defect classification). - **Detection Gap is its own field.** `Found by` / `Should have been caught by` . A different question from "why did it break," with a different remediation budget. - **Prevention splits Quick Win vs Systemic.** So the cheap fix doesn't quietly absorb the attention the architectural fix needed. - **Cost:** under two days of work. It’s a low-barrier experiment, not a massive platform project. ## What the field looks like Shape of a real one, sanitized: Look at WHY-4. A latent contract mismatch that had been sitting in the codebase for *years*, traced back to the exact PR that introduced it, with a named prevention. No engineer fixing a one-line schema bug was ever going to write that paragraph voluntarily β€” the effort/reward math doesn't work for a human. It works fine for an agent that already has the git history loaded. ## What we got immediately - **Depth nobody would have paid for by hand.** Chains reach past the symptom to a process or design gap, and the cited ancestor PR is often years old. That archaeology is the expensive part, and it's the part that got automated. - **Consistency across teams.** Two engineers on two services produce RCAs that parse with the same code. This is what makes downstream analytics possible. - **Detection Gap became a test backlog for free.** `Should have been caught by` , read across the corpus, is a list of the gates you don't have. Ours started saying "pre-merge contract checks" out loud, repeatedly, without anyone deciding that was the theme. - **Reviewers get the *** why* before they read the diff. The RCA lands on the ticket before the PR merges, shifting review focus from symptom to cause. - **Zero marginal cost per bug.** The reason RCA programs die is that they tax the person least able to afford it β€” someone mid-fix, under pressure. Remove the tax and the practice survives on its own. ## πŸ“ˆ What compounds over time This is the actual payoff, and none of it works without structure. Given a corpus of these, you can derive: - **Root-cause Pareto.** Which handful of causes account for most of your defects. Old technique, still undefeated, previously impossible on free text. - **Component hotspot heatmap.** Bug density per module normalized by churn β€” not "which file has bugs" but "which file has bugs *for its size and change rate*." The normalization is what makes it actionable instead of just pointing at your biggest file. - **Cross-domain blast radius.** Pair `(domain where the cause was introduced)` with `(domain where the symptom surfaced)` . A Sankey of those pairs shows you where your module boundaries are lying to you. - **Detection-gap trend.** Aggregated `Should have been caught by` *is* your quality-gate investment plan, ranked by how often each missing gate would have paid off. It replaces annual budget debates with an empirical list. - **Regression genealogy.** Follow WHY-N citations backwards to find PRs that keep reappearing as ancestors of later bugs. Change-failure attribution at the PR level, derived rather than guessed. - **Latency-to-detection distribution.** How long defects sit before anyone notices, per area. Turns "we should improve observability" into "these three areas leak to customers and the rest don't." - **Category drift.** A rising Architecture share is a different organizational signal than a rising Process share. Watching the *mix* move over quarters is far more informative than watching any total. - **Semantic recurrence detection.** Embed cause chains; flag when a new bug's chain closely matches a closed one. The most actionable signal in the system, because a match means the prevention didn't take. - **Prevention follow-through.** Did the `Systemic` item ever get filed? Ever get done? The gap between quick wins shipped and systemic fixes shipped is technical debt accrual, made measurable. ## Where it ended up This started as one team's experiment β€” a single field, one squad's bugs, a skill written in under two days. It didn't stay there. The field now lives on every ticket in Jira, org-wide, and that changed what the data *is*. A corpus scoped to one team answers questions about that team. A corpus spanning every domain answers questions no single team could ask: - **Cross-domain cause tracing.** Where a defect surfaces is often not where it was introduced. With per-team data, every cross-boundary cause looks like someone else's problem. - **Shared root causes.** Four teams hitting the same class of failure is a platform problem wearing four disguises β€” visible only when their RCAs share a taxonomy. - **Comparable dashboards.** A domain can see its own defect profile against the org baseline to recognize: "our detection gaps skew toward integration testing and nobody else's do." - **Org-level gate planning.** Detection gaps aggregated across all domains rank which missing gate pays off most company-wide. It spread for mundane reasons: cheap to build, no behavior change required, visibly useful before anyone had to be convinced. Nobody adopted a process β€” they adopted a field that was already filled in. ## The uncomfortable parts Machine-authored RCA has failure modes hand-written RCA doesn't. Naming them is part of the value: - **Category distribution is not yet trustworthy.** Ours skews heavily toward one category. Either the codebase really is like that, or the model has a prior toward the most intellectually satisfying explanation. Until checked against a human-labeled sample, treat category *trend* as signal and category *level* as a hypothesis. - **Citations need validating, not trusting.** Links are only good news if they resolve. Citation resolvability is a metric to compute, not an assumption to make. - **Chains are context-bounded.** A chain that stops early might mean the root cause was genuinely shallow, or that the model couldn’t see any further. The data does not distinguish those two. - **Prompt drift breaks trends.** Change the skill and you've changed the measurement instrument. Version it and log it so anomalies in the trend line stay explainable. - **Never aggregate by author.** Non-negotiable. The moment RCA data feeds performance reviews, the field becomes fiction and every downstream insight dies with it. ## Takeaways - **The bottleneck in RCA was never willingness β€” it was context cost.** Move authoring to the moment context is free and the practice sustains itself. - **Structure is the whole product.** Fixed sections are what let you ask questions of the corpus later. - **Citations turn documents into a graph.** Everything genuinely novel in the analytics list above depends on the cause chain carrying links rather than adjectives. - **Build the data-quality view before the insight views.** Otherwise you get beautiful charts sitting on unverified model output. **Individual RCAs are the byproduct. The corpus is the asset.** P.S. Drop a comment or message me on LinkedIn if you'd like the exact prompt/skill definition we use to generate these structured RCA fields. Happy to share it! # SoftwareEngineering #SoftwareArchitecture #AIEngineering #DeveloperProductivity #DevOps