{"slug": "the-hard-part-of-ai-root-cause-analysis-is-no-longer-the-model", "title": "The hard part of AI root cause analysis is no longer the model", "summary": "Coroot's engineering team found that the main challenge in AI root cause analysis is no longer the model's reasoning ability but the data harness feeding it. By isolating the model from data-fetching tools, they tested multiple LLMs on a real network delay scenario and discovered that even with full context, models often miss the root cause due to reasoning traps. This suggests that improving model reasoning, not just data pipelines, is the next frontier for AI-driven RCA.", "body_md": "[← All posts](/blog/)Engineering\n\n# The hard part of AI root cause analysis is no longer the model\n\nEvery few weeks someone tells me root cause analysis is a solved problem now: pipe your telemetry into an LLM, let it tell you what broke. I wish it were that easy. After years on this, I think \"can AI do RCA?\" is the wrong question, because doing RCA with an LLM is really two separate jobs, and the answer is different for each. They break in completely different ways, so it's worth pulling them apart.\n\nOne is reasoning: can the model take the data in front of it and connect the dots? A service slows down. Three facts are on the table at once: it's starved of CPU, the node's CPU is maxed, and a neighbor on that node is eating all of it. A model that reasons ties them into one story, a noisy neighbor. A weaker one reports three unrelated \"issues\", or grabs the loudest symptom and calls it the cause.\n\nThe other is the harness: everything around the model. What data you put in front of it, in what shape. Usually it means tool-calling, letting the model decide what to fetch and when to stop. Plenty goes wrong here, and none of it is about whether the model *could* reason. It just never got the right data.\n\nPeople mix these two up all the time. A model gives a bad answer, and everyone says LLMs can't do RCA. But usually the model just never got the data it needed. It's not that it couldn't reason, it never had a fair shot. And until you separate the two, you can't tell which one is the real problem.\n\n## Take the harness out of the picture[ #](#take-the-harness-out-of-the-picture)\n\nSo we did, on purpose. With [Coroot's AI RCA](/blog/anatomy-of-ai-powered-root-cause-analysis/), we don't hand the model tools and send it off to investigate. Instead, a deterministic pipeline does the heavy lifting: it correlates the signals and turns them into findings. The model gets those findings in one focused context, not the raw telemetry. No tools, no agent loop. Everything it needs to find the answer is already there.\n\nThat boils the whole thing down to reasoning. If the model has the full context and still misses the root cause, there's no one else to blame. Not the harness, not missing data. Just the model. And that's finally something you can measure cleanly.\n\nSo here's the experiment. Take a real failure where the context already holds the answer, hand that same dump to a bunch of models, and see which ones can distill it into the actual root cause. No fetching, no deciding what to look at. Just reasoning. And it's harder than it sounds: even with the answer sitting in the data, there are traps in there that can walk a model straight into the wrong conclusion.\n\n## The test[ #](#the-test)\n\nI picked one scenario: a network delay between the `catalog`\n\nservice and its Postgres database, `db-main`\n\n. The queries slow down, timeouts spread, and `front-end`\n\nstarts serving 502s. But nothing is actually wrong with the database or the service. The culprit is a Chaos Mesh `NetworkChaos`\n\nexperiment running in the cluster, injecting delay on the `catalog`\n\n↔`db-main`\n\npath, and it shows up right in the Kubernetes events. So the fix is to delete the experiment, and just as importantly, the schedule that would spin it right back up.\n\nYou can see the problem fan out: `front-end`\n\nshows errors and latency, but tracing the dependency chain leads through `catalog`\n\n, where the real signal lives, the TCP network and connection latency to `db-main`\n\n.\n\nNotice the map also flags things that aren't the root cause: latency on `kafka`\n\n, CPU on `catalog`\n\nand `db-main`\n\n, storage on the database. In a distributed system one problem bleeds into metrics everywhere, and some of them point the wrong way. Take the database. When the round-trip time between `catalog`\n\nand `db-main`\n\nwent up, the client started getting its query responses slower. But Postgres times a query from the first byte it receives to the last byte it sends back to the client, so that network delay gets counted as part of the query time. Read `pg_stat_statements`\n\nand it looks like the database suddenly got slower at the exact same queries.\n\nIt didn't. The extra time was on the wire, not inside Postgres. A naive read blames the database and moves on, which is exactly the trap. And it isn't a bug that the pre-processing surfaces a signal like this. Telling a real cause from its downstream effects is the model's job in the next step, and it takes real reasoning. More data here isn't a problem, as long as the model can reason well enough not to be fooled by it.\n\nAlongside those traps, the real evidence is all there. Coroot traced the propagation path, found the network RTT to `db-main`\n\ntracking the slowdown, and flagged the Kubernetes event showing the chaos experiment started right when things broke. The answer is right there in the prompt, nothing hidden, nothing behind a tool call.\n\nI then reused that exact same prompt against every model, asking each the same three things: what's the root cause, what's the cause-and-effect chain, and what's the immediate fix.\n\nNo score, just pass or fail. I couldn't turn this into an honest number (how many points is a half-right explanation worth?), so the bar is simple: did it pin the chaos experiment, explain the ripple out to `front-end`\n\n, and point at deleting both the experiment *and* its schedule?\n\n## What \"right\" looks like[ #](#what-right-looks-like)\n\nHere's the bar. Claude Opus 4.8 nails it.\n\nThe summary runs the chain in the right order: `front-end`\n\nerrors, back through `catalog`\n\n's slow Postgres queries, back to the `NetworkChaos`\n\ninjecting delay, with the correlated RTT and TCP spikes cited as proof.\n\nThe evidence holds up: RTT and connection time climbing with the slow queries, the slow `gorm.Query`\n\nspans, the Kubernetes events marking when the experiment started and recovered, plus a note that `db-main`\n\n's CPU profile is clean, so this isn't a database problem.\n\nAnd the fix is actually useful: delete the experiment now to get connectivity back, then delete its schedule so it can't fire again. That second step is the easy one to miss, and exactly what you'd want to know at 3am.\n\nNow let's see who else clears it.\n\n## Results[ #](#results)\n\nEleven models, same context, one shot each. Input is ~9,800 tokens, output ~1,000, so the cost column is the real per-incident price.\n\nResult key: ✅ correct root cause and a correct, complete fix. 🟡 found the root cause, but the fix was incomplete or it ignored the formatting instructions. ❌ missed the root cause, or the fix wouldn't actually work.\n\n| Model | Result | Cost | Screenshots | Notes |\n|---|---|---|---|---|\n| Claude Opus 4.8 | ✅ | $0.0743 |\n|\n\n[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/gpt-5-5-summary.png),[evidence](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/gpt-5-5-evidence.png)`front-end`\n\n502s, and caught both fix steps.[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/gemini-3-1-pro-summary.png),[evidence](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/gemini-3-1-pro-evidence.png)`db-main`\n\nRTT spike to the 502s, both fix steps.[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/deepseek-v4-pro-summary.png),[evidence](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/deepseek-v4-pro-evidence.png)[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/glm-5-2-summary.png),[evidence](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/glm-5-2-evidence.png)`Schedule`\n\nspawning the `NetworkChaos`\n\n, and prioritized deleting the schedule to stop recurrence.[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/nemotron-3-ultra-summary.png),[evidence](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/nemotron-3-ultra-evidence.png)`bash`\n\n, and the fix skips the schedule.[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/minimax-m3-summary.png),[evidence](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/minimax-m3-evidence.png)[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/gemma-4-31b-summary.png)[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/deepseek-v4-flash-summary.png),[evidence](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/deepseek-v4-flash-evidence.png)[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/qwen3-6-35b-summary.png)[summary](/assets/images/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/qwen3-coder-next-summary.png)## What this tells us[ #](#what-this-tells-us)\n\nIt helps to split the field three ways:\n\n**Closed frontier models**: Opus 4.8, GPT-5.5, Gemini 3.1 Pro. We don't know their size, but almost certainly huge.** Big open-weight models**, the kind you'd still run in the cloud: DeepSeek V4, GLM-5.2, MiniMax M3, Nemotron 3 Ultra.** Small open-weight models**you could actually self-host: Gemma 4 31B and the Qwens.\n\nNo surprises at the top: all three frontier models passed cleanly, and the big open models mostly kept up.\n\nThe surprise was at the small end. Among the models you could realistically run on your own hardware, size barely mattered, and the standout was the smallest of the bunch: Gemma 4 31B was the only self-hostable model to find the root cause at all. The bigger Qwen3.6 35B and Qwen3 Coder Next both missed it. And going huge was no guarantee either: Nemotron 3 Ultra, at 550 billion parameters, only made the middle tier. If you want something you can run yourself, Gemma was the one to beat.\n\nLook closer at who passed and who didn't, though, and the split comes down to two kinds of failure, only one of which is really about the model.\n\nThe ❌ cases are real reasoning failures. Qwen3 Coder Next saw slow queries and blamed the database, the obvious wrong answer, while the chaos event sat right there in the context. Qwen3.6 found the experiment but then deleted the wrong resource, handing back a command that wouldn't run. Both had everything they needed and still missed, and no harness can fix that.\n\nThe 🟡 cases are more hopeful. They all got the reasoning right and only stumbled on packaging: raw widget IDs leaking into the evidence, a stray `bash`\n\nin a code block, deleting the chaos object but not the schedule that recreates it. Those are formatting and instruction-following slips, the kind you fix on the harness side, not the model. For these models, the reasoning was fine.\n\nA word on cost, since the table shows a big spread. At first glance the gap between models looks huge. But the LLM isn't analyzing raw telemetry here, that work is already done before it's called. It just reads a small, ready-made context and reasons over it, one short call of around 10k tokens. At that size even the most expensive frontier model runs a few cents per incident, which makes reaching for a top model look perfectly reasonable.\n\nOne incident isn't a benchmark, but it lines up with what we keep seeing. The reasoning part of AI RCA is basically solved: frontier models nail it, and even small models you can self-host do a decent job, though they're still not as predictable. Finding the root cause, once the context is there, isn't really the open problem anymore.\n\nThe hard part is the harness. Telemetry keeps growing fast, and if you hand all that raw data to an LLM to sort through, it gets slow and expensive in a hurry. So the real work isn't finding a smarter model. It's preparing the right, compact context for the model before you call it.", "url": "https://wpnews.pro/news/the-hard-part-of-ai-root-cause-analysis-is-no-longer-the-model", "canonical_source": "https://coroot.com/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/", "published_at": "2026-07-10 13:42:06+00:00", "updated_at": "2026-07-10 14:06:08.856566+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools", "ai-research"], "entities": ["Coroot", "Chaos Mesh", "Postgres", "Kubernetes"], "alternates": {"html": "https://wpnews.pro/news/the-hard-part-of-ai-root-cause-analysis-is-no-longer-the-model", "markdown": "https://wpnews.pro/news/the-hard-part-of-ai-root-cause-analysis-is-no-longer-the-model.md", "text": "https://wpnews.pro/news/the-hard-part-of-ai-root-cause-analysis-is-no-longer-the-model.txt", "jsonld": "https://wpnews.pro/news/the-hard-part-of-ai-root-cause-analysis-is-no-longer-the-model.jsonld"}}