How I Triaged 8,400 Production Errors Into 11 Real Bugs With Claude Code

A developer built a pipeline using Claude Code to triage 8,400 production errors per week, clustering them into root causes and identifying 11 genuine bugs that had been hidden under noise. The system fetches structured error data and repo context, uses an agent to return a verdict per cluster, and auto-mutes non-issues with reasons. The developer shared the setup, prompt structure, and five mistakes made before the approach worked.

My error tracker had 8,400 events a week across ~340 distinct issues, and nobody on the team actually triaged them. I built a small pipeline that feeds structured error data plus repo context into Claude Code and forces it to return a verdict per issue — and it surfaced 11 genuine bugs that had been hiding under the noise for months. Here's the setup, the prompt structure, and the five things I got wrong before it worked. Every team I've worked on has the same dead ritual: someone sets up an error tracker in week one, everyone watches the dashboard for about a month, and then the volume outgrows human attention and the tab quietly stops getting opened. That's exactly where we were. Concretely: ResizeObserver loop limit exceeded browser warning, and network aborts from users closing tabs mid-requestThe math is what kills you. Say a careful engineer needs 4 minutes per issue to open the stack trace, find the corresponding code, and decide whether it's real. That's 22 hours to get through 340 issues once. Nobody has 22 hours, so the honest team behavior is: read the top 5, ignore the rest, and wait for a customer to complain about the rest. I wanted to know whether the tail was actually worth reading. Not "can AI fix my bugs" — just: can an agent do the boring 4-minute pass, 340 times, well enough that I only look at what survives? The whole thing is about 200 lines of Python 3.13 and one carefully-shaped prompt. Four stages: php flowchart LR A Error tracker API -- B Normalize to JSON B -- C Cluster by cause C -- D Agent verdict per cluster D -- E{Real bug?} E -- |yes| F Reproduce + failing test E -- |no| G Auto-mute with reason My first attempt was embarrassingly lazy — I pasted dashboard screenshots into a session and asked "what looks real here?" The answers were confident and useless, because a screenshot has the top frame of a stack trace and nothing else. Pull the real payload instead. Every error tracker has a REST API; mine gives me issues plus their latest event: python import json, os, urllib.request BASE = "https://errors.example-tracker.com/api/0" def fetch path: str : req = urllib.request.Request f"{BASE}{path}", headers={"Authorization": f"Bearer {os.environ 'TRACKER TOKEN' }"}, with urllib.request.urlopen req as res: return json.loads res.read def issue payload issue : event = fetch f"/issues/{issue 'id' }/events/latest/" frames = f for f in event "stacktrace" "frames" if f.get "in app" third-party frames are noise for triage return { "id": issue "id" , "title": issue "title" , "culprit": issue "culprit" , "count": issue "count" , "users affected": issue "userCount" , "first seen": issue "firstSeen" , "last seen": issue "lastSeen" , "release": event.get "release" , "frames": {"file": f "filename" , "line": f "lineno" , "fn": f "function" } for f in frames -6: deepest 6 in-app frames , "message": event.get "message", "" :2000 , } The in app filter matters more than anything else here. An unfiltered React stack trace is 40 frames of framework internals and 3 frames of my code, and every token spent on framework internals is a token not spent reasoning about my code. Error trackers group by a fingerprint — usually a hash of the exception type plus the top frame. That's a syntactic grouping, and it splits one bug into many issues constantly. In my dataset, one date-parsing bug appeared as 9 separate issues because it threw from 9 different call sites. So before triage, I have the agent do a cheap clustering pass over just the metadata no code reading yet : You will receive a JSON array of error issues. Group them by ROOT CAUSE, not by exception type or stack frame. Two issues share a root cause if fixing one line of code would plausibly resolve both. Different call sites into the same broken helper = same cause. Same exception type from unrelated modules = different causes. Return JSON: { "cause label": str, "issue ids": int , "why": str } If you are unsure, keep them separate. Over-splitting is cheap; over-merging hides bugs. 340 issues collapsed into 112 causes . That alone cut the expensive stage by two thirds. This is the step that turned the output from plausible to useful. For each cause cluster, I run Claude Code inside the repo so it can open the files named in the frames: claude -p "$ cat prompts/triage.md " \ --append-system-prompt "You are triaging one production error cluster. \ Read the referenced files before forming any opinion. Never guess at \ code you have not opened." \ < clusters/${cluster id}.json The difference is stark. Without repo access, on a TypeError: Cannot read properties of undefined reading 'timezone' , I get: This suggests the user object may be undefined. Consider adding a null check before accessing timezone . With repo access, on the same error: formatSlot at src/scheduling/slots.ts:88 reads user.prefs.timezone . prefs is populated by hydrateUser , which early-returns at line 41 when user.status === 'pending' . Invited-but-not-activated users therefore reach formatSlot with prefs undefined. The 6 events all carry a release after 2026-06-02, which is when the invite flow started rendering the schedule preview. One of those is a fortune cookie. The other is a bug report I can act on. My second big mistake: my first prompt asked "what's the fix for this error?" — and a model asked for a fix will always produce a fix. I got beautiful null checks for errors that were bots probing /wp-admin . The fix is a schema where "this isn't worth fixing" is a first-class, equally valid answer: VERDICT SCHEMA = { "type": "object", "required": "classification", "confidence", "evidence" , "properties": { "classification": { "enum": "real bug", our code is wrong "environment", browser quirk, extension, network abort "hostile traffic", scanners, bots, probing "already fixed", code path no longer exists on main "insufficient data", cannot decide from what was provided }, "confidence": {"enum": "high", "medium", "low" }, "evidence": { "type": "array", "items": {"type": "string"}, "description": "file:line references that justify the verdict", }, "user impact": {"type": "string"}, "suggested fix": {"type": "string", "null" }, }, } Two rules in the prompt do the heavy lifting: Every evidence entry must be a file:line you actually opened. If you cannot cite code you have read, the classification must be insufficient data . insufficient data is a correct and respected answer. A wrong real bug costs an engineer an hour; an honest insufficient data costs nothing. Out of 112 clusters: 61 hostile traffic or environment , 28 already fixed dead code paths still throwing from cached bundles , 12 insufficient data , 11 real bugs . For the 11 survivors, I didn't let the agent open a PR with a patch. It had to first write a test that fails on main for the reason described in the verdict: claude -p "Write a failing test that reproduces this verdict. \ Run it. It MUST fail with the error described in the verdict, \ for the described reason. Do not modify source code in this step. \ If you cannot make it fail for that reason, output REPRO FAILED \ and stop." 3 of the 11 came back REPRO FAILED. Two of those three were misdiagnoses that read completely convincingly — the reasoning was internally coherent and pointed at the wrong function. The reproduce-first gate is the only thing that caught them, and it's the single most valuable rule in this whole pipeline. The remaining 8 became PRs. 7 merged. Total agent cost for the run: about $14 . 1. Syntactic grouping is not causal grouping. Your error tracker groups by stack hash because that's what it can compute cheaply. One bug scattered across 9 issues looks like 9 low-priority nuisances; merged, it's a P1. The cheap metadata-only clustering pass was the highest leverage 20 lines in the project. 2. Volume is the worst possible priority signal. My noisiest issue had 3,100 events and zero user impact. My most expensive bug had 6 events and blocked every invited user from seeing their schedule. Sort by users affected × "is this on a path where someone spends money," never by raw count. 3. A stack trace without the source is a horoscope. Both are vague enough to feel true and unfalsifiable enough to be safe. Give the agent the repo, tell it to open the files, and require file:line citations — the quality jump isn't incremental, it's categorical. 4. If you don't make "no action" a valid output, you'll get action. This generalizes way past error triage. Any time you ask a model for a fix, a finding, or a recommendation, you have to build an equally respectable escape hatch or you're just measuring its willingness to produce output. Naming insufficient data and explicitly saying it was a good answer cut my false positives more than any prompt tuning did. 5. Reproduce before you fix — no exceptions. 3 of 11 verdicts evaporated at the reproduction step. Confident, well-cited, coherent, and wrong. The failing test is the only artifact in this whole chain that can't be argued with, and it's what makes the output trustworthy enough to stop reviewing every verdict by hand. Two directions I'm working on: The broader lesson I keep re-learning: agents are excellent at the boring 4-minute pass you'd never do 340 times, and mediocre at the judgment call you'd make in 10 seconds. Build for the first thing and gate the second. If your error tracker has a tail you've never read, there's a decent chance there's a real bug in it. Mine had 11. If you try this, I'd genuinely like to know your hit rate — drop it in the comments, especially if it's low, because that's the more interesting data point. Follow me here on Dev.to for more write-ups on building with AI coding agents, and if you want to try the pipeline yourself, Claude Code https://claude.com/claude-code plus your tracker's REST API is the entire dependency list. 🚀