{"slug": "how-i-triaged-8400-production-errors-into-11-real-bugs-with-claude-code", "title": "How I Triaged 8,400 Production Errors Into 11 Real Bugs With Claude Code", "summary": "A developer built a pipeline using Claude Code to triage 8,400 production errors per week, clustering them into root causes and identifying 11 genuine bugs that had been hidden under noise. The system fetches structured error data and repo context, uses an agent to return a verdict per cluster, and auto-mutes non-issues with reasons. The developer shared the setup, prompt structure, and five mistakes made before the approach worked.", "body_md": "My error tracker had 8,400 events a week across ~340 distinct issues, and nobody on the team actually triaged them. I built a small pipeline that feeds structured error data plus repo context into Claude Code and forces it to return a verdict per issue — and it surfaced 11 genuine bugs that had been hiding under the noise for months. Here's the setup, the prompt structure, and the five things I got wrong before it worked.\n\nEvery team I've worked on has the same dead ritual: someone sets up an error tracker in week one, everyone watches the dashboard for about a month, and then the volume outgrows human attention and the tab quietly stops getting opened.\n\nThat's exactly where we were. Concretely:\n\n`ResizeObserver loop limit exceeded`\n\nbrowser warning, and network aborts from users closing tabs mid-requestThe math is what kills you. Say a careful engineer needs 4 minutes per issue to open the stack trace, find the corresponding code, and decide whether it's real. That's **22 hours** to get through 340 issues once. Nobody has 22 hours, so the honest team behavior is: read the top 5, ignore the rest, and wait for a customer to complain about the rest.\n\nI wanted to know whether the tail was actually worth reading. Not \"can AI fix my bugs\" — just: **can an agent do the boring 4-minute pass, 340 times, well enough that I only look at what survives?**\n\nThe whole thing is about 200 lines of Python 3.13 and one carefully-shaped prompt. Four stages:\n\n``` php\nflowchart LR\n    A[Error tracker API] --> B[Normalize to JSON]\n    B --> C[Cluster by cause]\n    C --> D[Agent verdict per cluster]\n    D --> E{Real bug?}\n    E -->|yes| F[Reproduce + failing test]\n    E -->|no| G[Auto-mute with reason]\n```\n\nMy first attempt was embarrassingly lazy — I pasted dashboard screenshots into a session and asked \"what looks real here?\" The answers were confident and useless, because a screenshot has the top frame of a stack trace and nothing else.\n\nPull the real payload instead. Every error tracker has a REST API; mine gives me issues plus their latest event:\n\n``` python\nimport json, os, urllib.request\n\nBASE = \"https://errors.example-tracker.com/api/0\"\n\ndef fetch(path: str):\n    req = urllib.request.Request(\n        f\"{BASE}{path}\",\n        headers={\"Authorization\": f\"Bearer {os.environ['TRACKER_TOKEN']}\"},\n    )\n    with urllib.request.urlopen(req) as res:\n        return json.loads(res.read())\n\ndef issue_payload(issue):\n    event = fetch(f\"/issues/{issue['id']}/events/latest/\")\n    frames = [\n        f for f in event[\"stacktrace\"][\"frames\"]\n        if f.get(\"in_app\")  # third-party frames are noise for triage\n    ]\n    return {\n        \"id\": issue[\"id\"],\n        \"title\": issue[\"title\"],\n        \"culprit\": issue[\"culprit\"],\n        \"count\": issue[\"count\"],\n        \"users_affected\": issue[\"userCount\"],\n        \"first_seen\": issue[\"firstSeen\"],\n        \"last_seen\": issue[\"lastSeen\"],\n        \"release\": event.get(\"release\"),\n        \"frames\": [\n            {\"file\": f[\"filename\"], \"line\": f[\"lineno\"], \"fn\": f[\"function\"]}\n            for f in frames[-6:]  # deepest 6 in-app frames\n        ],\n        \"message\": event.get(\"message\", \"\")[:2000],\n    }\n```\n\nThe `in_app`\n\nfilter matters more than anything else here. An unfiltered React stack trace is 40 frames of framework internals and 3 frames of my code, and every token spent on framework internals is a token not spent reasoning about my code.\n\nError trackers group by a *fingerprint* — usually a hash of the exception type plus the top frame. That's a syntactic grouping, and it splits one bug into many issues constantly. In my dataset, one date-parsing bug appeared as **9 separate issues** because it threw from 9 different call sites.\n\nSo before triage, I have the agent do a cheap clustering pass over just the metadata (no code reading yet):\n\n```\nYou will receive a JSON array of error issues.\nGroup them by ROOT CAUSE, not by exception type or stack frame.\n\nTwo issues share a root cause if fixing one line of code would\nplausibly resolve both. Different call sites into the same broken\nhelper = same cause. Same exception type from unrelated modules\n= different causes.\n\nReturn JSON: [{ \"cause_label\": str, \"issue_ids\": [int], \"why\": str }]\nIf you are unsure, keep them separate. Over-splitting is cheap;\nover-merging hides bugs.\n```\n\n340 issues collapsed into **112 causes**. That alone cut the expensive stage by two thirds.\n\nThis is the step that turned the output from plausible to useful. For each cause cluster, I run Claude Code *inside the repo* so it can open the files named in the frames:\n\n```\nclaude -p \"$(cat prompts/triage.md)\" \\\n  --append-system-prompt \"You are triaging one production error cluster. \\\nRead the referenced files before forming any opinion. Never guess at \\\ncode you have not opened.\" \\\n  < clusters/${cluster_id}.json\n```\n\nThe difference is stark. Without repo access, on a `TypeError: Cannot read properties of undefined (reading 'timezone')`\n\n, I get:\n\nThis suggests the user object may be undefined. Consider adding a null check before accessing\n\n`timezone`\n\n.\n\nWith repo access, on the same error:\n\n`formatSlot()`\n\nat`src/scheduling/slots.ts:88`\n\nreads`user.prefs.timezone`\n\n.`prefs`\n\nis populated by`hydrateUser()`\n\n, which early-returns at line 41 when`user.status === 'pending'`\n\n. Invited-but-not-activated users therefore reach`formatSlot()`\n\nwith`prefs`\n\nundefined. The 6 events all carry a`release`\n\nafter 2026-06-02, which is when the invite flow started rendering the schedule preview.\n\nOne of those is a fortune cookie. The other is a bug report I can act on.\n\nMy second big mistake: my first prompt asked \"what's the fix for this error?\" — and a model asked for a fix will always produce a fix. I got beautiful null checks for errors that were bots probing `/wp-admin`\n\n.\n\nThe fix is a schema where \"this isn't worth fixing\" is a first-class, equally valid answer:\n\n```\nVERDICT_SCHEMA = {\n    \"type\": \"object\",\n    \"required\": [\"classification\", \"confidence\", \"evidence\"],\n    \"properties\": {\n        \"classification\": {\n            \"enum\": [\n                \"real_bug\",           # our code is wrong\n                \"environment\",        # browser quirk, extension, network abort\n                \"hostile_traffic\",    # scanners, bots, probing\n                \"already_fixed\",      # code path no longer exists on main\n                \"insufficient_data\",  # cannot decide from what was provided\n            ]\n        },\n        \"confidence\": {\"enum\": [\"high\", \"medium\", \"low\"]},\n        \"evidence\": {\n            \"type\": \"array\",\n            \"items\": {\"type\": \"string\"},\n            \"description\": \"file:line references that justify the verdict\",\n        },\n        \"user_impact\": {\"type\": \"string\"},\n        \"suggested_fix\": {\"type\": [\"string\", \"null\"]},\n    },\n}\n```\n\nTwo rules in the prompt do the heavy lifting:\n\nEvery\n\n`evidence`\n\nentry must be a`file:line`\n\nyou actually opened. If you cannot cite code you have read, the classification must be`insufficient_data`\n\n.\n\n`insufficient_data`\n\nis a correct and respected answer. A wrong`real_bug`\n\ncosts an engineer an hour; an honest`insufficient_data`\n\ncosts nothing.\n\nOut of 112 clusters: **61 hostile traffic or environment**, **28 already fixed** (dead code paths still throwing from cached bundles), **12 insufficient data**, **11 real bugs**.\n\nFor the 11 survivors, I didn't let the agent open a PR with a patch. It had to first write a test that fails on `main`\n\nfor the reason described in the verdict:\n\n```\nclaude -p \"Write a failing test that reproduces this verdict. \\\nRun it. It MUST fail with the error described in the verdict, \\\nfor the described reason. Do not modify source code in this step. \\\nIf you cannot make it fail for that reason, output REPRO_FAILED \\\nand stop.\"\n```\n\n**3 of the 11 came back REPRO_FAILED.** Two of those three were misdiagnoses that read completely convincingly — the reasoning was internally coherent and pointed at the wrong function. The reproduce-first gate is the only thing that caught them, and it's the single most valuable rule in this whole pipeline.\n\nThe remaining 8 became PRs. 7 merged. Total agent cost for the run: about **$14**.\n\n**1. Syntactic grouping is not causal grouping.** Your error tracker groups by stack hash because that's what it can compute cheaply. One bug scattered across 9 issues looks like 9 low-priority nuisances; merged, it's a P1. The cheap metadata-only clustering pass was the highest leverage 20 lines in the project.\n\n**2. Volume is the worst possible priority signal.** My noisiest issue had 3,100 events and zero user impact. My most expensive bug had 6 events and blocked every invited user from seeing their schedule. Sort by `users_affected`\n\n× \"is this on a path where someone spends money,\" never by raw count.\n\n**3. A stack trace without the source is a horoscope.** Both are vague enough to feel true and unfalsifiable enough to be safe. Give the agent the repo, tell it to open the files, and require file:line citations — the quality jump isn't incremental, it's categorical.\n\n**4. If you don't make \"no action\" a valid output, you'll get action.** This generalizes way past error triage. Any time you ask a model for a fix, a finding, or a recommendation, you have to build an equally respectable escape hatch or you're just measuring its willingness to produce output. Naming `insufficient_data`\n\nand explicitly saying it was a good answer cut my false positives more than any prompt tuning did.\n\n**5. Reproduce before you fix — no exceptions.** 3 of 11 verdicts evaporated at the reproduction step. Confident, well-cited, coherent, and wrong. The failing test is the only artifact in this whole chain that can't be argued with, and it's what makes the output trustworthy enough to stop reviewing every verdict by hand.\n\nTwo directions I'm working on:\n\nThe broader lesson I keep re-learning: agents are excellent at the boring 4-minute pass you'd never do 340 times, and mediocre at the judgment call you'd make in 10 seconds. Build for the first thing and gate the second.\n\nIf your error tracker has a tail you've never read, there's a decent chance there's a real bug in it. Mine had 11.\n\nIf you try this, I'd genuinely like to know your hit rate — drop it in the comments, especially if it's low, because that's the more interesting data point.\n\n**Follow me here on Dev.to** for more write-ups on building with AI coding agents, and if you want to try the pipeline yourself, [Claude Code](https://claude.com/claude-code) plus your tracker's REST API is the entire dependency list. 🚀", "url": "https://wpnews.pro/news/how-i-triaged-8400-production-errors-into-11-real-bugs-with-claude-code", "canonical_source": "https://dev.to/yureki_lab/how-i-triaged-8400-production-errors-into-11-real-bugs-with-claude-code-484f", "published_at": "2026-08-27 14:32:38+00:00", "updated_at": "2026-08-27 14:48:25.016108+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-agents"], "entities": ["Claude Code"], "alternates": {"html": "https://wpnews.pro/news/how-i-triaged-8400-production-errors-into-11-real-bugs-with-claude-code", "markdown": "https://wpnews.pro/news/how-i-triaged-8400-production-errors-into-11-real-bugs-with-claude-code.md", "text": "https://wpnews.pro/news/how-i-triaged-8400-production-errors-into-11-real-bugs-with-claude-code.txt", "jsonld": "https://wpnews.pro/news/how-i-triaged-8400-production-errors-into-11-real-bugs-with-claude-code.jsonld"}}