My error tracker had 8,400 events a week across ~340 distinct issues, and nobody on the team actually triaged them. I built a small pipeline that feeds structured error data plus repo context into Claude Code and forces it to return a verdict per issue — and it surfaced 11 genuine bugs that had been hiding under the noise for months. Here's the setup, the prompt structure, and the five things I got wrong before it worked.
Every team I've worked on has the same dead ritual: someone sets up an error tracker in week one, everyone watches the dashboard for about a month, and then the volume outgrows human attention and the tab quietly stops getting opened.
That's exactly where we were. Concretely:
ResizeObserver loop limit exceeded
browser warning, and network aborts from users closing tabs mid-requestThe math is what kills you. Say a careful engineer needs 4 minutes per issue to open the stack trace, find the corresponding code, and decide whether it's real. That's 22 hours to get through 340 issues once. Nobody has 22 hours, so the honest team behavior is: read the top 5, ignore the rest, and wait for a customer to complain about the rest.
I wanted to know whether the tail was actually worth reading. Not "can AI fix my bugs" — just: can an agent do the boring 4-minute pass, 340 times, well enough that I only look at what survives?
The whole thing is about 200 lines of Python 3.13 and one carefully-shaped prompt. Four stages:
flowchart LR
A[Error tracker API] --> B[Normalize to JSON]
B --> C[Cluster by cause]
C --> D[Agent verdict per cluster]
D --> E{Real bug?}
E -->|yes| F[Reproduce + failing test]
E -->|no| G[Auto-mute with reason]
My first attempt was embarrassingly lazy — I pasted dashboard screenshots into a session and asked "what looks real here?" The answers were confident and useless, because a screenshot has the top frame of a stack trace and nothing else.
Pull the real payload instead. Every error tracker has a REST API; mine gives me issues plus their latest event:
import json, os, urllib.request
BASE = "https://errors.example-tracker.com/api/0"
def fetch(path: str):
req = urllib.request.Request(
f"{BASE}{path}",
headers={"Authorization": f"Bearer {os.environ['TRACKER_TOKEN']}"},
)
with urllib.request.urlopen(req) as res:
return json.loads(res.read())
def issue_payload(issue):
event = fetch(f"/issues/{issue['id']}/events/latest/")
frames = [
f for f in event["stacktrace"]["frames"]
if f.get("in_app") # third-party frames are noise for triage
]
return {
"id": issue["id"],
"title": issue["title"],
"culprit": issue["culprit"],
"count": issue["count"],
"users_affected": issue["userCount"],
"first_seen": issue["firstSeen"],
"last_seen": issue["lastSeen"],
"release": event.get("release"),
"frames": [
{"file": f["filename"], "line": f["lineno"], "fn": f["function"]}
for f in frames[-6:] # deepest 6 in-app frames
],
"message": event.get("message", "")[:2000],
}
The in_app
filter matters more than anything else here. An unfiltered React stack trace is 40 frames of framework internals and 3 frames of my code, and every token spent on framework internals is a token not spent reasoning about my code.
Error trackers group by a fingerprint — usually a hash of the exception type plus the top frame. That's a syntactic grouping, and it splits one bug into many issues constantly. In my dataset, one date-parsing bug appeared as 9 separate issues because it threw from 9 different call sites.
So before triage, I have the agent do a cheap clustering pass over just the metadata (no code reading yet):
You will receive a JSON array of error issues.
Group them by ROOT CAUSE, not by exception type or stack frame.
Two issues share a root cause if fixing one line of code would
plausibly resolve both. Different call sites into the same broken
helper = same cause. Same exception type from unrelated modules
= different causes.
Return JSON: [{ "cause_label": str, "issue_ids": [int], "why": str }]
If you are unsure, keep them separate. Over-splitting is cheap;
over-merging hides bugs.
340 issues collapsed into 112 causes. That alone cut the expensive stage by two thirds.
This is the step that turned the output from plausible to useful. For each cause cluster, I run Claude Code inside the repo so it can open the files named in the frames:
claude -p "$(cat prompts/triage.md)" \
--append-system-prompt "You are triaging one production error cluster. \
Read the referenced files before forming any opinion. Never guess at \
code you have not opened." \
< clusters/${cluster_id}.json
The difference is stark. Without repo access, on a TypeError: Cannot read properties of undefined (reading 'timezone')
, I get:
This suggests the user object may be undefined. Consider adding a null check before accessing
timezone
.
With repo access, on the same error:
formatSlot()
atsrc/scheduling/slots.ts:88
readsuser.prefs.timezone
.prefs
is populated byhydrateUser()
, which early-returns at line 41 whenuser.status === 'pending'
. Invited-but-not-activated users therefore reachformatSlot()
withprefs
undefined. The 6 events all carry arelease
after 2026-06-02, which is when the invite flow started rendering the schedule preview.
One of those is a fortune cookie. The other is a bug report I can act on.
My second big mistake: my first prompt asked "what's the fix for this error?" — and a model asked for a fix will always produce a fix. I got beautiful null checks for errors that were bots probing /wp-admin
.
The fix is a schema where "this isn't worth fixing" is a first-class, equally valid answer:
VERDICT_SCHEMA = {
"type": "object",
"required": ["classification", "confidence", "evidence"],
"properties": {
"classification": {
"enum": [
"real_bug", # our code is wrong
"environment", # browser quirk, extension, network abort
"hostile_traffic", # scanners, bots, probing
"already_fixed", # code path no longer exists on main
"insufficient_data", # cannot decide from what was provided
]
},
"confidence": {"enum": ["high", "medium", "low"]},
"evidence": {
"type": "array",
"items": {"type": "string"},
"description": "file:line references that justify the verdict",
},
"user_impact": {"type": "string"},
"suggested_fix": {"type": ["string", "null"]},
},
}
Two rules in the prompt do the heavy lifting:
Every
evidence
entry must be afile:line
you actually opened. If you cannot cite code you have read, the classification must beinsufficient_data
.
insufficient_data
is a correct and respected answer. A wrongreal_bug
costs an engineer an hour; an honestinsufficient_data
costs nothing.
Out of 112 clusters: 61 hostile traffic or environment, 28 already fixed (dead code paths still throwing from cached bundles), 12 insufficient data, 11 real bugs.
For the 11 survivors, I didn't let the agent open a PR with a patch. It had to first write a test that fails on main
for the reason described in the verdict:
claude -p "Write a failing test that reproduces this verdict. \
Run it. It MUST fail with the error described in the verdict, \
for the described reason. Do not modify source code in this step. \
If you cannot make it fail for that reason, output REPRO_FAILED \
and stop."
3 of the 11 came back REPRO_FAILED. Two of those three were misdiagnoses that read completely convincingly — the reasoning was internally coherent and pointed at the wrong function. The reproduce-first gate is the only thing that caught them, and it's the single most valuable rule in this whole pipeline.
The remaining 8 became PRs. 7 merged. Total agent cost for the run: about $14.
1. Syntactic grouping is not causal grouping. Your error tracker groups by stack hash because that's what it can compute cheaply. One bug scattered across 9 issues looks like 9 low-priority nuisances; merged, it's a P1. The cheap metadata-only clustering pass was the highest leverage 20 lines in the project.
2. Volume is the worst possible priority signal. My noisiest issue had 3,100 events and zero user impact. My most expensive bug had 6 events and blocked every invited user from seeing their schedule. Sort by users_affected
× "is this on a path where someone spends money," never by raw count.
3. A stack trace without the source is a horoscope. Both are vague enough to feel true and unfalsifiable enough to be safe. Give the agent the repo, tell it to open the files, and require file:line citations — the quality jump isn't incremental, it's categorical.
4. If you don't make "no action" a valid output, you'll get action. This generalizes way past error triage. Any time you ask a model for a fix, a finding, or a recommendation, you have to build an equally respectable escape hatch or you're just measuring its willingness to produce output. Naming insufficient_data
and explicitly saying it was a good answer cut my false positives more than any prompt tuning did.
5. Reproduce before you fix — no exceptions. 3 of 11 verdicts evaporated at the reproduction step. Confident, well-cited, coherent, and wrong. The failing test is the only artifact in this whole chain that can't be argued with, and it's what makes the output trustworthy enough to stop reviewing every verdict by hand.
Two directions I'm working on:
The broader lesson I keep re-learning: agents are excellent at the boring 4-minute pass you'd never do 340 times, and mediocre at the judgment call you'd make in 10 seconds. Build for the first thing and gate the second.
If your error tracker has a tail you've never read, there's a decent chance there's a real bug in it. Mine had 11.
If you try this, I'd genuinely like to know your hit rate — drop it in the comments, especially if it's low, because that's the more interesting data point.
Follow me here on Dev.to for more write-ups on building with AI coding agents, and if you want to try the pipeline yourself, Claude Code plus your tracker's REST API is the entire dependency list. 🚀