{"slug": "17-prs-a-day-one-qa-how-we-automated-e2e-failure-triage", "title": "17 PRs a Day, One QA: How We Automated E2E Failure Triage", "summary": "At pdf.net, a single QA engineer automated end-to-end test failure triage using a GitHub Action and Claude API, handling 17 daily pull requests from 18 AI-augmented developers. The system analyzes failed tests, diffs, and logs to classify failures as PR-related, flaky, or infrastructure issues, then creates tickets and Slack notifications accordingly. The entire solution costs less per month than one hour of engineering time.", "body_md": "Picture this: 18 AI-augmented developers merging an average of 17 pull requests every working day — 33 on the peak day. And at the time, exactly one QA engineer watching over all of it. The fact that things didn't fall apart daily looks like a miracle. Spoiler: it's not a miracle, it's e2e tests on staging — plus a bot that triages their failures so humans don't have to.\n\nI'm that QA engineer. Here's how we at [pdf.net](https://pdf.net/) got a GitHub Action + Claude to triage every red run: the architecture, the code, and the rakes we stepped on — the bot blaming innocent people, losing tickets, and staying silent when silence was the worst option. Headline number up front: the whole thing costs less per month than one hour of engineering time.\n\nOur team is AI-first by design: developers ship with AI assistants, and instead of scaling manual testing along with headcount, we bet on automation. Autotests aren't a safety net \"just in case\" — they're the primary trigger that we're doing something wrong.\n\nWe release once a day, and originally the release was where everything got verified: fifteen-plus PRs roll up, tests run, and if something's red, the one QA engineer investigates. A single release could contain several broken PRs at once, each to be identified among 17+ suspects while the release waits. Mornings turned into detective work.\n\nSo we shifted the loop left. Quick infra note: every PR gets a Vercel preview (a dev environment — running the full e2e scope there makes no sense: it's slow, and half the features sit behind flags that don't match production), while staging is effectively pre-prod: every merge lands there, and code ships to production from there. E2e now runs on every merge to staging — the suspect range becomes one fresh merge instead of a day's batch, and the release turns into a final \"works assembled\" check plus manual testing that automation can't cover.\n\nThe price: red runs now happen potentially after every one of those 17 merges, each needs a quick investigation, and until it's done the release is in question. Hiring a person for \"look at a failed run and decide who to hand it to\" is overkill — that's not a role, it's a distraction. The reasoning wrote itself: if our tests are automation, triaging their failures should be automation too.\n\nHandily, the failure sources are a closed list — a PR regression in the main repo, the neighboring backend repo (its deploys trigger our e2e too), external services, CI infrastructure, or a flake. Manual triage was always the same mechanical chain over structured inputs: failed tests, diffs, logs → a three-way call. Exactly the kind of task LLMs are good at.\n\nThe triage is a reusable composite GitHub Action, invoked after every e2e run — red and green. Around it: Allure TestOps (test results), Linear (tickets), Slack (notifications), Claude API (the brain).\n\n```\n                      e2e run on staging\n                             │\n               red ──────────┴────────── green\n                │                          │\n           AI verdict                 Recovery: close tickets,\n                │                     rebase-nudge PR authors\n     ┌──────────┴───────────────┐\nlikely-pr-related      likely-not-pr-related\n     │                 / insufficient-data\nLinear ticket                   │\n+ Slack thread         Slack heads-up + ONE auto-rerun\n                                │\n                     ┌──────────┴──────────┐\n                rerun green            rerun red\n                     │                      │\n              it was a flake —      escalation: ticket\n              note it, move on      + @oncall mention\n```\n\nOn a red run the bot collects failed tests with steps from Allure, job logs, the deploy commit range, and the diffs of the PRs that went in — and asks Claude (Opus 4.8, Sonnet 5 fallback) for a verdict plus a fix hypothesis:\n\n``` js\nconst verdict = await askClaude({ failedTests, jobLogs, deployRange, prDiffs })\n// => 'likely-pr-related' | 'likely-not-pr-related' | 'insufficient-data'\n\nif (attempt === 1 && verdict !== 'likely-pr-related') {\n  await slack.post(headsUp(verdict))  // \"doesn't look PR-related — rerunning\"\n  return requestRerun()               // exactly one auto-rerun\n}\n\n// PR-related, or a red rerun → Linear ticket + Slack post,\n// on a red rerun — with an on-call mention\nawait createLinearIssue({ verdict, hypothesis, failedTests })\n```\n\nThe key decision: the verdict picks a route, it doesn't just write text. PR-related → ticket with a hypothesis and a Slack thread. Anything else → no ticket yet: a heads-up and one auto-rerun. Green rerun — it was a flake, no noise. Red rerun — ticket plus an on-call mention.\n\n```\n🔴 e2e failed on staging — 3 tests\nVerdict: likely-pr-related\nHypothesis: regression in PR #1234 — the file upload handler\nchanged, tests fail on the \"attach document\" step\nTicket: QA-482 · Staging: 🔴 red — release on hold\n```\n\nTickets land in one Linear queue, with labels telling frontend deploys from backend ones. One manual habit pays for it all: a confirmed real bug gets a `bug`\n\nlabel — so \"how many bugs did autotests catch, and how much was noise\" is a plain queue filter.\n\nNoise control, four mechanisms: dedup (repeat failures match open auto-tickets instead of spawning new ones), auto-close (green run closes the ticket, posts recovery with outage duration, and nudges open PRs to rebase), two channels (ticket-backed posts go to the important channel, rerun chatter stays in the working one), and manual muting (tickets closed without a fix go to Canceled, not Done — recovery leaves them alone).\n\nNeat on paper. Now the four rakes.\n\nRecovery writes to three places: a Slack post, the Done transition, and a `recovered-at: <sha>`\n\nmarker comment so the next green run skips the ticket. The first version wrote the marker first. Slack post succeeds, Done transition fails on a network hiccup — the marker is already there, and every following run says \"already processed\". Forever.\n\nThe fix is a classic idempotent sequence — each step guarded by its own marker, \"fully processed\" written last:\n\n``` js\nif (!hasSlackPostedComment(issue.comments)) {\n  const resp = await slack.postMessage(buildRecovery({ issue, attribution, cause }))\n  await linear.addComment(issue.id, `slack-recovery-posted:${resp.ts}`)\n}\nif (issue.stateType !== 'completed') {\n  await linear.transitionToDone(issue.id)\n}\nif (!hasRecoveredAtComment(issue.comments)) {\n  await linear.addComment(issue.id, `recovered-at:${sha7}`)  // strictly last\n}\n```\n\nFail in the middle — the next run completes the rest without duplicating anything. Bonus: state lives in the ticket's own comments — no external database, human-readable, garbage-collected with the ticket.\n\nRecovery posts ended with \"thanks for the fix, @author\" — author computed from the attached fix-PR, or, as a fallback, from the commits in the green run's deploy range. But when a ticket closed because an external service came back to life, the fallback still found an \"author\": some random person whose innocent PR rode the same deploy. Publicly thanked for a fix that never existed. A couple of those and trust in the bot is gone.\n\nThe fix is a recovery-cause classifier — a pure function, no AI:\n\n``` js\ntype RecoveryCause = 'code-fix' | 'external' | 'ambiguous'\n\nconst classifyFixCause = ({ deployRange, originalVerdict }): RecoveryCause => {\n  const codeFiles = deployRange.files.filter(f => isCodeFile(f.filename))\n  if (codeFiles.length === 0) return 'external'       // only docs/lockfiles deployed\n  if (originalVerdict === 'likely-not-pr-related') return 'ambiguous'\n  return 'code-fix'\n}\n```\n\nAttribution now fires only for `code-fix`\n\n; `external`\n\nand `ambiguous`\n\nuse a no-names template. `ambiguous`\n\ndoubles as telemetry: how often the model's verdict diverged from the actual cause.\n\nThe first dispatcher was `if (hasFailures) runAnalysis() else runRecovery()`\n\n. Breaks on a mixed run: old ticket's tests are fixed, but something new failed — the run is red, recovery never fires, the fixed ticket hangs open. The fix: both paths run on every invocation, recovery first, each no-oping when idle:\n\n```\nawait runRecovery(ctx)   // close everything that recovered\nawait runAnalysis(ctx)   // triage everything that failed\n```\n\nImmediate blast radius: our \"rerun helped\" signal was just `runAttempt >= 2`\n\n, which was safe while recovery ran only on green runs. Once it ran always, the bot cheerfully posted \"✅ recovered — auto-rerun helped\" on a red rerun. Hence the explicit guard: `runAttempt >= 2 && failureSet.size === 0`\n\n. Every check that implicitly relied on the old call order has to become explicit.\n\nOur e2e lives in the frontend repo and walks the full user scenario through the UI — so it catches backend breakage too, and the backend's staging deploy triggers the same workflow via `workflow_dispatch`\n\n. The GitHub Actions quirk you learn the hard way: a dispatched run has no push context. `GITHUB_EVENT_BEFORE`\n\nis empty, there is no deploy range — and the bot died silently: no ticket, no post. Red staging, and silence. The worst failure mode for a triage system: everyone assumes quiet bot = all under control.\n\nThe fix — the deploy range became an explicit input:\n\n```\n- uses: our-org/staging-triage@v1.6\n  with:\n    deploy-repo: our-org/backend        # whose deploy triggered the e2e\n    deploy-before-sha: ${{ inputs.before-sha }}\n    deploy-head-sha: ${{ inputs.head-sha }}\n```\n\nEverything downstream — diffs for the verdict, authors for attribution and mentions — now comes from the repo that actually deployed. One triage, both repos.\n\nYou can't unit-test a system whose behavior lives across CI, Linear, Slack, and Allure. Two things worked: a smoke PR walked commit by commit through the whole state series — first failure (ticket created?), repeat failure (dedup quiet?), mixed run (both paths fire?), green run (closed + recovery post?) — real runs, real APIs, no mocks. And a kill switch: manual `workflow_dispatch`\n\nis suppressed unless `STAGING_TRIAGE_SIMULATED=true`\n\n, so you can poke the workflow without spamming Linear.\n\nThe most dangerous failure mode of an LLM classifier isn't a wrong verdict — it's confident text about nothing: \"possibly a flake\", \"check the environment\", \"probably a cold start\". So the prompt is built on bans, not wishes:\n\n`likely-pr-related`\n\n→ file:line and what to change; `likely-not-pr-related`\n\n→ facts with quotes, what's missing, where a human should look.`insufficient-data`\n\nis declared a correct, useful answer — otherwise the model invents hypotheses for the sake of pretty text.What still happens: hypotheses right about the area, wrong in the details — the first critical bug the bot caught, it pointed at the right place, but a human wrote the actual fix, just faster. That's fine. The bot doesn't replace an engineer — it replaces the first 15–30 minutes of their work.\n\nHonest part first: we haven't collected rigorous metrics, and two months is a small sample. But releases got noticeably faster — bugs are localized when the code lands on staging, not on release morning, and the 17-suspects detective game is gone. The `bug`\n\nlabel groundwork means in six months there'll be real numbers.\n\nCost: one model call per red run (recovery needs no AI), typically 30–80K input tokens — $0.3–0.5 per red run at Opus 4.8 prices, ~$2.5 worst case, $30–70 a month total. The QA team has grown since — I went from the only QA to leading it — but triage stayed with the bot: people do automation and agent experiments, not red-run duty. The automation was born from a shortage of hands and stayed because it's simply better.\n\nThey do. [Cognition](https://cognition.com/) sells [Devin Auto-Triage](https://devin.ai/auto-triage); GitHub shipped a [“Fix with Copilot”](https://github.blog/changelog/2026-05-18-one-click-fixes-for-failing-actions-with-copilot-cloud-agent/) button on failed workflow runs. We saw all that and deliberately built our own — because our task isn't \"fix the red CI\". It's a decision about the fate of a release: classify the cause, pick a route (ticket / rerun / escalation), dedup, auto-close, wake nobody without a reason — welded to our exact stack (Allure TestOps, Linear, Slack, two repos) and our processes.\n\nAnd the price tag: for an AI-first team this is days, not weeks — the code is almost entirely written with Claude Code. The honest division of labor: the human part is the prototype and the architecture — verdict separated from routing, idempotent recovery steps, pure-function classifiers, state in ticket comments instead of a database. AI writes great code inside a given structure and speeds iteration up by an order of magnitude; the structure that survives six releases in two months is set by a human.\n\nWhy not open-source it, then? Similar \"AI triages your failed CI\" actions sit at 3–11 GitHub stars, and ours needs the exact Linear + Slack + Allure TestOps + Anthropic combination — that audience is vanishingly small, while baseline \"AI explains the failure\" is arriving in GitHub out of the box. What's worth open-sourcing here are the ideas, not the code — which is exactly what this article is.\n\nAI-first means AI-first.", "url": "https://wpnews.pro/news/17-prs-a-day-one-qa-how-we-automated-e2e-failure-triage", "canonical_source": "https://dev.to/denis_skvortsov/17-prs-a-day-one-qa-how-we-automated-e2e-failure-triage-260g", "published_at": "2026-07-15 13:30:13+00:00", "updated_at": "2026-07-15 14:03:51.693184+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning", "large-language-models", "mlops"], "entities": ["pdf.net", "Claude", "GitHub Action", "Allure TestOps", "Linear", "Slack", "Vercel"], "alternates": {"html": "https://wpnews.pro/news/17-prs-a-day-one-qa-how-we-automated-e2e-failure-triage", "markdown": "https://wpnews.pro/news/17-prs-a-day-one-qa-how-we-automated-e2e-failure-triage.md", "text": "https://wpnews.pro/news/17-prs-a-day-one-qa-how-we-automated-e2e-failure-triage.txt", "jsonld": "https://wpnews.pro/news/17-prs-a-day-one-qa-how-we-automated-e2e-failure-triage.jsonld"}}