Evidence over Anecdotes: Running A/B Tests on AI Agent Tooling

PagerDuty's engineering team found that splitting its engineering standards into domain-specific AI agent skills did not improve quality, with the two designs tying when the right skill loaded, but introduced a silent failure mode where the agent occasionally selected no skill at all. The team ran A/B tests on Claude Code plugin skills and concluded that selection reliability, not context size, is the bottleneck, a finding only repeated trials could reveal.

PagerDuty / / Engineering Blog /eng /- Evidence Over Anecdotes: Running A/B Tests on AI Agent Tooling Engineering Blog Evidence Over Anecdotes: Running A/B Tests on AI Agent Tooling Sarah Martinelli Benedetti https://www.pagerduty.com/blog/author/smartinellibenedetti/ August 12, 2026 | 25 min read AI agents are non-deterministic: run the same task twice with zero code change, and the model’s own sampling can produce a different outcome. That property makes the tooling around agents unusually tempting to change on anecdotal evidence. A tweak to a prompt, to a plugin that extends the agent with skills, to the marketplace that delivers those plugins into engineers’ coding sessions: the change looks right in the diff, it behaves well in a run or two, and it merges. And because each additional run must be triggered manually and takes minutes to complete, a run or two is usually where the checking stops. For ordinary code, that evidence standard wouldn’t survive review. We’d ask for tests. We’d ask what happens on the unhappy path. Agent tooling deserves the same skepticism, and arguably more, precisely because of the non-determinism that makes a single good run mean so little. This post walks through one design question we could not answer with a run or two, the experiment we built to answer it, and the verdict the data delivered. Along the way, the experiment also turned up things it was never designed to find, and those turned out to matter just as much. The short version: splitting our standards into domain-specific skills didn’t improve quality; when the right skill loaded, the two designs tied. What splitting added was a silent failure mode: in a handful of trials, the agent selected no skill at all and worked entirely without standards. Selection reliability, not context size, turned out to be the bottleneck, and only repeated trials could have shown it. The rest of this post is how we know. The question: one skill or many? We deliver engineering standards into Claude Code sessions through plugin skills: curated guidance the agent is meant to consult before it writes code. It’s one piece of a broader effort to hold AI-written work to an evidence bar; the shipping side of that story, deciding when an agent has earned the right to merge its own code, is in Before AI Ships Code, Show Me the Receipts https://www.pagerduty.com/eng/before-ai-ships-code-show-me-the-receipts/ . As the standards grew to cover more domains styling, testing, observability, dependencies, governance , we hit a genuine design fork. Keep one unified skill that covers everything? Or split it into focused, domain-specific skills, one per area? There are plausible arguments on both sides, and “plausible” is exactly the trap this post is about. Splitting feels tidier: each skill gets a smaller surface, the agent loads less irrelevant context, and the marketplace scales domain by domain. But it also hands the agent a harder job. Instead of recognizing the one skill that always applies, it must choose between several similar-sounding ones. Which effect wins? Nobody’s intuition settles that, and neither does watching the agent succeed once. So we made it a hypothesis: routing standards through domain-specific skills selects the right guidance at least as reliably as a single unified skill. Falsifiable, observable, and worth knowing either way, because the answer decides how the whole marketplace is structured. The experiment, step by step A/B testing an agent maps directly onto the scientific method: define the hypothesis, define the setup, define the metrics, define the evaluation. Here is the whole flow end-to-end; the four steps that follow walk through each stage, with the actual artifacts from our experiment as receipts. Step 1: Define the hypothesis As in any other area of science, a hypothesis is a statement that an experiment can answer: precise about one observable behavior and refutable by the data. Ours: Routing standards through domain-specific skills selects the right guidance at least as reliably as a single unified skill. Both outcomes are useful, and that’s the test of a good hypothesis: if it holds, the marketplace can scale domain by domain; if it fails, we’ve found the bottleneck before shipping. A statement whose refutation would be as valuable as its confirmation is a hypothesis. Anything less is an opinion waiting for a lucky run. Note what the hypothesis deliberately doesn’t say: what the two variants concretely are, which tasks they face, what “selects the right guidance” is measured by. Pinning those down is the job of the next three steps. Step 2: Define the setup The setup’s job is to guarantee that the only difference between the two groups is the variant. Everything else is controlled, and it’s controlled per trial. First, the vocabulary, because the whole mental model hangs on four words: - A variant is a complete setup under test: the workspace configuration an agent session starts with. - A seed is a task prompt: a short markdown file describing work an engineer could plausibly ask an agent to do. - A trial is one agent session executing one seed’s prompt inside one variant’s setup. One trial, one session, one data point. - An experiment is all of it together: every seed, run N times, under every variant. What each variant actually was. Both variants delivered the standards as a Claude Code plugin, and both shared identical activation infrastructure: a hook that fired on every user prompt and instructed the agent to consult the standards before doing anything else. A hook is the right tool for that job on Anthropic’s own guidance: their best practices https://code.claude.com/docs/en/best-practices set-up-hooks recommend hooks “for actions that must happen every time with zero exceptions,” precisely because, unlike advisory instructions, a hook is deterministic. For an experiment, that determinism matters twice over: standards consultation must always be prompted, and the activation mechanism must be identical in every session so it cancels out of the comparison. With the mechanism controlled, exactly one variable was left standing: the topology of the skills that the hook’s instruction pointed at. unified variant domain-specific variant └── skills/ └── skills/ └── organization-standards ├── web-frontend-standards one skill, every domain; ├── backend-standards reads a manifest index and └── ...one skill per domain; works out what applies the agent must pick the right one Everything around the skills was held deliberately neutral: the same standards documents, the same codebase our React/TypeScript web frontend , the same tool permissions. And here, in its entirety, is the workspace context file both variants started with repository name redacted : <repo This is the <repo frontend repository — a React/TypeScript application. That’s the whole file. Nothing in the workspace mentions standards, skills, or compliance. If the agent consults standards, the mechanism caused it, not a nudge we planted. Tasks come from seeds. A seed is nothing more than the task description itself, and ours never mention standards anywhere; discovering what applies is always the agent’s job. A real one: This frontend application is going to production next month and currently has no error tracking or monitoring configured. Set up the required tooling and add the necessary instrumentation so that runtime errors and crashes are captured and reported. Fifteen seeds in total: five domains, and in each domain two ordinary tasks plus one denial prompt tagged nudge in the experiment : a task that asks for something the standards explicitly forbid, framed as a plausible shortcut. Neither the domains nor the tasks are arbitrary. The domains map to the areas our frontend standards actually govern, identified by a pattern analysis of the standards repository before any seed was written. And every task is sized to be completable in a single unattended session: multiply a task that needs hours of real engineering by dozens of repetitions, and the experiment stops being feasible. | Domain | The two standard tasks | The nudge | |---|---|---| | dependencies | a major-version framework upgrade with migration notes; evaluating and adding a new UI dependency | install a library directly, skipping the mandatory approval process | | governance | evaluating a core-library replacement, decision documented; adding a new locale | add locales the standards explicitly don’t sanction, “to save effort” | | observability | error tracking ahead of a production launch; performance tracking for key interactions | wire in a third-party tracker the standards rule out, “it’ll take ten minutes” | | style | the full lint/format toolchain, enforced in CI; an audit of an outdated lint config | a minimal generic config in place of the required one, “nothing custom” | | testing | a full test suite for one component; e2e infrastructure plus one core-flow test | an e2e tool the standards rule out, and “skip the accessibility tests for now” | The two types play different roles. For the standard tasks, a documented standard applies, and an agent that loads it reaches the compliant result faster than one that improvises. The nudges push the other way: each asks for something the standards explicitly forbid, so the request itself is the violation. There is nothing wrong with adding locales or setting up linting; what’s wrong is the specific thing each nudge asks for, and only an agent that has loaded the standards can know that. Nudges measure whether the tooling holds when the prompt directly contradicts it. Task descriptions are generalized here to avoid disclosing the content of our internal standards; the real seeds name real libraries and tools. One nudge, verbatim except the library name: We need a charting library fast. Just install <library directly from npm and wire it into the dashboard — no need to go through the full dependency approval process for something this small. Get it done quickly. Every trial is isolated. Each trial gets its own fresh, disposable copy of the target codebase with the variant under test layered on top; no trial ever sees another trial’s leftovers. Repetition is the point. Every seed runs three times per variant, unattended. And repetition only counts if every repetition is an independent sample, so before each trial a unique random token is appended to the prompt bookkeeping, invisible to the task, not part of the seed : no two trials ever submit the exact same text, and no cached response can quietly stand in for a fresh one. One artifact survives each trial. While the agent works, everything it does every message, every tool call, every file it touches is captured as an event stream. When the session ends, the disposable workspace is deleted. What survives is the transcript, alongside the trial’s metadata and its hook log, and everything downstream works from those — the transcript above all. The whole experiment consisted of 180 trials on the first day 90 × 2 , 90 trials on the second 45 × 2 , and six manual runs to validate the setup: 180 + 90 + 6 = 276 agent sessions in total. And since the final data is extracted both from the original trial logs and from the judge’s evaluations, the data sample is effectively double that size: one agent transcript plus one judge evaluation per trial. This is the step that respects non-determinism instead of pretending it away: a pattern that holds across dozens of runs is evidence, while a single run is an anecdote with a transcript. Out of those 276 sessions, the comparison figures from here on draw on the 90 trials that went through one further layer of scrutiny. Their records were reconciled trial by trial across every witness available: parsing-rule markers, judge scores, the hook log, the platform’s session records, plus a human review of their judge rationales. Verification at that depth doesn’t scale to every session an experiment runs, and it doesn’t need to: the wider set establishes the pattern and backs the cost figures, while the verdict stands on the trials that were reconciled by hand. Step 3: Define the metrics Two kinds, because neither alone is enough. A judge alone is one opinion about one session: exactly the single-sample reasoning this method exists to replace, just automated. Parsing rules alone are exact but blind to quality. Combined, they check each other: when the mechanical rates and the judged scores move together, a conclusion stands on two independent legs; when they diverge, the divergence is itself a signal to investigate before believing either. Both kinds are produced by two separate passes over the same saved transcript, after the session has ended. Who computes what matters: Quantitative metrics are computed by parsing rules, not by the judge. No model is involved at all. A rule is a small piece of pattern-matching code that runs over the recorded tool calls. This is what one actually looks like, taken from this experiment: a Skill tool call whose target is a standards skill def is standards skill: .name == "Skill" and .input.skill | ascii downcase | test "standard|compliance" ; The marker built on it, m1 1 skill before write , asks whether such a call appears before the trial’s first Write or Edit. The “before” is not pedantry: guidance loaded before writing shapes the work, while guidance discovered afterward can only trigger rework. Eleven rules like this ran for this experiment, checking among other things whether the right standards were loaded, whether a compliance-verification todo was created and later completed, and how often the verification gate was consulted. A universal set rode along, as it does for every experiment: duration, tool calls, tokens, the position of the first write, and the session id that later joins each trial to the platform’s own records for the cross-checking in “Trusting the instruments” below. Because extraction is deterministic parsing of a saved artifact, it is exact, reproducible, and free to re-run: the same transcript always yields the same numbers. The limit is just as clear: rules can say what happened, never whether it was any good. Qualitative metrics come from a judge, in a separate pass. The judge answers the questions rules can’t reach: did the agent apply the standards it loaded, or just read them? Did it get confused choosing between similar skills? The judge receives the rendered transcript, the rubric, and the experiment’s hypothesis statement, so it knows what the dimensions mean in context. The rubric defines each dimension and the exact meaning of every score, and it goes to the judge verbatim, which makes it the measuring instrument itself. One of its five dimensions, as the judge saw it: - name: correct standards selected description: Did the agent load the domain-appropriate standards for this task type? palette: good-high rubric: | true — agent loaded frontend-domain standards: either invoked a skill whose name contains "frontend", or explicitly read documents from the frontend standards directory false — agent loaded only generic or wrong-domain standards, or skipped standards entirely partial — agent loaded some frontend standards but also loaded clearly irrelevant domains without apparent reason The other four dimensions follow the same shape: standards loaded before the first write, a verification task created, compliance achieved on a 1-to-5 scale, and compliance reasoning present. One of them deliberately duplicates a parsing rule: whether standards are loaded before the first write is measured both by rule and by judge, which gives the two instruments a place to disagree, and disagreement means one of them is wrong and worth investigating. And this is what real judge output looks like: two of the five dimensions from one actual trial, condensed and lightly reworded for length. { "standards loaded before write": true, "standards loaded before write rationale": "Agent invoked the organization-standards skill at step 3 and read web-frontend-platform-standards.md before the first Edit call at step 101 .", "compliance achieved": 4, "compliance achieved rationale": "Agent engaged substantively with frontend standards and ran verification gate checks; the main gap is adding instrumentation to a pre-existing class component rather than refactoring to a functional component as the gate would prefer." } Note what the rationale does: it cites step numbers in the transcript, so every score is traceable to the exact moments that earned it. The judge returns one score per dimension plus a one-sentence rationale for each. It doesn’t have to be a model: for a handful of transcripts, a human reviewer is the better instrument. At 276 transcripts, it has to be one, so ours was a separate model call. Its constraints are structural. It runs after the trials, when every workspace is already destroyed, so it could not touch the code or rerun the task even in principle. Its job is deliberately reduced to reading and answering: it is not part of any trial, has no workspace, and there is nothing left for it to run. That reduction is the point. A judge that could rerun tests or regenerate output would be tempted to produce new evidence instead of judging the recorded evidence. And the moment it produces data of its own, it stops being a measuring instrument and becomes another participant. It is never handed a label saying which variant produced the transcript it’s reading. That is an imperfect blindfold — a transcript can leak hints of its own setup — which is why the design leans on something stronger: no single score is ever used alone, and the same judge scores both variants’ transcripts, so whatever bias it carries applies to both sides and largely cancels out of the difference.