A Better Benchmark Harness for SRE Agents

PagerDuty's SRE Agent led a new fault-injection benchmark across 17 Kubernetes microservice scenarios, scoring highest on issue localization, root-cause diagnosis, evidence grounding, mitigation recommendations, and explanation quality, according to PagerDuty's report on the harness. The harness injects realistic faults into live Kubernetes microservices and scores each agent on those five criteria using a panel of judges built on different models to reduce single-model bias. PagerDuty said the two factors that mattered most were the strength of the underlying model and how much context the agent could reach.

Summary This report shows how to create a benchmark harness that injects realistic faults into Kubernetes microservices and has SRE agents work through them. Each agent is scored on five things: locating the fault, diagnosing the root cause, identifying a mitigation, grounding its conclusions in evidence, and explaining its reasoning clearly. A panel of judges built on different models does the scoring, which keeps any single model’s bias out of the results. Initial comparative results demonstrate that PagerDuty’s SRE Agent consistently led the benchmark across 17 scenarios on issue localization, diagnosis, evidence grounding, mitigation recommendations, and explanation quality. The results also point to two factors that matter most: the strength of the underlying model and how much context the agent can reach. Both give teams a clear direction for improving how AI resolves incidents. Why Build a Benchmark Harness for SRE Agents A reliable benchmark measures whether changes make an SRE agent better at resolving incidents. Building a basic SRE agent is easy. You take a powerful LLM, give it access to code and observability via MCP and put it all together in a simple harness. Job done. But to get from basic to great requires asking additional questions. How well does it work now? Is it solving the problem effectively? Is it doing it efficiently? How do you know the complexity you have added actually made things better? As you work to improve your agent, you need a reliable way to quantify the changes. The obvious first step is to replay the agent on past incidents and compare its output with what your post-incident review says happened. We do this, and it’s useful, but it has hard limits as an evaluation method: - The reference data is only as good as the information you have from post-incident reviews, observability, collaboration tools, and other systems that you use during incident resolution. For example, often reviews are written for people, not for scoring an agent, so they’re incomplete or contested, and may not have all the required information. - Replaying past signals, events and actions is hard, and hindsight often leaks through. The agent has to see code, config, and telemetry as they were when the incident started, and nothing after. Any gap in that tooling lets the fix, postmortem, or later telemetry leak into the answer, including the title of the incident. - It only works for agents you control. We can wire time-travel tooling into our own agents. A third-party agent queries live systems through its own integrations, so it can’t be replayed fairly. - Past incidents can’t be changed. They can’t be re-run to measure variance, modified to test a specific capability, or evaluated with a data source removed or added . - You get the incidents you had, not necessarily the ones you need. Complex incidents are rare, many trace back to third-party outages whose cause isn’t visible in our own telemetry or code, and coverage follows whatever happened to break. Injecting faults into a live system addresses each of these. The ground truth is exact because we caused it. Every agent investigates the present through its normal integrations. Every scenario can be repeated, varied, and ablated. And we choose which faults, technologies, and languages to cover instead of waiting for them to fail. Replay still has a role for signals a synthetic system can’t reproduce, such as incident-call transcripts see Impact of transcript data , but for benchmarking we needed something new. Another benefit of having a good benchmarking suite is for our customers: there are now many general-purpose and SRE-specific AI agents, and they want to know how PagerDuty’s SRE Agent compares. Answering that honestly means testing every agent the way a customer would run it, on the same faults and against the same ground truth. A well-designed benchmarking suite will allow us to do this as fairly as possible. Our first question was whether we could adopt or adapt a benchmark that already exists. There are a number of public benchmark harnesses available, but they all presented a few drawbacks: 1. They normally rely on calling agents directly, with structured outputs shaped to the benchmark. When interacting with third-party agents, it is not always possible to define the interaction process in a way that is optimised for that benchmark. For example, PagerDuty’s SRE Agent uses a chat interface, it can be guided to provide a given output shape but not forced to. Should it be punished because it doesn’t output JSON? 2. Frequently, they run on publicly available microservice demo apps such as OpenTelemetry’s Astronomy Shop https://github.com/open-telemetry/opentelemetry-demo or the SAIL Group’s DeathStarBench https://github.com/delimitrou/DeathStarBench which are captured in LLM training. This means that the models underlying SRE agents already know these systems, and in some cases they are even able to recall failure injection points from training data we saw this happen multiple times when we tested our agents on ITBench . 3. Few benchmarks included code as a source of data for agents to use, potentially because of the issue specified in the previous point or because the code contains the answer explicitly written in a feature flag. One of the key questions we wanted to ask is “what is the value of providing code”. To answer this question we needed a clean codebase that did not contain the answer. 4. Most public harnesses are built around a single fault-injection mechanism, e.g. through changing a feature flag to trigger the event or using Chaos Mesh https://chaos-mesh.org/ to break the Kubernetes k8s substrate. Ours uses four: Helm value overrides, Chaos Mesh, real PR merges, and AWS control-plane changes, with the ability to add more as we desire such as third party failures or security incidents. This allows us to test a wider range of issues and probe agents against more complex scenarios. For these reasons, we created our own benchmark harness. It is a purpose-built financial application that runs on Kubernetes microservices, emits telemetry, has alerting rules on Grafana and sends those alerts to PagerDuty or the agent’s own SaaS platform . With this standard infrastructure in place, we can integrate the stack with a variety of agents, and grant them access to metrics, logs, alerting and code to use in their RCA. This accounted for all the basic information that is ingested by all SRE agents but does not include some of the highest value information that we have for PagerDuty’s SRE Agent: - Live transcripts of incident calls are accessible via the Scribe Agent; we have found that the interaction between human and agent responders provides significant value in complex incidents. - Historical incidents and memories saved by the agent allow it to learn how a specific system works, and increase performance over a generic agent. Neither of these are easy to include in a benchmarking suite. LLMs can generate a fake incident call, but you don’t want them leaking the answer. Past incidents need to be similar enough to be useful but not the same. Memories should be clues, not solutions. Likewise, the general-purpose agents were run in default configurations, without SRE-specific instruction files, skills, or subagents that a customer might add. We are publishing this report to outline how we are testing our agents and highlight the strengths and limitations of our work. Future reports will cover different aspects of that testing in more detail as we continue to work on it. This report is focused on the underlying principles we have adopted when building the benchmark, and the initial results we have achieved using it. This benchmark has been undertaken knowing that there is a conflict of interest, we are both testing and building agents based on the same vision of what an SRE agent should be. To reduce this bias, we have implemented the following protections: - A similar, neutral, prompting approach is used across all agents, with variations only to account for differences in how the agents take inputs, provide outputs and ensure they are providing the information required for the scoring. - Scoring is done using a 3-model judge ensemble from 3 different providers to prevent same provider bias e.g. an agent using one model family might be preferentially scored by another model from the same family . - Scoring against pre-defined sub-criteria written at scenario creation. - A contamination scanner that runs on every sweep, checking for leakage from e.g., source code. - We are publishing our methodology, allowing people to critique it and raise concerns with our approach. At the moment, we are keeping this codebase private since we do not want it leaking into the LLMs training data , but we intend to make it available in the future to allow others to validate our results. How Does the Benchmark Methodology Work What code substrate was used Our benchmark harness deploys a realistic, purpose-built financial app that runs on a number of interconnected Kubernetes services. It is a mono-repo consisting of 11 micro-services written in Go, Python, TypeScript and Elixir using Kafka, Redis, PostgreSQL and Secrets Manager, all running on EKS on AWS. They are instrumented to emit metrics and logs to Grafana Cloud. On the Grafana stack a number of alerting rules were created on telemetry such as HTTP errors, request latency, Kubernetes infrastructure health, etc. Additional services are stubbed out and will be added in future iterations using additional coding languages e.g. Java and technologies e.g. MySQL to provide a variety of scenarios and issues that can be used to test the agents. This organic growth has allowed us to test whilst we have continued to grow the benchmark. Each service has a runbook that outlines how it works and how to debug “common faults”. These have been written by an LLM to replicate typical runbooks from effective teams and have been made available to all agents. An external load-generation service posts requests to the financial app’s endpoints and exercises the whole system. This load-generation system does not emit MELT metrics, events, logs and traces data to the shared Grafana Cloud and is hidden from the agents. What scenarios were tested We designed a number of self-contained incident scenarios, where operationally realistic faults are injected, leading alert rules to fire. These alerts are the trigger for agents to start their analyses. Faults can be injected using four different approaches: - Helm value overrides – allowing for changes in the EKS configuration, e.g., port number, pod count or load-generation workloads - Chaos Mesh https://github.com/chaos-mesh/chaos-mesh scenario injection. For this reason, we have had to prevent the SRE agents from having access to the EKS logs, which would leak the presence of Chaos Mesh - Real PR merges changing code functionality and visible to agents through PR merges - AWS control-plane changes Scenarios were classified in the following four categories: - Configuration . Every service’s code is correct, but it’s running on a wrong or out-of-date input: an environment variable, a connection endpoint, a credential lifetime, a certificate, or a cloud network rule. The fault usually shows up as errors or latency in a service upstream of the one that’s misconfigured. So the agent has to trace symptoms back through the dependency graph to the specific bad value, not just the service that’s failing. - Code . A logic or contract bug in the application itself. Examples of the pattern are two services disagreeing on the shape of a response, or error handling that turns a client error into a server error. These produce HTTP errors or malformed responses. The discriminating evidence is often in the source code or a recent change, not only in the telemetry, which is why code access matters most here. - Data integrity . A correctness fault where error-rate and latency dashboards stay green, for example duplicate financial records being written. The only signal is the incorrect data itself. The agent has to reason about what the system should have done, not look for something that’s visibly broken. - Operational . The faulty component works correctly, but it’s under resource pressure or degraded in a way that cascades. This covers: - connection-pool and concurrency exhaustion - added latency or packet loss between services - CPU and memory starvation - pod failures - rules tuned wrongly, so valid requests get declined They were often designed to have an impact on a service different from the one that had the problem, testing the SRE agent’s ability to traverse a codebase to determine the topology and call structure, identifying the causal service, not just the symptomatic one. The ground truth was written by an LLM when the scenario was devised and the evidence produced not the agents’ performance against that evidence was reviewed after a couple of runs to ensure that it matched the designed scenario. Which SRE Agents were benchmarked We have tested the following SRE agents: - PagerDuty SRE Agent running in Deep Investigation mode uses GPT 5.6 Sol - Claude Code using Opus 5.5 and Sonnet 5 - Codex using GPT 5.6 Sol, Terra and Luna4 Each agent ran in parallel on the same scenario, ensuring that they all had access to the same data. Default configurations were used for all agents apart from models. All agents were provided with: - Access to Grafana Cloud for MELT data - Source code either via Github MCP or through the source code on disk - Alert details of the triggering alert – this either directly triggered the agent or was passed to the agent as part of the initial prompt. We specifically do not provide access to the k8s substrate directly since we are using Chaos Mesh and direct access would leak that information. We do export filtered logs from k8s to Grafana Cloud with all Chaos Mesh related logs filtered out to prevent this leakage. Post-run reviews at the trace level have found no evidence of the agents finding Chaos Mesh during their analysis. Note: for PagerDuty’s SRE Agent we use the incident triggered by the first alert we receive, no matter what service it impacted. We do not use any of the PagerDuty features such as cross-service alert grouping and between runs all incidents are redacted and memories wiped to prevent them from being found using our past incident functionality, which identifies past incidents that are similar to this one and could be helpful in solving it. In all other respects, it is treated identically to all the other agents. These functionalities would create an uneven setup and, although testing has shown that they improve the results, it would defeat the purpose of this study. For each agent being exercised, a custom adapter is created that defines how to: 1. Set up the agent for the scenario gather credentials, define tools available, etc. 2. Invoke the agent 3. Obtain its outputs and metadata 4. Tear down the agent after outputs are obtained How does the benchmark process work Our benchmarking process runs as follows. 1. Erases all memories from SRE Agents including redacting past incidents in PagerDuty 2. Loops over each scenario: 1. Check that the baseline is healthy, that there are no open incidents and everything is “fresh” k8s is up to date with the latest version, MELT data is being emitted, etc. 2. Sleep for 30 minutes to provide a baseline of “good” MELT data 3. Inject the fault, updating the running services as appropriate. 4. Monitoring within Grafana Cloud detects this fault and emits an alert when the monitors cross a threshold. We use this event to: 1. Trigger an incident within PagerDuty – this triggers the PagerDuty SRE Agent’s investigation 2. Trigger the benchmark harness to start investigations for other agents. 5. Monitor for agents to complete their analysis or until a 30-minute timeout expires, whichever is first 6. Resets to the baseline state 3. Collects all results and latency 4. Runs a script to check for leakage between scenarios was the last scenario still ongoing when this scenario started , from past runs of the benchmarking and from repositories the agent should not have access to. 5. Runs a judge script, assesses all agents using three judges against the full rubric. 6. Produces a summary report Scoring and Metrics All successful agent responses are scored by 3 LLM judges DeepSeek, Claude, GPT to mitigate the risk of same family or provider bias. Agent failures are excluded from the analysis not scored as zero . There is also a per-judge deviation check to ensure that the performance is comparable across judges. The judges are provided with: - The response from the agent to the human. The agent is guided to produce this in a structured format, making it easier for the judge to review, but the ability to dictate this structure varies by agent. - The service with the underlying fault – this is the service name in code, but we accept a number of variations, e.g., payment-gateway or Payment Gateway - The ground truth description of the fault - The ground truth description of the recommended mitigation action - A prompt guiding the judge on how it is to perform the task – this is versioned and comparisons in this report are all with the same judging version. In an incident, the root-cause is often a set of contributing factors and the mitigation of the issue may not imply resolving the issue but ensuring that there’s no impact. For example, when a change causes a problem in the system, we often roll it back to a previous version instead of trying to solve the bug during the incident. As such, the performance of an SRE Agent cannot be judged across a single axis. We split judging over the following axes to determine the performance of SRE Agents - Localization: has the agent correctly identified the service where the issue is originating this is not necessarily the service where the issue is manifesting ? Not finding the true diagnosis but identifying the service which is causing the fault can often help human responders to resolve an incident. - Diagnosis : Is the cause of the issue correctly diagnosed? Often called Root Cause Analysis, this scores whether the agent has correctly diagnosed the fault. Partial credit can be given for partial diagnoses - Mitigation: does the agent correctly identify the ground truth mitigation action? In an incident you want to stop the bleeding fast. Identifying the quickest way to stop that is a high priority. This is scored by comparing the agent’s response against a free-text description of possible mitigations. - Evidence Grounding : Is the diagnosis grounded in evidence? Does it cite logs or code to back up its diagnosis? Providing this information gives confidence to the responders that the agent is correct and ensures that the agent is actually diagnosing the issue rather than just relying on a hallucinated response. - Explanation Quality : How well is the diagnosis written. - Misdirection lower value is better : does the agent point the responder to a component or mechanism that the ground truth does not implicate? This axis rewards agents that focus on a precise diagnosis path, and penalizes ones that cite several potential root causes without committing to a specific item. Theoretically, an agent that cites every possible root cause in the world would get a perfect score without actually being very useful, and misdirection helps unmask these cases. - Average : The average of all of the above with the exception of Misdirection, where we do 1 – Misdirection The LLM judging has been benchmarked against a human evaluation showing agreement, but a full analysis of this work is ongoing. A contamination scanner is also run at this point to verify that there has been no answer-key leakage, GitHub leakage, and cross-scenario contamination. This has identified agents that have found other ways to diagnose the problem, e.g., through broader-scoped keys accessing the benchmarking repo directly. Contaminated agents were removed from the analysis and work was undertaken to prevent future leaks. In the latest runs, no contamination was found, but the scanner is still run to identify any regression. The results presented here are for a single trial per agent. This is a known statistical limitation and ideally we would run a larger number of trials, but, with each scenario taking up to 1 hour to run, we have prioritised diversity of scenarios and quality of the benchmarking suite over statistical rigour at this stage. Final results will be presented using a larger number of trials. Results In this benchmark study, PagerDuty SRE Agent scored first, followed by Claude Code Opus 5.5 second, with Codex Sol 5.6 placing third. The latest complete sweep of 17 scenarios, in equal conditions, produced the following results. | | Average Score | Localization | Diagnosis | Mitigation | Evidence Grounding | Explanation Quality | Misdirection | Latency s | |---|---|---|---|---|---|---|---|---| | PagerDuty SRE Agent Sol 5.6 | 0.77 | 0.76 | 0.65 | 0.75 | 0.99 | 0.93 | 0.41 | 251 | | Claude Code Opus 5.5 | 0.68 | 0.56 | 0.59 | 0.65 | 0.97 | 0.90 | 0.56 | 210 | | Codex Sol 5.6 | 0.66 | 0.59 | 0.58 | 0.68 | 0.85 | 0.78 | 0.53 | 111 | | Codex Terra 5.6 | 0.62 | 0.59 | 0.57 | 0.51 | 0.90 | 0.78 | 0.53 | 93 | | Claude Code Sonnet 5 | 0.59 | 0.47 | 0.52 | 0.47 | 0.95 | 0.90 | 0.65 | 169 | | Codex Luna 5.6 | 0.56 | 0.35 | 0.51 | 0.50 | 0.86 | 0.73 | 0.53 | 274 | Note: All values are means per scenario. For Misdirection and Latency, lower is better; for all other metrics, higher is better. The average score is a simple average of Localization, Diagnosis, Mitigation, Evidence Grounding, Explanation Quality and 1 – Misdirection . Earlier sweep-to-sweep comparisons suggest variability of roughly 2pp, though this sweep is a single trial. On that basis, PagerDuty’s SRE Agent led or was within noise of the lead on every quality metric. It was not the fastest. Holding the model constant, the harness and context also mattered: on Sol 5.6, PagerDuty’s SRE Agent scored 0.77 against Codex’s 0.66. Model strength and operational context appear to add up rather than substitute for each other. Aside from the headline result, one of the most valuable aspects of this work has been that it has identified areas of weakness in PagerDuty’s SRE Agent. The first iteration of its Deep Investigation mode we developed was a complex, multi-agent approach see our blog https://www.pagerduty.com/eng/inside-pagerdutys-sre-agent-how-we-built-deep-incident-investigation/ on how we addressed some of the software engineering challenges . This worked well with the models that we first started developing PagerDuty’s SRE Agent with, but when we tested on the benchmark, it scored lower than a number of the other harnesses on a subset of the above scenarios, about middle rank and was also the slowest. Digging into it we found that: 1. We had over-engineered based on a small subset of test incidents that did not reflect the typical use case 2. Models had improved significantly, in particular with their recall with large contexts, reducing the need for a lot of that engineering 3. Improving our harness with better structuring of data and guidance for the agent allowed it to better compare hypotheses across services Early runs of the benchmarking highlighted these issues and provided motivation to try a different approach. We rebuilt the agent using a new topology, which resulted in the 20pp increase that we have observed, as well as making it both faster and cheaper to run. We acknowledge that this could introduce bias in the results for PagerDuty’s SRE Agent, but after identifying that we had an issue, we avoided “tuning the agent to the benchmark”. Rather, we improved the agent based on the types of issues we identified. Additionally, PagerDuty’s SRE Agent supports investigations lasting minutes to hours, incorporating new signals as an incident unfolds, including call transcripts, messages in Slack or Teams, and additional alerts. This design adds latency compared with the other agents tested. To keep conditions comparable, we evaluated all agents without these additional inputs, so the benchmark measures diagnostic performance without the benefit of ongoing human input or follow-up alerts. How live human input data affects SRE Agent diagnosis During an incident call, conversations between responders can provide great value to steer the direction of an investigation. However, it can also lead to misdirected conclusions. For example, when someone confidently claims that “I found the problem It is a query in the database ” it doesn’t always mean that they have found the problem. Hence, the agent needs to know what to do with incoming live human data. Our platform, via another agent Scribe agent , can capture raw and processed transcriptions and detect live key moments. However, estimating the value of transcript and key moments data in our purpose-built benchmark harness would require creating synthetic live incident call transcript data. Since we expect real call transcripts to include notes about the interactions with the SRE agent e.g., a responder might say “The SRE agent is suggesting that the problem might be related to a recent deployment on Service ABC” , we would need to do this for every agent, for every scenario, for every run Generating high quality and high fidelity synthetic transcripts that take into account live interactions with the agent being tested is a complex and hard to verify project in itself, and therefore we did not include it in the scope of our benchmark harness. To estimate the value of human input data ingested by PagerDuty’s SRE Agent, we instead ran a separate analysis using an incident replay harness on our own past incidents. Estimates on recent incidents point to up to a 5-percentage-point increase in Diagnosis scores of runs where processed transcript data was available versus a baseline without transcripts. Limitations No benchmark is perfect, but that does not mean that it is not useful. In running this benchmark, we noted that: - This is a test system, with test load. It does not handle the volume of production traffic. It does not have the historical complexity that live systems have. - We have made heavy use of the same coding agents we are using as SRE agents to develop. This could make the code easier or harder for the agents to process when they share the same pre- and post-training priors. - The benchmark has been developed at the same time as we have developed the PagerDuty SRE Agent’s Deep Investigation mode. One has inevitably influenced the other, but we endeavored to prevent “training on the test” and focus on generic approaches rather than specific faults. - We have not undertaken a full LLM judge/human comparison with a statistical analysis. Initial comparisons show a good, but not perfect, agreement. This work is ongoing. - We are not using the full capabilities of any agent. The unused capabilities include: - K8s access – some agents support direct access to k8s, providing additional diagnostic information - Chat/call transcript data allowing for human-agent interaction - Past incident data and agent memories. - For general-purpose agents: SRE-specific instruction files, skills, subagents, and additional MCP integrations. This is a single trial per agent, and failed runs are excluded from quality scores see completion rate . - We have focused on triage and diagnosis, with a smaller focus on remediation. This reflects the current capabilities of most SRE agents. But as remediation becomes more prevalent, we will need to increase the level of assessment there. Conclusions The benchmark provides a repeatable way to test changes to PagerDuty’s SRE Agent across scenarios and agent versions. The benchmarking framework helped us develop new capabilities of the SRE Agent such as Deep Investigation. It has provided us with an opportunity to test changes and upgrades in a repeatable manner against different versions of the agent and allowed us to investigate different parameters such as model quality vs. performance is a more powerful model always better, or is a faster, cheaper model sometimes more effective? . The real value of this benchmark is that we are now able to perform ablation studies on things like: - Code Access: How valuable is having access to code? Do we need it? - Traces: Many companies don’t have tracing data, or it is incomplete. However, when present, it can be very useful. In this study, we ran without tracing, but we will also perform analyses with trace data available to see the impact that has on performance. - Topology information: Does having a precalculated topology help, or is it enough for the agent to build it on-the-fly from logs, traces and/or code - Model differences - Harness differences The statistical rigor is also a key question: what is the variance of the different agents against the same scenario? How reliable are the judges? Quantification of these parameters will bring greater clarity to the numbers we measure and conclusions we can draw. We also want to: - Generate more scenarios: - In areas where we find the agents are not performing. - To balance the number of scenarios across the four pillars. - Cover the diversity of languages and technologies we employ. - Harder scenarios. - Add new languages and technologies, including mock 3rd-party services or security incidents. - Compound errors, with multiple concurrent scenarios where the signals interfere, confusing agents. - Test with degraded telemetry. At the moment, we are storing 100% of the MELT data, and it has been well written specifically for this benchmarking. What if the logging is not perfect and there are gaps? What is the value of tracing? Having the framework opens up a huge number of opportunities to develop and improve PagerDuty’s SRE Agent https://www.pagerduty.com/platform/ai-agents/sre/ . At the moment, we can conclude that, with every agent evaluated in a comparable restricted configuration, PagerDuty’s SRE Agent performed as well as or better than the other agents assessed. As we continue to develop, we expect to see it improve in performance and also add new capabilities, such as the ability to conduct remediation actions.