DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

A new benchmark, DistDebug-Bench, tests whether LLM agents can pinpoint the code that caused failures in distributed systems, and finds that even the best model, GPT-6 Astra, reaches only 72.6% block coverage with 46.0 block precision across 120 real incidents from 11 production systems. The benchmark's authors report that coverage falls by 13–22 percentage points when agents must identify causal code blocks rather than just the affected file, and that 68% of incident reports never name the causal method while 63% contain no usable stack trace from inside the project. The incidents, drawn from systems including Ray (21), Elasticsearch (19), ZooKeeper (18), TiKV (11), HDFS (10) and Kubernetes (10), are scored on diagnosis against human-labeled causal code rather than on the submitted patch, with the median labeled block just 5 lines long.

The localization gap Finding the file is only the start: across these configurations, coverage falls by 13–22 percentage points when agents must identify the causal code blocks. A benchmark for distributed-system diagnosis When a distributed system fails, can your agent find the code that caused it? DistDebug-Bench gives agents 120 real failures from 11 production distributed systems. Failures surface far from their cause, and reports, logs, and stack traces show only part of what happened. | Top five configurations by block coverage | | | | |---|---|---|---| | Model | Block coverage 95% CI | Block precision | Tokens / run | |---|---|---|---| | GPT-6 Astra | 72.6 65.7, 79.3 | 46.0 | 130K | | Claude Opus 5.5 | 71.8 65.2, 78.2 | 49.0 | 149K | | GPT-5.5 | 70.0 63.1, 76.7 | 37.3 | 474K | | GPT-5.4 | 66.6 59.7, 73.4 | 41.6 | 687K | | GPT-5.3-Codex | 63.2 55.9, 70.4 | 42.7 | 417K | The benchmark Each incident is a real, closed bug from a production system’s issue tracker, most often a logic, concurrency, or error-handling fault. The agent gets the report, its logs or stack traces, and the repository before the fix, and must name the code blocks that caused the failure. Most coding benchmarks grade the patch; DistDebug-Bench scores the diagnosis itself, against human-labeled causal code. | System | What it is | Language | Incidents | |---|---|---|---| | Ray | AI compute engine | Python, C++ | 21 | | Elasticsearch | Search and analytics engine | Java | 19 | | ZooKeeper | Coordination service | Java | 18 | | TiKV | Transactional key-value store | Rust | 11 | | HDFS | Distributed file system | Java | 10 | | Kubernetes | Container orchestration | Go | 10 | | etcd | Consistent key-value store | Go | 8 | | Kafka | Event streaming platform | Java, Scala | 8 | | Ignite 3 | Distributed database | Java | 7 | | HBase | Wide-column store | Java | 5 | | Ignite | In-memory database | Java | 3 | 68% of reports never name the causal method 63% have no usable stack trace from inside the project 46% need more than one code block to explain 5 lines is the median labeled block, so answers must be precise Findings Block coverage is the share of the labeled root-cause code blocks a diagnosis reaches; block precision is the share of the code it names that is part of the cause. Without a pointer to the code, coverage drops sharply. Reports without direct code clues show more run-to-run variation. Means average three runs within each incident and weight the 120 incidents equally; whiskers are 95% incident-bootstrap confidence intervals. Use DistDebug-Bench Run Mini-SWE-Agent on ZooKeeper-2845 with the paper’s protocol. From a benchmark checkout, the runner prepares an anonymized workspace at the buggy ZooKeeper revision, runs the agent in a sandbox, and the scorer applies the paper’s evaluator. Python 3.10+, Linux with bubblewrap, and a configured model API key are required. python3 -m venv .venv source .venv/bin/activate pip install -e ". agent " Set OPENAI API KEY in your environment. python -m distdebug bench.run \ --config configs/mini-swe-agent/gpt-5.4.yaml \ --bug zookeeper-2845 --runs 1 python -m distdebug bench.score \ results/mini-swe-agent-gpt-5.4 --attempts 1 configs/mini-swe-agent/ has a config for each model in the paper. Runs call your configured model provider. Each attempt writes trajectory.json , diagnosis.json , and diagnosis scores.json . To evaluate another agent, have it write the first two. Run your agent on all 120 incidents, three attempts each, and send its diagnoses and trajectories. We rescore every submission with the official evaluator before listing it on the leaderboard leaderboard/ . diagnosis.json and the full trajectory for every attempt, plus the model, harness, settings, and token use or cost. The guide covers the local leakage check, packaging a run, and the diagnosis.json schema.