# DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? > Source: > Published: 2026-11-13 05:19:29+00:00 ### The localization gap Finding the file is only the start: across these configurations, coverage falls by **13–22 percentage points** when agents must identify the causal code blocks. A benchmark for distributed-system diagnosis When a distributed system fails, can your agent find the code that caused it? DistDebug-Bench gives agents 120 real failures from 11 production distributed systems. Failures surface far from their cause, and reports, logs, and stack traces show only part of what happened. | Top five configurations by block coverage | | | | |---|---|---|---| | Model | Block coverage 95% CI | Block precision | Tokens / run | |---|---|---|---| | GPT-6 Astra | 72.6 [65.7, 79.3] | 46.0 | 130K | | Claude Opus 5.5 | 71.8 [65.2, 78.2] | 49.0 | 149K | | GPT-5.5 | 70.0 [63.1, 76.7] | 37.3 | 474K | | GPT-5.4 | 66.6 [59.7, 73.4] | 41.6 | 687K | | GPT-5.3-Codex | 63.2 [55.9, 70.4] | 42.7 | 417K | The benchmark Each incident is a real, closed bug from a production system’s issue tracker, most often a logic, concurrency, or error-handling fault. The agent gets the report, its logs or stack traces, and the repository before the fix, and must name the code blocks that caused the failure. Most coding benchmarks grade the patch; DistDebug-Bench scores the diagnosis itself, against human-labeled causal code. | System | What it is | Language | Incidents | |---|---|---|---| | Ray | AI compute engine | Python, C++ | 21 | | Elasticsearch | Search and analytics engine | Java | 19 | | ZooKeeper | Coordination service | Java | 18 | | TiKV | Transactional key-value store | Rust | 11 | | HDFS | Distributed file system | Java | 10 | | Kubernetes | Container orchestration | Go | 10 | | etcd | Consistent key-value store | Go | 8 | | Kafka | Event streaming platform | Java, Scala | 8 | | Ignite 3 | Distributed database | Java | 7 | | HBase | Wide-column store | Java | 5 | | Ignite | In-memory database | Java | 3 | **68%** of reports never name the causal method **63%** have no usable stack trace from inside the project **46%** need more than one code block to explain **5 lines** is the median labeled block, so answers must be precise Findings Block coverage is the share of the labeled root-cause code blocks a diagnosis reaches; block precision is the share of the code it names that is part of the cause. Without a pointer to the code, coverage drops sharply. Reports without direct code clues show more run-to-run variation. Means average three runs within each incident and weight the 120 incidents equally; whiskers are 95% incident-bootstrap confidence intervals. Use DistDebug-Bench Run Mini-SWE-Agent on ZooKeeper-2845 with the paper’s protocol. From a benchmark checkout, the runner prepares an anonymized workspace at the buggy ZooKeeper revision, runs the agent in a sandbox, and the scorer applies the paper’s evaluator. Python 3.10+, Linux with bubblewrap, and a configured model API key are required. ``` python3 -m venv .venv source .venv/bin/activate pip install -e ".[agent]" # Set OPENAI_API_KEY in your environment. python -m distdebug_bench.run \ --config configs/mini-swe-agent/gpt-5.4.yaml \ --bug zookeeper-2845 --runs 1 python -m distdebug_bench.score \ results/mini-swe-agent-gpt-5.4 --attempts 1 ``` `configs/mini-swe-agent/` has a config for each model in the paper. Runs call your configured model provider. Each attempt writes `trajectory.json`, `diagnosis.json`, and `diagnosis_scores.json`. To evaluate another agent, have it write the first two. Run your agent on all 120 incidents, three attempts each, and send its diagnoses and trajectories. We rescore every submission with the official evaluator before listing it on the [leaderboard](leaderboard/). `diagnosis.json` and the full trajectory for every attempt, plus the model, harness, settings, and token use or cost. The guide covers the local leakage check, packaging a run, and the `diagnosis.json` schema.