The localization gap
Finding the file is only the start: across these configurations, coverage falls by 13–22 percentage points when agents must identify the causal code blocks.
A benchmark for distributed-system diagnosis
When a distributed system fails, can your agent find the code that caused it?
DistDebug-Bench gives agents 120 real failures from 11 production distributed systems. Failures surface far from their cause, and reports, logs, and stack traces show only part of what happened.
| Top five configurations by block coverage | |||
|---|---|---|---|
| Model | Block coverage 95% CI | Block precision | Tokens / run |
| --- | --- | --- | --- |
| GPT-6 Astra | 72.6 [65.7, 79.3] | 46.0 | 130K |
| Claude Opus 5.5 | 71.8 [65.2, 78.2] | 49.0 | 149K |
| GPT-5.5 | 70.0 [63.1, 76.7] | 37.3 | 474K |
| GPT-5.4 | 66.6 [59.7, 73.4] | 41.6 | 687K |
| GPT-5.3-Codex | 63.2 [55.9, 70.4] | 42.7 | 417K |
The benchmark
Each incident is a real, closed bug from a production system’s issue tracker, most often a logic, concurrency, or error-handling fault. The agent gets the report, its logs or stack traces, and the repository before the fix, and must name the code blocks that caused the failure. Most coding benchmarks grade the patch; DistDebug-Bench scores the diagnosis itself, against human-labeled causal code.
| System | What it is | Language | Incidents |
|---|---|---|---|
| Ray | AI compute engine | Python, C++ | 21 |
| Elasticsearch | Search and analytics engine | Java | 19 |
| ZooKeeper | Coordination service | Java | 18 |
| TiKV | Transactional key-value store | Rust | 11 |
| HDFS | Distributed file system | Java | 10 |
| Kubernetes | Container orchestration | Go | 10 |
| etcd | Consistent key-value store | Go | 8 |
| Kafka | Event streaming platform | Java, Scala | 8 |
| Ignite 3 | Distributed database | Java | 7 |
| HBase | Wide-column store | Java | 5 |
| Ignite | In-memory database | Java | 3 |
68% of reports never name the causal method
63% have no usable stack trace from inside the project
46% need more than one code block to explain
5 lines is the median labeled block, so answers must be precise
Findings
Block coverage is the share of the labeled root-cause code blocks a diagnosis reaches; block precision is the share of the code it names that is part of the cause.
Without a pointer to the code, coverage drops sharply.
Reports without direct code clues show more run-to-run variation.
Means average three runs within each incident and weight the 120 incidents equally; whiskers are 95% incident-bootstrap confidence intervals.
Use DistDebug-Bench
Run Mini-SWE-Agent on ZooKeeper-2845 with the paper’s protocol. From a benchmark checkout, the runner prepares an anonymized workspace at the buggy ZooKeeper revision, runs the agent in a sandbox, and the scorer applies the paper’s evaluator. Python 3.10+, Linux with bubblewrap, and a configured model API key are required.
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[agent]"
python -m distdebug_bench.run \
--config configs/mini-swe-agent/gpt-5.4.yaml \
--bug zookeeper-2845 --runs 1
python -m distdebug_bench.score \
results/mini-swe-agent-gpt-5.4 --attempts 1
configs/mini-swe-agent/ has a config for each model in the paper. Runs call your configured model provider.
Each attempt writes trajectory.json, diagnosis.json, and diagnosis_scores.json. To evaluate another agent, have it write the first two.
Run your agent on all 120 incidents, three attempts each, and send its diagnoses and trajectories. We rescore every submission with the official evaluator before listing it on the leaderboard.
diagnosis.json and the full trajectory for every attempt, plus the model, harness, settings, and token use or cost.
The guide covers the local leakage check, packaging a run, and the diagnosis.json schema.