{"slug": "distdebug-bench-can-llm-agents-diagnose-bugs-in-distributed-systems", "title": "DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?", "summary": "A new benchmark, DistDebug-Bench, tests whether LLM agents can pinpoint the code that caused failures in distributed systems, and finds that even the best model, GPT-6 Astra, reaches only 72.6% block coverage with 46.0 block precision across 120 real incidents from 11 production systems. The benchmark's authors report that coverage falls by 13–22 percentage points when agents must identify causal code blocks rather than just the affected file, and that 68% of incident reports never name the causal method while 63% contain no usable stack trace from inside the project. The incidents, drawn from systems including Ray (21), Elasticsearch (19), ZooKeeper (18), TiKV (11), HDFS (10) and Kubernetes (10), are scored on diagnosis against human-labeled causal code rather than on the submitted patch, with the median labeled block just 5 lines long.", "body_md": "### The localization gap\n\nFinding the file is only the start: across these configurations, coverage falls by **13–22 percentage points** when agents must identify the causal code blocks.\n\nA benchmark for distributed-system diagnosis\n\nWhen a distributed system fails, can your agent find the code that caused it?\n\nDistDebug-Bench gives agents 120 real failures from 11 production distributed systems. Failures surface far from their cause, and reports, logs, and stack traces show only part of what happened.\n\n| Top five configurations by block coverage |  |  |  | \n|---|---|---|---|\n| Model | Block coverage 95% CI | Block precision | Tokens / run | \n|---|---|---|---|\n| GPT-6 Astra | 72.6 [65.7, 79.3] | 46.0 | 130K | \n| Claude Opus 5.5 | 71.8 [65.2, 78.2] | 49.0 | 149K | \n| GPT-5.5 | 70.0 [63.1, 76.7] | 37.3 | 474K | \n| GPT-5.4 | 66.6 [59.7, 73.4] | 41.6 | 687K | \n| GPT-5.3-Codex | 63.2 [55.9, 70.4] | 42.7 | 417K | \n\nThe benchmark\n\nEach incident is a real, closed bug from a production system’s issue tracker, most often a logic, concurrency, or error-handling fault. The agent gets the report, its logs or stack traces, and the repository before the fix, and must name the code blocks that caused the failure. Most coding benchmarks grade the patch; DistDebug-Bench scores the diagnosis itself, against human-labeled causal code.\n\n| System | What it is | Language | Incidents | \n|---|---|---|---|\n| Ray | AI compute engine | Python, C++ | 21 | \n| Elasticsearch | Search and analytics engine | Java | 19 | \n| ZooKeeper | Coordination service | Java | 18 | \n| TiKV | Transactional key-value store | Rust | 11 | \n| HDFS | Distributed file system | Java | 10 | \n| Kubernetes | Container orchestration | Go | 10 | \n| etcd | Consistent key-value store | Go | 8 | \n| Kafka | Event streaming platform | Java, Scala | 8 | \n| Ignite 3 | Distributed database | Java | 7 | \n| HBase | Wide-column store | Java | 5 | \n| Ignite | In-memory database | Java | 3 | \n\n**68%** of reports never name the causal method\n\n**63%** have no usable stack trace from inside the project\n\n**46%** need more than one code block to explain\n\n**5 lines** is the median labeled block, so answers must be precise\n\nFindings\n\nBlock coverage is the share of the labeled root-cause code blocks a diagnosis reaches; block precision is the share of the code it names that is part of the cause.\n\nWithout a pointer to the code, coverage drops sharply.\n\nReports without direct code clues show more run-to-run variation.\n\nMeans average three runs within each incident and weight the 120 incidents equally; whiskers are 95% incident-bootstrap confidence intervals.\n\nUse DistDebug-Bench\n\nRun Mini-SWE-Agent on ZooKeeper-2845 with the paper’s protocol. From a benchmark checkout, the runner prepares an anonymized workspace at the buggy ZooKeeper revision, runs the agent in a sandbox, and the scorer applies the paper’s evaluator. Python 3.10+, Linux with bubblewrap, and a configured model API key are required.\n\n```\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -e \".[agent]\"\n\n# Set OPENAI_API_KEY in your environment.\npython -m distdebug_bench.run \\\n    --config configs/mini-swe-agent/gpt-5.4.yaml \\\n    --bug zookeeper-2845 --runs 1\npython -m distdebug_bench.score \\\n    results/mini-swe-agent-gpt-5.4 --attempts 1\n```\n\n`configs/mini-swe-agent/` has a config for each model in the paper. Runs call your configured model provider.\n\nEach attempt writes `trajectory.json`, `diagnosis.json`, and `diagnosis_scores.json`. To evaluate another agent, have it write the first two.\n\nRun your agent on all 120 incidents, three attempts each, and send its diagnoses and trajectories. We rescore every submission with the official evaluator before listing it on the [leaderboard](leaderboard/).\n\n`diagnosis.json` and the full trajectory for every attempt, plus the model, harness, settings, and token use or cost.\nThe guide covers the local leakage check, packaging a run, and the `diagnosis.json` schema.", "url": "https://wpnews.pro/news/distdebug-bench-can-llm-agents-diagnose-bugs-in-distributed-systems", "canonical_source": "https://distdebug.com/", "published_at": "2026-11-13 05:19:29+00:00", "updated_at": "2026-11-13 05:50:34.273337+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "large-language-models", "ai-tools"], "entities": ["DistDebug-Bench", "GPT-6 Astra", "Claude Opus 5.5", "GPT-5.5", "GPT-5.4", "GPT-5.3-Codex", "Ray", "ZooKeeper"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/distdebug-bench-can-llm-agents-diagnose-bugs-in-distributed-systems", "markdown": "https://wpnews.pro/news/distdebug-bench-can-llm-agents-diagnose-bugs-in-distributed-systems.md", "text": "https://wpnews.pro/news/distdebug-bench-can-llm-agents-diagnose-bugs-in-distributed-systems.txt", "jsonld": "https://wpnews.pro/news/distdebug-bench-can-llm-agents-diagnose-bugs-in-distributed-systems.jsonld"}}