Bluebox at All Things Open 2026 #
Come meet the Bluebox team at All Things Open in Raleigh, NC, Monday and Tuesday, October 19th and 20th at the Raleigh Convention Center. It is the largest
Bluebox runs on Anthropic's Sonnet 5.5 model since October 1. In our evals with SREGym, Sonnet 5.5 diagnosed more problems correctly than Sonnet 5 (86.7% vs 73.3%), used 3.3x fewer tokens, finished 3.7x faster, and cost 4.3x less per investigation.
We evaluate Bluebox continuously, against our own benchmarks and against public ones. SREGym is a public, open benchmark of fault scenarios. Each scenario has a target application on Kubernetes and an injected fault, with a known ground truth and an LLM-as-a-judge that scores the investigation.
We ran 20 SREGym problems, 3 attempts each. We ran the suite once with Sonnet 5 and once with Sonnet 5.5. Nothing else changed: the same graders, ground truth, environment, and 1500-second timeout.
| Sonnet 5 | Sonnet 5.5 | Change | |
|---|---|---|---|
| Correct diagnoses | 73.3% | 86.7% | +13.4 points |
| Tokens | ~5.57M | ~1.69M | 3.3x fewer |
| Time* | 607 s | 162 s | 3.7x faster |
| Cost | $2.56 | $0.60 | 4.3x less |
*Agent execution time. It excludes pod startup.
Sonnet 5 made about twice as many model calls per attempt (47.5 vs 20.9), and each call carried more tokens. Sonnet 5.5 got to an answer with less work.
Investigations in Bluebox are now faster and more accurate. Ask Bluebox to investigate an incident and you'll typically get a diagnosis in under three minutes instead of ten, with more correct root causes along the way. It also costs far less to run, so you can ask more questions and investigate more incidents on the same budget.
To note, we cannot compare these results with the public SREGym leaderboard. We run a private fork of SREGym with changes:
The results are valid for one comparison: Sonnet 5 vs Sonnet 5.5 on Bluebox, but we want to keep in mind that this comparison is early:
We'll re-run the same 20 problems with richer Kubernetes data. This brings our setup closer to the public SREGym leaderboard, though our results still won't be directly comparable. We will cover the full methodology in a later post.
In this post: the questions Bluebox answers about a release, and the prompts to ask them. Then how we run them automatically in our own GitHub Actions pipeline,
The Bluebox MCP server is now available to everyone. Connect Claude Code, Cursor, Visual Studio Code, Kiro, Codex, or any other MCP client, and your coding agent can