The scale of the testing here is what actually matters. They ran 8 different models across 54 benchmarks, comparing the results when using the "Terminus-2" setup versus three different native harnesses. This is a critical detail because it highlights how much the "harness" (the wrapper that executes the agent's code) affects the final score. If you're seeing a model perform well on a leaderboard, you have to ask if it's because the model is smart or if the harness is just doing the heavy lifting for it.
The Harbor-Index benchmark results #
The most useful output here is the Harbor-Index, which is a curated subset of 82 high-difficulty tasks pulled from 29 different benchmarks. They basically filtered out the "easy" stuff using an audit-and-fix loop to ensure the tasks actually challenge the models.
The performance numbers are a reality check for the current state of AI agents:
-
Maximum Pass Rate: No model-harness configuration managed to break a 30% success rate.
-
Top Performer: GPT-5.5 (using Codex) hit a ceiling of 28.0%. When the strongest model in the world can't even clear 30% on a curated set of "hard" tasks, it shows that we are still very far from reliable, autonomous agentic workflows in real-world scenarios.
How the infrastructure works #
For anyone trying to build a custom evaluation pipeline from scratch, the Harbor Adapters approach is the way to go. Instead of writing a new integration for every single benchmark, they've standardized the interface.
If you are looking to implement a similar AI workflow for testing, the logic follows this pattern:
-
Adapter Layer: Create a wrapper that maps the agent's output (e.g., a tool call or a bash command) to the specific requirements of the benchmark environment.
-
Parity Testing: Run the same prompt through both the native harness and the adapter to ensure the results are identical.
-
Difficulty Filtering: Use a "difficulty filter" to remove tasks where the pass rate is too high, leaving only the edge cases that actually test reasoning.
This setup allows for a much deeper dive into failure modes. Instead of just seeing a "Fail" grade, you can analyze whether the agent failed because of a logic error or because the adapter failed to execute the command correctly.
The fact that they are open-sourcing these adapters means we can finally stop relying on the "cherry-picked" results often found in model release notes and start running a real-world, standardized battery of tests across different LLM agents.
Next Claude Code makes it way too easy to accidentally create the →