# Harbor-Index proves that most agent benchmarks are actually too

> Source: <https://promptcube3.com/en/threads/8989/>
> Published: 2026-09-07 16:35:28+00:00

# Harbor-Index proves that most agent benchmarks are actually too

The scale of the testing here is what actually matters. They ran 8 different models across 54 benchmarks, comparing the results when using the "Terminus-2" setup versus three different native harnesses. This is a critical detail because it highlights how much the "harness" (the wrapper that executes the agent's code) affects the final score. If you're seeing a model perform well on a leaderboard, you have to ask if it's because the model is smart or if the harness is just doing the heavy lifting for it.

## The Harbor-Index benchmark results

The most useful output here is the Harbor-Index, which is a curated subset of 82 high-difficulty tasks pulled from 29 different benchmarks. They basically filtered out the "easy" stuff using an audit-and-fix loop to ensure the tasks actually challenge the models.

The performance numbers are a reality check for the current state of AI agents:

- **Maximum Pass Rate:** No model-harness configuration managed to break a 30% success rate.
- **Top Performer:** GPT-5.5 (using Codex) hit a ceiling of 28.0%.

When the strongest model in the world can't even clear 30% on a curated set of "hard" tasks, it shows that we are still very far from reliable, autonomous agentic workflows in real-world scenarios.

## How the infrastructure works

For anyone trying to build a custom evaluation pipeline from scratch, the Harbor Adapters approach is the way to go. Instead of writing a new integration for every single benchmark, they've standardized the interface.

If you are looking to implement a similar AI workflow for testing, the logic follows this pattern:

1. **Adapter Layer:** Create a wrapper that maps the agent's output (e.g., a tool call or a bash command) to the specific requirements of the benchmark environment.

2. **Parity Testing:** Run the same prompt through both the native harness and the adapter to ensure the results are identical.

3. **Difficulty Filtering:** Use a "difficulty filter" to remove tasks where the pass rate is too high, leaving only the edge cases that actually test reasoning.

This setup allows for a much deeper dive into failure modes. Instead of just seeing a "Fail" grade, you can analyze whether the agent failed because of a logic error or because the adapter failed to execute the command correctly.

The fact that they are open-sourcing these adapters means we can finally stop relying on the "cherry-picked" results often found in model release notes and start running a real-world, standardized battery of tests across different LLM agents.

[Next Claude Code makes it way too easy to accidentally create the →](/en/threads/8939/)