Harbor-Index proves that most agent benchmarks are actually too

Harbor-Index, a curated subset of 82 high-difficulty tasks from 29 benchmarks, found that no model-harness configuration exceeded a 30% pass rate, with GPT-5.5 (using Codex) reaching a ceiling of 28.0%. The evaluation, which ran 8 models across 54 benchmarks, demonstrates that current AI agents are far from reliable in real-world scenarios. The Harbor Adapters approach standardizes benchmark integration and is open-sourced to enable standardized testing across LLM agents.

Harbor-Index proves that most agent benchmarks are actually too The scale of the testing here is what actually matters. They ran 8 different models across 54 benchmarks, comparing the results when using the "Terminus-2" setup versus three different native harnesses. This is a critical detail because it highlights how much the "harness" the wrapper that executes the agent's code affects the final score. If you're seeing a model perform well on a leaderboard, you have to ask if it's because the model is smart or if the harness is just doing the heavy lifting for it. The Harbor-Index benchmark results The most useful output here is the Harbor-Index, which is a curated subset of 82 high-difficulty tasks pulled from 29 different benchmarks. They basically filtered out the "easy" stuff using an audit-and-fix loop to ensure the tasks actually challenge the models. The performance numbers are a reality check for the current state of AI agents: - Maximum Pass Rate: No model-harness configuration managed to break a 30% success rate. - Top Performer: GPT-5.5 using Codex hit a ceiling of 28.0%. When the strongest model in the world can't even clear 30% on a curated set of "hard" tasks, it shows that we are still very far from reliable, autonomous agentic workflows in real-world scenarios. How the infrastructure works For anyone trying to build a custom evaluation pipeline from scratch, the Harbor Adapters approach is the way to go. Instead of writing a new integration for every single benchmark, they've standardized the interface. If you are looking to implement a similar AI workflow for testing, the logic follows this pattern: 1. Adapter Layer: Create a wrapper that maps the agent's output e.g., a tool call or a bash command to the specific requirements of the benchmark environment. 2. Parity Testing: Run the same prompt through both the native harness and the adapter to ensure the results are identical. 3. Difficulty Filtering: Use a "difficulty filter" to remove tasks where the pass rate is too high, leaving only the edge cases that actually test reasoning. This setup allows for a much deeper dive into failure modes. Instead of just seeing a "Fail" grade, you can analyze whether the agent failed because of a logic error or because the adapter failed to execute the command correctly. The fact that they are open-sourcing these adapters means we can finally stop relying on the "cherry-picked" results often found in model release notes and start running a real-world, standardized battery of tests across different LLM agents. Next Claude Code makes it way too easy to accidentally create the → /en/threads/8939/