Ask HN: Anyone interested in building a harness-only benchmark?

A developer and maintainer of the coding agent Dirac proposed building a community-driven benchmark to evaluate LLM harnesses, aiming to create a leaderboard of harness performance across diverse real-world tasks grouped by underlying models and reasoning efforts. The proposal, posted on Hacker News, invites contributors to decide task criteria and measurements, with the initiator pledging not to influence the benchmark to avoid conflicts of interest.

There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance multiple axis on a set of diverse real world tasks 1 , grouped by underlying models and reasoning efforts. Anyone can contribute results. The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person. If there is sufficient interest, I will create a discord. Disclosure: I am the maintainer of a coding agent called Dirac https://github.com/dirac-run/dirac so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen. 1 Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, preferably from an opensource repo. Comments URL: https://news.ycombinator.com/item?id=49181014 https://news.ycombinator.com/item?id=49181014 Points: 1 Comments: 2