Introducing HTB-Challenger: Benchmarking LLM models on Hack The Box Challenges

Choosing an LLM model for offensive security work is difficult. Public benchmarks can help compare model capabilities, but they rarely show how much it costs to complete a task from start to finish. The price per million tokens does not tell the whole story: different models may require vastly different numbers of tokens and tool calls before they solve a problem - or fail to solve it. On top of that, as Winston Churchill allegedly said, “The only statistics you can trust are those you falsified yourself.” I previously spent a lot of time evaluating new LLMs on offensive security tasks with Strix. I am still enthusiastic about Strix, and it remains my go-to tool for penetration testing, but it is no longer the best fit for the repeated model comparisons I want to run. Strix performs autonomous security testing against complex applications, so a single run can require many steps, tool calls, and tokens. That makes repeated testing expensive. Strix is also evolving rapidly and improving with each version. Results produced with different versions are therefore not directly comparable: changes in the agent could be mistaken for changes in model performance. A fair comparison would require pinning one Strix version and rerunning every model against it. I therefore revived an older idea and turned it into my new testing approach: the HTB-Challenger Benchmark.