So far, I have used my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. to test how well more than 15 LLMs can solve Hack The Box challenges, and two have become personal favorites. The first is GPT-5.6 Luna. Its overall score was not high, but it was solid on simpler tasks and, most importantly, it was extremely cheap. My other favorite sits at the opposite end of the performance ranking and is also from OpenAI: GPT-5.6 Sol. It is the best-performing model I have tested so far, but that performance comes at a price. Its overall median cost per challenge was about 16 times higher than Luna’s, although the ratio varied substantially by difficulty. So I wondered: wouldn’t it be great if you could use both models in one workflow and reduce the cost by automatically switching between them when a task becomes too difficult for Luna? You could start with Luna and bring in Sol only when the task exceeds Luna’s capabilities. In theory, this should deliver almost Sol-level performance at a much lower cost. After a few dead ends, I found a solution that worked for my use case and that I could use to solve Hack The Box challenges. In this post, I briefly explain what I did and what the results were.
OpenAI just dropped GPT-5.6 Sol and it changes the vision game