Solving Hack The Box Challenges with DeepSeek V4 Flash 0731

I tested the previous version of DeepSeek V4 Flash, now called DeepSeek V4 Flash 0423, with Strix back in April, and I absolutely fell in love with it. It delivered great results at a very low price and became my go-to model for most tasks that didn’t require the capabilities of frontier models.

That’s why I was looking forward to testing the latest, improved version on my HTB-Challenger BenchmarkThe HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the HTB-Challenger Benchmark page. . But the results… Oh dear, oh dear, oh dear. Where do I even start?

This blog post is part of a series of tests for the

[HTB-Challenger Benchmark]The HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the[HTB-Challenger Benchmark page]. . See the[benchmark results page]for all results and the[benchmark methodology]to learn how the benchmark is calculated.

Okay, let’s start with the results: they are bad. It solved just 2 out of 16 HTB challenges and scored 8.2% on my benchmark. But that’s not the worst or most puzzling thing about this model.

What worried me more was the number of false positives: it reported incorrect flags in 9 challenges. Sometimes these flags were completely made up, like HTB{...}

or HTB{dummy} . In other cases, it submitted a fake flag from the attached source code or a random string that seemed to come out of nowhere. To put this into perspective, not one of the other models I’ve tested so far has reported even a single incorrect flag.

Another worrying thing was how badly this model used tools. Multiple times, I saw it call the execute_command

tool with an empty command or with random text instead of a command. Similarly, it called the write_file

tool several times with no file content. Again, none of the other models I’ve tested did this even once.

Yes, the model was cheap. The median test cost was a fraction of a cent, but in terms of results, I got what I paid for.

So what went wrong? #

I spent quite a lot of time wondering why the results were so bad, especially after the previous version did so well in my Strix test. The best explanation I could come up with is this:

During my HTB-Challenger testing, I don’t provide the model with task-specific guidance in the form of skills or special instructions for each security domain. I also don’t tell it how to approach the job or validate its findings to avoid false positives. This is deliberate: the goal is to test the model’s built-in offensive-security capabilities.

This is very different from how tools like Strix work. They provide the model with skills and detailed instructions that guide it through the different phases of testing and help it validate its findings. My best guess is that this extra scaffolding made the difference: DeepSeek V4 Flash can perform well within a highly structured agent, but it struggles badly when it has to plan its own approach, use tools reliably, and validate findings with much less guidance.

Cost vs. Benchmark Score #

The highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.

Overall benchmark results #

Number of challenges: 16Number of solved challenges: 2Number of false positives: 9Runs where the model gave up: 3Runs that reached the step or cost limit: 1Runs where the model got stuck: 1Benchmark score: 8.2%

Results by challenge difficulty #

All resource-usage metrics are medians per challenge.