{"slug": "evaluating-glm-5-3-on-hack-the-box-challenges", "title": "Evaluating GLM 5.3 on Hack The Box Challenges", "summary": "Z.ai's GLM 5.3, a retrained version of GLM 5.2, scored 47.2% on the HTB-Challenger Benchmark, solving 10 of 16 Hack The Box challenges with a median cost of $0.49 per challenge, placing it near Muse Spark 1.2 on the cost-vs-score chart. The model performed well on CyberBench for vulnerability discovery but poorly on ExploitBench and ExploitGym, indicating weak exploitation skills, which the benchmark author says limits its usefulness for cybersecurity tasks.", "body_md": "# Evaluating GLM 5.3 on Hack The Box Challenges\n\nThe **GLM 5.3** model was released just last week. The interesting fact is that it’s just a retrained version of **GLM 5.2**, which doesn’t give you much hope for good results. But [Z.ai’s marketing](https://z.ai/blog/glm-5.3) gave us sentences like: *“As we scaled post-training, cyber capability developed faster than we expected.”* and graphs showing better results in CyberGym than both\n\n**Mythos 5** and\n\n[GPT-5.6 Sol](/2026/08/17/solving-htb-challenges-with-openai-gpt-5.6.html).\n\nAlso, the last model I tested from Z.ai was **GLM 5.1**, and it was excellent. So I thought: “OK, maybe, just maybe, this time the marketing isn’t overhyped. Maybe we really can get a new open-weight champion…” Well, spoiler alert: we didn’t.\n\nThis blog post is part of a series of tests for the\n\n[HTB-Challenger Benchmark]The HTB-Challenger Benchmark evaluates LLMs’ ability to find and exploit security vulnerabilities. It tests models against selected Hack The Box challenges of varying difficulty and measures their performance. For more information, visit the[HTB-Challenger Benchmark page]. . See the[benchmark results page]for all results and the[benchmark methodology]to learn how the benchmark is calculated.\n\nGLM 5.3 landed on my **Cost vs. Benchmark Score** chart right next to [Muse Spark 1.2](/2026/08/17/solving-htb-challenges-with-meta-muse-spark-1.2.html) from the company-who-must-not-be-named, and that’s not a good neighborhood. It basically means that its results were mediocre while the price per task was high.\n\nTo explain these results, we don’t need to look too far. If you read the **Emergent Cyber Capability** section in the [GLM 5.3 release notes](https://z.ai/blog/glm-5.3) carefully - and thanks to [this article](https://www.artificialintelligence-news.com/news/zhipu-glm-5-3-benchmarks-explained/) for bringing it to my attention - you’ll see that it scored very well on CyberBench, which measures how good the model is at finding vulnerabilities, but much worse on ExploitBench and ExploitGym, which measure a model’s ability to create working exploits. In other words, **GLM 5.3** may be quite good at finding security issues, but it’s not very good at exploiting them, especially when you give it a time limit.\n\nAnd that explains the poor results in my benchmark very well. I test each model on Hack The Box challenges, and succeeding in each challenge requires not only finding the vulnerability but also exploiting it. On top of that, each model has only a limited number of steps per test.\n\nYou may think that exploitation skills are not important if you want to use the model only for finding vulnerabilities, but I would disagree. The thing is that even when you’re just finding vulnerabilities, you still need a quick way to validate them and rule out false positives. The best way is to do this automatically. Your security testing harness could ask the model to actually try exploiting each discovered vulnerability to verify that it’s real. And if that’s not possible, you need the model to create at least a proof of concept - step-by-step instructions on how to exploit the vulnerability, ideally wrapped in a script or Python code. For both of these verification methods, you need the model to have good exploitation skills.\n\nSo, unfortunately, there’s nothing to see here. **GLM 5.3** is not the next big thing for cybersecurity tasks.\n\n## Cost vs. Benchmark Score\n\nThe highlighted point is this model. Models closer to the upper-left achieve a higher benchmark score at a lower median cost per challenge.\n\n## Overall benchmark results\n\n**Number of challenges:** 16**Number of solved challenges:** 10**Number of false positives:** 0**Runs where the model gave up:** 0**Runs that reached the step or cost limit:** 3**Runs where the model got stuck:** 3**Benchmark score:** 47.2%\n\n| Metric | Per challenge (median) | Total |\n|---|---|---|\n| Model steps | 16.5 | 609 |\n| Model cost | $0.49 | $11.15 |\n| Duration | 00:11:52 | 05:55:55 |\n| Number of input tokens | 0.24M | 18.36M |\n| Number of output tokens | 0.02M | 1.13M |\nNumber of `read_file` tool calls |\n0.5 | 32 |\nNumber of `write_file` tool calls |\n1.0 | 37 |\nNumber of `execute_command` tool calls |\n18.5 | 653 |\nNumber of `web_search` tool calls |\n0.0 | 3 |\n\n## Results by challenge difficulty\n\nAll resource-usage metrics are medians per challenge.\n\n| Metric | Very Easy | Easy | Medium | Hard |\n|---|---|---|---|---|\nResults |\n||||\n| Number of challenges | 4 | 4 | 4 | 4 |\n| Number of solved challenges | 4 | 3 | 2 | 1 |\n| Number of false positives | 0 | 0 | 0 | 0 |\n| Runs where the model gave up | 0 | 0 | 0 | 0 |\n| Runs that reached the step or cost limit | 0 | 1 | 1 | 1 |\n| Runs where the model got stuck | 0 | 0 | 1 | 2 |\n| Benchmark score | 90.9% | 69.0% | 48.8% | 24.2% |\nMedian per challenge |\n||||\n| Model steps | 19 | 41 | 13.5 | 20 |\n| Model cost | $0.19 | $0.63 | $0.46 | $0.88 |\n| Duration | 00:05:47 | 00:39:48 | 00:14:35 | 00:23:33 |\n| Number of input tokens | 0.16M | 0.43M | 0.22M | 0.35M |\n| Number of output tokens | 0.02M | 0.10M | 0.03M | 0.08M |\nNumber of `read_file` tool calls |\n0.5 | 3.0 | 0.0 | 1.0 |\nNumber of `write_file` tool calls |\n0.0 | 6.0 | 1.0 | 0.5 |\nNumber of `execute_command` tool calls |\n17.0 | 29.5 | 16.0 | 28.5 |\nNumber of `web_search` tool calls |\n0.0 | 0.5 | 0.0 | 0.0 |", "url": "https://wpnews.pro/news/evaluating-glm-5-3-on-hack-the-box-challenges", "canonical_source": "https://theaq.blog/2026/08/21/evaluating-z-ai-glm-5.3-on-hack-the-box-challenges.html", "published_at": "2026-08-21 07:07:08+00:00", "updated_at": "2026-08-21 10:43:02.032047+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-products"], "entities": ["Z.ai", "GLM 5.3", "GLM 5.2", "Mythos 5", "GPT-5.6 Sol", "Muse Spark 1.2", "HTB-Challenger Benchmark", "Hack The Box"], "alternates": {"html": "https://wpnews.pro/news/evaluating-glm-5-3-on-hack-the-box-challenges", "markdown": "https://wpnews.pro/news/evaluating-glm-5-3-on-hack-the-box-challenges.md", "text": "https://wpnews.pro/news/evaluating-glm-5-3-on-hack-the-box-challenges.txt", "jsonld": "https://wpnews.pro/news/evaluating-glm-5-3-on-hack-the-box-challenges.jsonld"}}