Z.ai’s GLM-5.3 just dropped today and shipped with cyber capability claims:
As we scaled post-training, cyber capability developed faster than we expected. GLM-5.3 is state of the art on CyberGym for vulnerability discovery, and its gains are largest further up the exploitation chain, where it more than doubles GLM-5.2 on exploitation benchmarks.
Source: Z.ai: GLM-5.3: Frontier Coding with Emergent Cyber Capabilities We ran GLM-5.3 and the Grok models that dropped earlier this week against the same vulnerability detection benchmark we’ve used on every model this year, measuring real vulnerabilities in real open-source code, scored on precision, recall, and cost.
tl;dr: these new models are catching up to frontier model performance, but aren’t there yet; they show promising cost effectiveness.
What we’re testing #
We use the same benchmark we’ve run on every model so far this year: Insecure Direct Object Reference (IDOR) detection against real, open-source codebases with established ground truth. IDOR is a good stress test because there’s no dangerous function to pattern-match against, only a missing authorization check. A model has to actually reason about who’s allowed to access what.
Our benchmarking suite gives every model the same guided prompt: what an IDOR is, what to look for, what to skip, and a suggested investigation strategy. We score three things:
Precision: of everything flagged, what fraction is a real IDOR** Recall**: of every real IDOR in the dataset, what fraction got found** F1**: the harmonic mean of the two, which we use to rank overall performance** Cost**: the total model cost to run the suite divided by the number of confirmed true positive vulnerabilities found
The results #
| | | | | Claude Opus 5 | 65.6% | 74.1% | 58.8% | $0.44 | GPT-5.6 Luna | 48.0% | 78.8% | 34.5% | $0.08 | Kimi K3 | 36.9% | 76.3% | 24.4% | $0.33 | Claude Opus 4.7 | 36.4% | 80.0% | 23.5% | $0.66 | | | | | | | | | | | | | | | | GLM 5.2 | 26.8% | 82.6% | 16.0% | $0.23 | Claude Opus 4.8 | 23.6% | 68.0% | 14.3% | $1.04 | | | | | | DeepSeek V4 Flash | 12.5% | 88.9% | 6.7% | $0.02 |
Grok 4.6: a cheaper way to get Opus 4.7-level performance #
None of the three Grok 4.6 variants beat the frontier leaders, but they do perform similarly to Kimi K3 and older Claude Opus models at a fraction of the price.
Grok 4.6 Exacto’s 35.5% F1 is very close to Kimi K3’s 36.9%, at $0.20 per true positive versus $0.33 for Kimi K3, roughly 60% of the cost for comparable results. Against Claude Opus 4.7 (36.4% F1, $0.66 per true positive), Grok 4.6 Exacto gets within a point of the same F1 score at less than a third of the cost.
If your team already has Opus 4.7 in a pipeline and F1 in the mid-30s is good enough for the task, Grok 4.6 Exacto gets you there for meaningfully less.
GLM-5.3: matching Opus 4.8 for a fraction of the price #
GLM-5.3 posted 23.8% F1, essentially matching Claude Opus 4.8's 23.6%, at $0.15 per true positive against Opus 4.8's $1.04, roughly a seventh of the cost. Same tier of detection quality, for a fraction of the price.
We want to flag one thing before anyone reads too much into that number: on this run, GLM-5.3 actually scored lower than its own predecessor, GLM-5.2 (23.8% vs. 26.8% F1). IDOR detection is noisy from run to run on any model, and we haven’t yet confirmed whether that gap reflects an actual step backward or ordinary variance. We’re re-running both models with multiple trials to find out, and we’ll update this post with what we learn. Stay tuned.
Recall is still a challenge #
Looking at the benchmark results across most of these entries, we see that precision stays fairly high (roughly 70-90% across the board) while recall drops off hard, down to 13-17% for GLM-5.3 and the Grok 4.6 variants, and 6.7% for DeepSeek V4 Flash. These models are usually right when they flag something, but they just don’t find most of what’s actually there. That’s an important consideration if you’re using one of these models for something closer to comprehensive coverage than spot-checking.
Where this leaves things #
While Grok 4.6 and GLM-5.3 don’t unseat Claude Opus 5 or GPT-5.6 Luna, the two models leading this benchmark right now, they do compress the price you pay to fall in the same range as last generation’s frontier models. Grok 4.6 Exacto gets Kimi K3 and Opus 4.7-level detection for 30-60% of the cost. GLM-5.3 gets Opus 4.8-level detection for about a seventh of the cost, but with a variance question we’re still chasing down. Open-weight and lower-cost models keep getting more competitive on price. On raw detection quality against the current frontier, they’re still not there. We’ll keep running this benchmark on every model that ships and publish the numbers as they come in.
Lots of love,
The Semgrep Team