Gemini 3.7 Flash, Grok 4.6, GLM-5.3 and DeepSeek V4 Pro joined the frontier

In mid-August, Gemini 3.7 Flash, Grok 4.6, GLM-5.3, and DeepSeek V4 Pro 0813 joined the frontier of LLM releases, and Quesma's Baba Is Bench rerun shows Gemini 3.7 Flash solving every trial at over 20x lower cost than its predecessor, while Grok 4.6 solved all but one attempt and DeepSeek V4 Pro 0813 improved from 3/8 to 7/8 levels at 4x lower cost. GLM-5.3 matched GLM-5.2's results, and Qwen3.8 Max improved over Qwen3.7 Max but still fell short of completing all levels.

This mid August is as hot as July https://quesma.com/blog/baba-kimi-k3-opus-5/ when it comes to LLM releases: Grok 4.6 https://openrouter.ai/x-ai/grok-4.6 , Gemini 3.7 Flash https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash/ , DeepSeek V4 Pro 0813 https://api-docs.deepseek.com/news/news260813/ as well as open-weight models: Qwen3.8 Max https://qwen.ai/blog?id=qwen3.8 , Qwen3.8 27B https://huggingface.co/Qwen/Qwen3.8-27B , and GLM-5.3 https://z.ai/blog/glm-5.3 . Since these models are smart, I decided to rerun the Baba Is Bench https://quesma.com/benchmarks/babaisbench/ , to see how the models fare on a puzzle game. Even though the game is popular, we check for spoilers - and to our surprise, there are no signs of models knowing solutions ahead - unlike in SWE-bench Verified https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ . Results are stunning for Grok, Gemini Flash or DeepSeek. Don’t get deceived by minor version changes - they all give qualitatively different results For both GLM and Qwen the progress was more gradual. As previously, we do it in two rounds. First, asking to play the initial stage the Intro , then for those who pass at least 7/8 levels, also the next one the Lake . Stage 0: The Intro We show the models we analyze, against dimmed previous ones. | Model | 00 baba is you | 01 where do i go? | 02 now what is this? | 03 out of reach | 04 still out of reach | 05 volcano | 06 off limits | 07 grass yard | pass@1 ↓ | pass@3 ↓ | Turns ↓ | Output tokens ↓ | Total cost ↑ | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | Gemini 3.7 Flash | 100% | 100% | 53 | 22k | $5.38 | |||||||| | Claude Opus 5 | 100% | 100% | 7 | 23k | $17.15 | |||||||| | GPT-5.5 | 100% | 100% | 8 | 15k | $13.62 | |||||||| | Claude Fable 5 | 100% | 100% | 6 | 29k | $40.98 | |||||||| | GPT-5.6 Sol | 100% | 100% | 11 | 10k | $11.91 | |||||||| | Grok 4.6 | 96% | 100% | 25 | 52k | $14.01 | |||||||| | GLM-5.3 | 96% | 100% | 13 | 47k | $6.20 | |||||||| | Kimi K3 | 96% | 100% | 9 | 29k | $12.56 | |||||||| | Claude Opus 4.8 | 96% | 100% | 35 | 46k | $63.14 | |||||||| | GLM-5.2 | 96% | 100% | 12 | 75k | $6.20 | |||||||| | Gemini 3.1 Pro | 92% | 100% | 8 | 38k | $12.41 | |||||||| | Gemini 3.6 Flash | 88% | 100% | 92 | 82k | $124.31 | |||||||| | GPT-5.6 Terra | 88% | 100% | 41 | 55k | $34.28 | |||||||| | DeepSeek V4 Pro 0813 | 75% | 88% | 18 | 130k | $2.94 | |||||||| | DeepSeek V4 Flash | 75% | 100% | 55 | 190k | $2.82 | |||||||| | Grok 4.5 | 75% | 88% | 86 | 68k | $50.66 | |||||||| | DeepSeek V4 Flash 0731 | 75% | 88% | 22 | 156k | $1.16 | |||||||| | Claude Sonnet 5 | 67% | 88% | 22 | 151k | $42.21 | |||||||| | GPT-5.6 Luna | 63% | 88% | 118 | 129k | $59.32 | |||||||| | Qwen3.8 Max | 58% | 75% | 9 | 66k | $10.75 | |||||||| | Qwen3.6 27B | 29% | 38% | 24 | 82k | $5.59 | |||||||| | DeepSeek V4 Pro | 29% | 38% | 37 | 183k | $11.49 | |||||||| | Qwen3.7 Max | 25% | 38% | 21 | 132k | $16.98 | |||||||| | Qwen3.8 27B | 21% | 38% | 9 | 134k | $9.07 | The results are nothing short of stunning, as both results went up and prices went down significantly. - Gemini 3.7 Flash now solves every trial, and is over 20x cheaper than its predecessor, Gemini 3.6 Flash. - Grok 4.6 solved all but one attempt vs Grok 4.5 that missed 6 attempts and one level , while being over 3x cheaper. - DeepSeek V4 Pro 0813 solved 7/8 levels vs its previous version that solved only 3/8 - and while being 4x cheaper. With the GLM and Qwen family there is less success: - GLM-5.3 got exactly the same result as GLM-5.2. - Qwen3.8 Max is better than Qwen3.7 Max way more attempts solved , but still far from completing all levels. - Qwen3.8 27B has worse results than Qwen3.6 27B - though, all within experimental noise. I was surprised that it didn’t do better. Especially as Qwen3.8 27B scores 52 on Artificial Analysis https://news.ycombinator.com/item?id=49334544 . From what I’ve seen in the transcripts, it tried to solve in one go, without actually exploring the board. It matches Simon Willison’s experience that Qwen3.8 27B defaults to wildly overthinking things https://simonwillison.net/2026/Aug/16/qwen-38-27b/ . While Baba Is You clearly needs thinking, it is a balance of thinking and testing. Compare and contrast with previous Grok and Gemini Flash models that were doing a lot and thinking too little. It is interesting to look at the chart of agent turns and price. Grok and Gemini Flash moved from trigger-happy models, to more moderate ones, resulting in a significantly reduced cost. Stage 1: The Lake For models that passed at least 7 out of 8 intro levels, we run them on the next stage, the Lake . Only one attempt per level, as it is costly. Or at least - it used to be costly with the previous models, as now it is getting significantly better. | Model | Harness | Solved | Total time | Output tokens | Total cost | ||||||||||||||| |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 14 | 2h 36m | 10.7M | $53.25 | ||||||||||||||||| | 14 | 6h 46m | 16.3M | $61.96 | ||||||||||||||||| | 14 | 9h 11m | 1023.1M | $55.76 | ||||||||||||||||| | 14 | 9h 16m | 16.4M | $31.42 | ||||||||||||||||| | 13 | 8h 57m | 50.1M | $102.08 | ||||||||||||||||| | 13 | 20h 16m | 38.6M | $46.20 | ||||||||||||||||| | 13 | 22h 7m | 75.6M | $8.48 | ||||||||||||||||| | 12 | 9h 7m | 49.9M | $107.76 | ||||||||||||||||| | 12 | 24h 57m | 9.6M | $16.85 | ||||||||||||||||| | 11 | 23h 41m | 62.9M | $37.51 | ||||||||||||||||| | 11 | 29h 34m | 129.5M | $78.15 | ||||||||||||||||| | 8 | 5h 58m | 41.8M | $58.17 | ||||||||||||||||| | 6 | 7h 46m | 1118.1M | $269.48 | ||||||||||||||||| | 6 | 24h 11m | 92.8M | $4.31 | Time First, let’s look at the wall time, for models run via APIs. While it might change with providers, or model bandwidth at a given time, it gives a ballpark estimate of effective model speed. In everyday software engineering it is what we care for, anyway. And wow, Gemini 3.7 Flash is fast - not only per token which is widely cited but largely irrelevant , but per problem solved. Here, for most levels, only Claude Fable 5 is faster. Grok 4.6, while 2x slower than GPT-5.6 Sol, still manages to solve 13 of 15 levels GPT-5.6 Sol solves 14 . It’s a promotion to the league of top models. DeepSeek V4 Pro 0813 might be the slowest of the top models - but it is the first open-weight model that finally reached the same top league. It is way beyond GLM-5.2 or Kimi K3 results. GLM-5.3 was almost the same as GLM-5.2, to the point that two lines overlap; though, it solved one more level. Cost Now let’s look at costs. It’s beautiful. Grok 4.6 is cheaper than Fable 5, but more expensive than GPT-5.6 Sol or Opus 5. While Gemini 3.6 Flash was ridiculously expensive https://quesma.com/blog/baba-kimi-k3-opus-5/ , Gemini 3.7 Flash is one of the cheapest models. At 12 levels, it is twice as cheap as GPT-5.6 Sol, the previous cheapest model. At 13 tasks it is still twice as cheap. Only vs DeepSeek Pro is it slightly more expensive. What happens with DeepSeek V4 Pro 0813 is a pure wonder. Not only is it open-weight, but also the cheapest of all. GLM-5.3 was 2x cheaper than its predecessor. Some lessons about harnesses In previous experiments we used various harnesses. Here we used Harbor https://www.harborframework.com/ -default Terminus-2, except DeepSeek, which we ran on Claude Code. While there are Grok Build https://x.ai/build and DeepSeek Harness https://news.ycombinator.com/item?id=49285244 , results were slightly worse Grok or vastly inferior DeepSeek . That said, these are newer tools, and likely to get improved in the future. If there is interest, we’re happy to share more insight on running with various harnesses — they influence both cost and results. Conclusion Progress is fast - with more players joining the frontier game, and open-weight models reaching frontier-level results at a much lower price. And I am looking forward to seeing more models in the coming months. And to the first local model solving the Lake. As my colleague Piotr Grabowski noticed: “Second-tier labs caught up with the frontier” matches my experience: last few days I’ve been using Grok 4.6 or GLM-5.3 for easy/medium difficulty tasks and couldn’t see major difference between them and Opus 5/GPT-5.6 Sol for harder tasks yes, there’s still gap . And what is your experience?