{"slug": "gemini-3-7-flash-grok-4-6-glm-5-3-and-deepseek-v4-pro-joined-the-frontier", "title": "Gemini 3.7 Flash, Grok 4.6, GLM-5.3 and DeepSeek V4 Pro joined the frontier", "summary": "In mid-August, Gemini 3.7 Flash, Grok 4.6, GLM-5.3, and DeepSeek V4 Pro 0813 joined the frontier of LLM releases, and Quesma's Baba Is Bench rerun shows Gemini 3.7 Flash solving every trial at over 20x lower cost than its predecessor, while Grok 4.6 solved all but one attempt and DeepSeek V4 Pro 0813 improved from 3/8 to 7/8 levels at 4x lower cost. GLM-5.3 matched GLM-5.2's results, and Qwen3.8 Max improved over Qwen3.7 Max but still fell short of completing all levels.", "body_md": "This mid August is as [hot as July](https://quesma.com/blog/baba-kimi-k3-opus-5/) when it comes to LLM releases: [Grok 4.6](https://openrouter.ai/x-ai/grok-4.6), [Gemini 3.7 Flash](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash/), [DeepSeek V4 Pro 0813](https://api-docs.deepseek.com/news/news260813/) as well as open-weight models: [Qwen3.8 Max](https://qwen.ai/blog?id=qwen3.8), [Qwen3.8 27B](https://huggingface.co/Qwen/Qwen3.8-27B), and [GLM-5.3](https://z.ai/blog/glm-5.3).\n\nSince these models are smart, I decided to rerun the [Baba Is Bench](https://quesma.com/benchmarks/babaisbench/), to see how the models fare on a puzzle game. Even though the game is popular, we check for spoilers - and to our surprise, there are no signs of models knowing solutions ahead - unlike in [SWE-bench Verified](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/).\n\nResults are stunning for Grok, Gemini Flash or DeepSeek. Don’t get deceived by minor version changes - they all give qualitatively different results! For both GLM and Qwen the progress was more gradual.\n\nAs previously, we do it in two rounds. First, asking to play the initial stage **the Intro**, then for those who pass at least 7/8 levels, also the next one **the Lake**.\n\n### Stage 0: The Intro\n\nWe show the models we analyze, against dimmed previous ones.\n\n| Model | 00 baba is you | 01 where do i go? | 02 now what is this? | 03 out of reach | 04 still out of reach | 05 volcano | 06 off limits | 07 grass yard | pass@1 ↓ | pass@3 ↓ | Turns ↓ | Output tokens ↓ | Total cost ↑ |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| Gemini 3.7 Flash | 100% | 100% | 53 | 22k | $5.38 | ||||||||\n| Claude Opus 5 | 100% | 100% | 7 | 23k | $17.15 | ||||||||\n| GPT-5.5 | 100% | 100% | 8 | 15k | $13.62 | ||||||||\n| Claude Fable 5 | 100% | 100% | 6 | 29k | $40.98 | ||||||||\n| GPT-5.6 Sol | 100% | 100% | 11 | 10k | $11.91 | ||||||||\n| Grok 4.6 | 96% | 100% | 25 | 52k | $14.01 | ||||||||\n| GLM-5.3 | 96% | 100% | 13 | 47k | $6.20 | ||||||||\n| Kimi K3 | 96% | 100% | 9 | 29k | $12.56 | ||||||||\n| Claude Opus 4.8 | 96% | 100% | 35 | 46k | $63.14 | ||||||||\n| GLM-5.2 | 96% | 100% | 12 | 75k | $6.20 | ||||||||\n| Gemini 3.1 Pro | 92% | 100% | 8 | 38k | $12.41 | ||||||||\n| Gemini 3.6 Flash | 88% | 100% | 92 | 82k | $124.31 | ||||||||\n| GPT-5.6 Terra | 88% | 100% | 41 | 55k | $34.28 | ||||||||\n| DeepSeek V4 Pro 0813 | 75% | 88% | 18 | 130k | $2.94 | ||||||||\n| DeepSeek V4 Flash | 75% | 100% | 55 | 190k | $2.82 | ||||||||\n| Grok 4.5 | 75% | 88% | 86 | 68k | $50.66 | ||||||||\n| DeepSeek V4 Flash 0731 | 75% | 88% | 22 | 156k | $1.16 | ||||||||\n| Claude Sonnet 5 | 67% | 88% | 22 | 151k | $42.21 | ||||||||\n| GPT-5.6 Luna | 63% | 88% | 118 | 129k | $59.32 | ||||||||\n| Qwen3.8 Max | 58% | 75% | 9 | 66k | $10.75 | ||||||||\n| Qwen3.6 27B | 29% | 38% | 24 | 82k | $5.59 | ||||||||\n| DeepSeek V4 Pro | 29% | 38% | 37 | 183k | $11.49 | ||||||||\n| Qwen3.7 Max | 25% | 38% | 21 | 132k | $16.98 | ||||||||\n| Qwen3.8 27B | 21% | 38% | 9 | 134k | $9.07 |\n\nThe results are nothing short of stunning, as both results went up and prices went down significantly.\n\n- Gemini 3.7 Flash now solves every trial, and is over 20x cheaper than its predecessor, Gemini 3.6 Flash.\n- Grok 4.6 solved all but one attempt (vs Grok 4.5 that missed 6 attempts and one level), while being over 3x cheaper.\n- DeepSeek V4 Pro 0813 solved 7/8 levels vs its previous version that solved only 3/8 - and while being 4x cheaper.\n\nWith the GLM and Qwen family there is less success:\n\n- GLM-5.3 got exactly the same result as GLM-5.2.\n- Qwen3.8 Max is better than Qwen3.7 Max (way more attempts solved), but still far from completing all levels.\n- Qwen3.8 27B has worse results than Qwen3.6 27B - though, all within experimental noise.\n\nI was surprised that it didn’t do better. Especially as [Qwen3.8 27B scores 52 on Artificial Analysis](https://news.ycombinator.com/item?id=49334544). From what I’ve seen in the transcripts, it tried to solve in one go, without actually exploring the board. It matches Simon Willison’s experience that [Qwen3.8 27B defaults to wildly overthinking things](https://simonwillison.net/2026/Aug/16/qwen-38-27b/). While Baba Is You clearly needs thinking, it is a balance of thinking and testing. Compare and contrast with previous Grok and Gemini Flash models that were doing a lot and thinking too little.\n\nIt is interesting to look at the chart of agent turns and price. Grok and Gemini Flash moved from trigger-happy models, to more moderate ones, resulting in a significantly reduced cost.\n\n### Stage 1: The Lake\n\nFor models that passed at least 7 out of 8 intro levels, we run them on the next stage, **the Lake**.\nOnly one attempt per level, as it is costly. Or at least - it used to be costly with the previous models, as now it is getting significantly better.\n\n| Model | Harness | Solved | Total time | Output tokens | Total cost | |||||||||||||||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| 14 | 2h 36m | 10.7M | $53.25 | |||||||||||||||||\n| 14 | 6h 46m | 16.3M | $61.96 | |||||||||||||||||\n| 14 | 9h 11m | 1023.1M | $55.76 | |||||||||||||||||\n| 14 | 9h 16m | 16.4M | $31.42 | |||||||||||||||||\n| 13 | 8h 57m | 50.1M | $102.08 | |||||||||||||||||\n| 13 | 20h 16m | 38.6M | $46.20 | |||||||||||||||||\n| 13 | 22h 7m | 75.6M | $8.48 | |||||||||||||||||\n| 12 | 9h 7m | 49.9M | $107.76 | |||||||||||||||||\n| 12 | 24h 57m | 9.6M | $16.85 | |||||||||||||||||\n| 11 | 23h 41m | 62.9M | $37.51 | |||||||||||||||||\n| 11 | 29h 34m | 129.5M | $78.15 | |||||||||||||||||\n| 8 | 5h 58m | 41.8M | $58.17 | |||||||||||||||||\n| 6 | 7h 46m | 1118.1M | $269.48 | |||||||||||||||||\n| 6 | 24h 11m | 92.8M | $4.31 |\n\n### Time\n\nFirst, let’s look at the wall time, for models run via APIs. While it might change with providers, or model bandwidth at a given time, it gives a ballpark estimate of effective model speed. In everyday software engineering it is what we care for, anyway.\n\nAnd wow, Gemini 3.7 Flash is fast - not only per token (which is widely cited but largely irrelevant), but per problem solved. Here, for most levels, only Claude Fable 5 is faster.\n\nGrok 4.6, while 2x slower than GPT-5.6 Sol, still manages to solve 13 of 15 levels (GPT-5.6 Sol solves 14). It’s a promotion to the league of top models.\n\nDeepSeek V4 Pro 0813 might be the slowest of the top models - but it is the first open-weight model that finally reached the same top league. It is way beyond GLM-5.2 or Kimi K3 results.\n\nGLM-5.3 was almost the same as GLM-5.2, to the point that two lines overlap; though, it solved one more level.\n\n### Cost\n\nNow let’s look at costs. It’s beautiful.\n\nGrok 4.6 is cheaper than Fable 5, but more expensive than GPT-5.6 Sol or Opus 5.\n\nWhile [Gemini 3.6 Flash was ridiculously expensive](https://quesma.com/blog/baba-kimi-k3-opus-5/), Gemini 3.7 Flash is one of the cheapest models. At 12 levels, it is twice as cheap as GPT-5.6 Sol, the previous cheapest model. At 13 tasks it is still twice as cheap. Only vs DeepSeek Pro is it slightly more expensive.\n\nWhat happens with DeepSeek V4 Pro 0813 is a pure wonder. Not only is it open-weight, but also the cheapest of all.\n\nGLM-5.3 was 2x cheaper than its predecessor.\n\n## Some lessons about harnesses\n\nIn previous experiments we used various harnesses. Here we used [Harbor](https://www.harborframework.com/)-default Terminus-2, except DeepSeek, which we ran on Claude Code. While there are [Grok Build](https://x.ai/build) and [DeepSeek Harness](https://news.ycombinator.com/item?id=49285244), results were slightly worse (Grok) or vastly inferior (DeepSeek). That said, these are newer tools, and likely to get improved in the future. If there is interest, we’re happy to share more insight on running with various harnesses — they influence both cost and results.\n\n## Conclusion\n\nProgress is fast - with more players joining the frontier game, and open-weight models reaching frontier-level results at a much lower price. And I am looking forward to seeing more models in the coming months. And to the first local model solving the Lake.\n\nAs my colleague Piotr Grabowski noticed:\n\n“Second-tier labs caught up with the frontier” matches my experience: last few days I’ve been using Grok 4.6 or GLM-5.3 for easy/medium difficulty tasks and couldn’t see major difference between them and Opus 5/GPT-5.6 Sol (for harder tasks yes, there’s still gap).\n\nAnd what is your experience?", "url": "https://wpnews.pro/news/gemini-3-7-flash-grok-4-6-glm-5-3-and-deepseek-v4-pro-joined-the-frontier", "canonical_source": "https://quesma.com/blog/baba-is-aug-2026/", "published_at": "2026-08-20 00:00:00+00:00", "updated_at": "2026-08-20 17:16:16.019792+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["Gemini 3.7 Flash", "Grok 4.6", "GLM-5.3", "DeepSeek V4 Pro 0813", "Qwen3.8 Max", "Qwen3.8 27B", "Quesma", "Baba Is Bench"], "alternates": {"html": "https://wpnews.pro/news/gemini-3-7-flash-grok-4-6-glm-5-3-and-deepseek-v4-pro-joined-the-frontier", "markdown": "https://wpnews.pro/news/gemini-3-7-flash-grok-4-6-glm-5-3-and-deepseek-v4-pro-joined-the-frontier.md", "text": "https://wpnews.pro/news/gemini-3-7-flash-grok-4-6-glm-5-3-and-deepseek-v4-pro-joined-the-frontier.txt", "jsonld": "https://wpnews.pro/news/gemini-3-7-flash-grok-4-6-glm-5-3-and-deepseek-v4-pro-joined-the-frontier.jsonld"}}