{"slug": "agents-on-rails-grok-4-6-glm-5-3-gemini-3-7-flash-and-opus-4-8", "title": "Agents on Rails: Grok 4.6, GLM 5.3, Gemini 3.7 Flash, and Opus 4.8", "summary": "Rails' Agents on Rails benchmark added Grok 4.6, GLM 5.3, Gemini 3.7 Flash, and Claude Opus 4.8, with Claude Opus 5 still leading at 58 of 63 tasks. The benchmark now includes 792 runs with full traces published in rails/ai-evals, revealing model habits such as test-suite runs and diff verbosity that do not predict solve rates. One model, Muse Spark 1.2, exposed an OpenRouter API key during runs, which was redacted before publication.", "body_md": "Last week we launched [Agents on Rails](/ai) and published [the first benchmark report](/2026/8/13/agents-on-rails-the-first-benchmark-report). The response was immediate: suggestions, questions, model requests, and more than a few “but have you tried…” messages. We love the enthusiasm! We want this benchmark to be useful to you, so for this run we added four new models, updated the insights, and uploaded the full traces of the first two rounds (every command, diff, and verdict).\n\nRead on for the new results, check the updated [leaderboard](/ai), or go straight to [the raw runs](https://github.com/rails/ai-evals).\n\nWe got asked this a lot and it is written in the [methodology](https://github.com/rails/ai-evals/blob/main/methodology.md), but in case you missed it: All models were run at their provider’s **default** effort level.\n\nWhat’s been added? The most requested model by the community was **Grok**, so we added 4.6 to the benchmark. The day we launched the benchmark, **Google’s Gemini 3.7 Flash** [ landed](https://x.com/GoogleDeepMind/status/2087948366294515977), and within hours, **GLM 5.3** was also [announced](https://x.com/Zai_org/status/2088132965922476159), so they also went into the benchmark. And finally, we were curious if **Claude Opus 4.8** would perform better than Opus 4, as some have experienced.\n\nAll four of these models are now on the leaderboard. Here’s how they did:\n\nUltimately, Claude Opus 5 still leads at 58 of 63, with Kimi K3 (open weights) and Claude Fable 5 one run behind. The full leaderboard, with costs, recall rates, and the per-task grid, lives on the [Agents on Rails](/ai) page, as usual.\n\nWith 756 scored runs on file, we started mining the trajectories and patches for something scores can’t show: *how* each model works. How often it runs the test suite, how many steps it spends verifying after its last edit, how verbose its diffs are, whether it writes tests, and, your most oft-mentioned sin, where it puts comments.\n\nSome interesting patterns emerged::\n\nThese habits are traits rather than tactics. Whether a task is easy or hard, each model works the same way it always does, and *none of these habits predict solve rate*. Running the suite 20 times doesn’t beat running it 3 times. Writing tests correlates with the task being hard, rather than winning.\n\nAnother pattern caught our eye (we’d call it a hint rather than a rule, each model only fails a handful of times, so the sample is small). When Claude Fable 5 failed, it was usually in the wrong place: only a quarter of its failed runs touched the files the fix lives in. When the GPT-5.6 models failed, nearly 80% of the time they found the right files and fixed them incorrectly. If that pattern holds up in bigger corpora, it changes how you review each family’s output.\n\nWe’re still preparing **lemans**, our Ruby harness, for its open source release, but the data doesn’t need to wait.\n\nEverything above was computed from the raw run directories, and those directories are now public in [rails/ai-evals](https://github.com/rails/ai-evals) with all 792 runs from both rounds, every command, diff, and verdict. Each run is one directory:\n\n`trajectory.json`\n\n— the full transcript: every model turn, every command, every output.`agent.patch`\n\n— the diff the agent actually shipped.`checks.json`\n\nand `verifier.log`\n\n— what the hidden tests said, check by check.`result.json`\n\n— reward, steps, tokens, cost, wall clock.Please dig in!\n\nOne note before you do: trajectories record whatever the model decides to look at. And one model decided to look at *everything*: Muse Spark 1.2 opened many of its runs with a full system recon (`env`\n\n), which exposed our OpenRouter API key. Before publishing, we swept every file and redacted key values. No other model leaked a thing, and most never touched the environment at all.\n\nBy the way, this is a lesson that lives on beyond the scope of this benchmark report. If you run your own agent sandbox, assume any secret in its environment will eventually end up in a trajectory file.\n\nThere’s also a few honest limits to keep in mind while you dig. Three attempts per model per task is a small n, so read the scores as 21 observations with ±5 points of noise, not 63 independent trials. And it’s one app in one framework, with a minimal setup — nothing here exercises the Solid stack, multiple databases, or sharding.\n\nAnd we’ve certainly not found everything! If you dig out something interesting like a failure pattern, a model quirk, or a task we graded too kindly, bring it to [rails/ai-evals](https://github.com/rails/ai-evals): open an issue with your find, or a pull request if you’ve built something on top of the data.\n\nThe current numbers are always on the [Agents on Rails](/ai) page. More models, tasks, and lemans itself are on the way.", "url": "https://wpnews.pro/news/agents-on-rails-grok-4-6-glm-5-3-gemini-3-7-flash-and-opus-4-8", "canonical_source": "https://rubyonrails.org/2026/8/17/agents-on-rails-grok-4-6-glm-5-3-gemini-3-7-flash-and-opus-4-8", "published_at": "2026-08-17 00:00:00+00:00", "updated_at": "2026-08-17 19:41:37.265870+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools", "ai-agents"], "entities": ["Rails", "Grok 4.6", "GLM 5.3", "Gemini 3.7 Flash", "Claude Opus 4.8", "Claude Opus 5", "Kimi K3", "Claude Fable 5"], "alternates": {"html": "https://wpnews.pro/news/agents-on-rails-grok-4-6-glm-5-3-gemini-3-7-flash-and-opus-4-8", "markdown": "https://wpnews.pro/news/agents-on-rails-grok-4-6-glm-5-3-gemini-3-7-flash-and-opus-4-8.md", "text": "https://wpnews.pro/news/agents-on-rails-grok-4-6-glm-5-3-gemini-3-7-flash-and-opus-4-8.txt", "jsonld": "https://wpnews.pro/news/agents-on-rails-grok-4-6-glm-5-3-gemini-3-7-flash-and-opus-4-8.jsonld"}}