{"slug": "how-much-better-are-large-cloud-models-at-optimizing-code", "title": "How much better are large cloud models at optimizing code?", "summary": "A follow-up experiment by developer Blandinium comparing three frontier cloud models against three large open-weight models via OpenRouter found the larger models far more reliable at code optimization, with the extracted LLVM IR approach yielding almost 2/3 valid candidates — better than Guided C++ achieved with smaller local models. Claude implemented its own hash table in LLVM IR for the count_matches benchmark, while Gemini's best format_list optimization used resize_and_overwrite with a lookup table for \"00\" through \"99\". Guided C++ still produced the best results overall but was the most expensive strategy due to its multiple iterations.", "body_md": "This is a follow-up post to my experiments with local models for code optimization. In the [previous post](/posts/cppcon-local/), I give more background on how and why the experiment was set up. I mention some limitations of the approach. And some things which, in retrospect, I would do differently. Nevertheless, only two things changed in this experiment: the models, and where inference ran. I picked three frontier models, and three large open-weight models. The inference ran in the cloud, via [OpenRouter](https://openrouter.ai/).\n\n## First impressions[#](#first-impressions)\n\nThe first thing that is immediately clear from these results, is how *much more reliable* the bigger models are. This picture is much greener than it was for local models:\n\nWhile the smaller models failed to produce useful results for LLVM IR, the larger models do much better. *For the top_words_from_file benchmark, the fastest result was an LLVM IR result*:\n\nThe choice of models still matters, but the differences between models are less significant than they were with the smaller models:\n\nGuided C++ still gives the best results:\n\nBut the multiple iterations make it the most expensive of the strategies in this experiment:\n\n## LLVM IR becomes a viable approach[#](#llvm-ir-becomes-a-viable-approach)\n\nBefore I ran this experiment, I uploaded an LLVM IR example into the web interface of some commercial AI agents. These agents run on the same models I used here, but they also use tools which this experiment did not. One of these agents chose to use a decompiler to convert the LLVM IR to C, optimize the C, and convert it back to LLVM IR. This reinforced my suspicion that optimizing LLVM IR directly just wasn’t going to work. But this experiment proved me wrong.\n\nReliability remains an issue with LLVM IR, but with the extracted IR approach we get almost 2/3 valid candidates. That is better than what Guided C++ achieved in the experiment with the smaller models.\n\nSome of these IR optimizations are quite advanced. For count_matches, Claude implemented its [own hash table](https://github.com/Blandinium/llmO/blob/main/results/llm-ir/cloud-six/claude-sonnet-5-medium/count_matches_cpp/optimized.ll) in LLVM IR.\n\nAnother interesting example is [this format_list](https://github.com/Blandinium/llmO/blob/main/results/llm-ir/cloud-six/gpt-5-6-sol-medium/format_list_cpp/optimized.ll) optimization. In a first pass, GPT computes the exact output length. It then reserves enough space in the std::string and writes the result directly into its buffer. To do this efficiently, it even implements its own integer-to-decimal conversion loop.\n\n## C++ Examples[#](#c-examples)\n\nThe [best performing format_list optimization](https://github.com/Blandinium/llmO/blob/main/results/guided-cpp/cloud-six/gemini-3-1-pro-medium/format_list_cpp/artifacts/guided-cpp__gemini-3-1-pro-medium__format_list__final/optimized_format_list.cpp) comes from Gemini. It uses `resize_and_overwrite` and a custom integer formatter with a lookup table for the strings “00” through “99”.\n\nJust like in Claude’s IR example we saw earlier, [GLM writes a custom hash table](https://github.com/Blandinium/llmO/blob/main/results/guided-cpp/cloud-six/glm-5-3-high/count_matches_cpp/artifacts/guided-cpp__glm-5-3-high__count_matches__final/optimized_count_matches.cpp) for count_matches. But it also dynamically chooses between three algorithms, depending on the size and the range of the input.\n\n## Conclusion[#](#conclusion)\n\nWhile the local models optimized code at a higher abstraction level, these larger models are much more willing to lower those abstractions and generate specialized low-level implementations. format_list is particularly interesting because it has no obvious algorithmic shortcut. The local models failed to make significant improvements to it. Several cloud models, however, left `-O3` comfortably behind. And they did so in very interesting ways.\n\nSo will the compiler of the future integrate an LLM? It very well might. And with today’s models, a compiler with LLM optmizations is more likely to be a cloud service than a binary you’ll run on your own hardware.", "url": "https://wpnews.pro/news/how-much-better-are-large-cloud-models-at-optimizing-code", "canonical_source": "https://tijl.blog/posts/cppcon-cloud/", "published_at": "2026-09-13 08:00:00+00:00", "updated_at": "2026-09-14 05:56:38.938637+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools", "developer-tools"], "entities": ["OpenRouter", "Claude", "Gemini", "GPT", "GLM", "LLVM IR", "Blandinium"], "alternates": {"html": "https://wpnews.pro/news/how-much-better-are-large-cloud-models-at-optimizing-code", "markdown": "https://wpnews.pro/news/how-much-better-are-large-cloud-models-at-optimizing-code.md", "text": "https://wpnews.pro/news/how-much-better-are-large-cloud-models-at-optimizing-code.txt", "jsonld": "https://wpnews.pro/news/how-much-better-are-large-cloud-models-at-optimizing-code.jsonld"}}