Can a local LLM beat -O3? A developer's experiment presented as a poster at CppCon tested whether a small, local LLM integrated into the Clang/LLVM compilation pipeline could match the -O3 optimization flag, using five deliberately inefficient C++ benchmark programs including fibonacci, format_list, repeated_sort, count_matches, and top_words_from_file. The author notes the post was written before testing larger cloud models, which "perform drastically better and make LLVM IR optimization a viable approach." The work follows Andrei Alexandrescu's June ACCU on Sea keynote predicting LLM-compiler integration, and the code is published on GitHub as llmO. During his keynote at ACCU on Sea https://www.youtube.com/watch?v=-RWdevA0gWI in June, Andrei Alexandrescu made some predictions. One of those was the integration of LLMs and compilers. Compilers can generate optimization remarks for LLMs to process. And LLMs can propose optimizations that are difficult for a conventional compiler to discover. If you haven’t yet, I very much recommend you to watch Andrei’s talk. It’s a great talk, and it led me down a rabbit hole of experimenting: what could an LLM do today? Integrated into the compiler pipeline, locally on my hardware? These experiments resulted in a poster, which I will present today at the registration reception of CppCon https://cppcon.org . The aim of the poster is to answer these questions: - How much performance improvements can a small, local LLM achieve as part of the compilation process? - How likely is the LLM to produce code that is functionally correct? - Is the IR optimization stage the right place to introduce LLM-based optimizations, or is it better to optimize the C++ code before it goes to the compiler? This blog post allows me to dive a bit deeper into my project, with links to the code https://github.com/Blandinium/llmO . ℹ️ This post was written before I tested with larger cloud models. They perform drastically better and make LLVM IR optimization a viable approach. Please also check my follow-up post /posts/cppcon-cloud with those results. The benchmark programs the-benchmark-programs The testing library consists of five exported functions, each implemented in a separate C++ file. They each test for a different optimization situation: - fibonacci https://github.com/Blandinium/llmO/blob/main/SUT/fibonacci.cpp : A basic recursive implementation, with an obvious algorithmic optimization opportunity. - format list https://github.com/Blandinium/llmO/blob/main/SUT/format list.cpp : Array serialization in a for loop, with opportunities for optimizing the string concatenation. - repeated sort https://github.com/Blandinium/llmO/blob/main/SUT/repeated sort.cpp : Gives the model an opportunity to notice unnecessary repeated work. - count matches https://github.com/Blandinium/llmO/blob/main/SUT/count matches.cpp : Finding elements of one array in another array, with a possible optimization in picking another data-structure. - top words from file https://github.com/Blandinium/llmO/blob/main/SUT/top words from file.cpp : The most complex of the five: file parsing and word-frequency counting, with a deliberately recursive implementation. The test cases contain deliberately inefficient implementation choices. The aim was to give both the compiler and the LLM opportunities for optimization. In retrospect, this might have been the choice that had the biggest effect on the outcome of the experiment. If I would repeat the experiment now, I would probably include test code with less obvious algorithmic opportunities, where the optimizations actually have to come from the more low-level tweaks a complier excels at. This experiment is based on Clang/LLVM because LLVM exposes the intermediate representation used by its optimization passes in a documented textual format: LLVM IR . This conveniently allows the LLM to work directly on a textual representation of the same IR that LLVM itself optimizes. How much does the compiler matter? how-much-does-the-compiler-matter The poster title mentions -O3 . But I also bench marked other optimization flags. This shows -O3 does not always give you the most efficient binary. It is however a reasonable starting point. In my results, the performance of -O3 tends to be close to the optimal performance. But in the end, you’ll only know what the optimal flags are for your application, if you measure. Although this is also an area where LLMs might help. Meta’s LLM Compiler work https://ai.meta.com/research/publications/meta-large-language-model-compiler-foundation-models-of-compiler-optimization/ , for example, includes models fine-tuned to predict compiler optimization sequences for code-size optimization. This is how different compiler flags performed on my test code: You might notice there is no result for -O0 for top words from file . The recursive implementation hits a stack overflow with the test data used during the benchmark. At -O1 and above, LLVM optimizes enough of the recursion away for the test to complete. How I measured performance how-i-measured-performance The final results are only as good as the method used to measure them. I took great care to make the measurements reliable. If you find weaknesses in my approach, I’d be very interested in your feedback. You’ll find my email address in the heading of my homepage / Each of the five test functions is optimized separately, either as LLVM IR or as C++ code, as described below. The optimized code is then compiled into a shared library this library contains all five benchmark functions: one of them optimized, the others with the original code . The librunner https://github.com/Blandinium/llmO/blob/main/librunner/main.cpp application loads this library, first runs a test suite to verify correctness, and only then benchmarks it. The runner itself is compiled once and the same binary is used for all benchmark runs. The benchmarking code https://github.com/Blandinium/llmO/blob/main/librunner/benchmarks.cpp first calibrates each benchmark by estimating how many iterations should take approximately 30 seconds. It then executes that number of iterations, cycling through a deterministic set of input values, and measures the resulting throughput in calls per second. This is the performance value collected by the Python scripts. For the final benchmark matrix https://github.com/Blandinium/llmO/blob/main/run final benchmark matrix.py , each benchmark is measured five times. Execution order is randomized in balanced blocks to reduce systematic effects from one implementation consistently running before another. The median calls-per-second value from the five measurements is used as the final result. Because small timing differences can easily be measurement noise, I use a 2% threshold when classifying a candidate as faster. When inspecting the result, I noticed 3 candidates containing undefined behavior, but still passed my correctness tests. I then added ASan/UBSan validation, and marked these 3 candidates as invalid in the final results on the poster. The optimization strategies the-optimization-strategies I set out on this experiment, thinking I would just run 2 LLM optimization passes for each testing function on each model: - Naive C++ https://github.com/Blandinium/llmO/blob/main/run naive cpp optimization.py : Feed the full C++ test file each file contains one of the five test functions to the model, together with a prompt telling it to optimize the code without changing the external API. - Full LLVM IR https://github.com/Blandinium/llmO/blob/main/run ir optimization.py : Similar to Naive C++, but sending one full LLVM IR file, created from the C++ code by compiling it with -O1 . The results of this were quite disappointing. Especially for LLVM IR, where only five of the forty tasks produced a valid candidate. Context length was a major failure mode, especially for full LLVM IR. This led me to try out other strategies: - Extracted LLVM IR https://github.com/Blandinium/llmO/blob/main/run ir optimization.py : The test function is extracted from the full LLVM IR module, using llvm-extract . The extracted IR is sent to the model, which is instructed to return only a replacement for the target function, without repeating the surrounding module. - Guided C++ https://github.com/Blandinium/llmO/blob/main/run iterative cpp optimization.py : This strategy tries to guide the model by providing it with information from the compiler. The LLM is given some of the compiler optimization hints, together with the code. As long as the compiler generates new hints, the LLM gets up to three iterations, each time with new compiler hints. If compilation fails, the LLM gets up to two chances to fix the code, based on the compiler error. Guided C++ adds compiler remarks, multiple optimization iterations, performance-based selection, and repair attempts. To investigate the contribution of repairs, I added another strategy: - Naive C++ + repair https://github.com/Blandinium/llmO/blob/main/run naive cpp repair.py : When compilation fails, the model gets a chance to fix its code, based on the compiler error. Results differed a lot by model. But when combined, this was the overall picture for each of the optimization strategies: The models the-models The choice of models matters a lot to the outcome of this experiment. I set myself the restriction that models had to be able to run locally, on my hardware. This hardware has plenty of RAM 128 GB , but no GPU suitable for LLM inference. The LLM had to run only on the CPU AMD Ryzen 9 5950X . We are used to compilers being tools that we can download, install, and run locally, often free of charge and open source. LLM-assisted compilation does not necessarily have to follow that model: a future compiler could rely on large models running in the cloud, much as many coding agents already do today. For this experiment, however, I deliberately limited myself to relatively small local models. I wanted to explore how useful an LLM could be as part of a compiler that still behaves like the compilers we use today: software that you can download and run on your own machine. These are the eight models I picked: - Four general/coding models of similar size 12–14B : Qwen3 14B https://huggingface.co/Qwen/Qwen3-14B , Qwen2.5-Coder 14B https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct , Gemma 4 12B https://huggingface.co/google/gemma-4-12B-it , Ministral 3 14B https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512 . - Two compiler-specialized models: LLM Compiler 7B https://huggingface.co/facebook/llm-compiler-7b and 13B https://huggingface.co/facebook/llm-compiler-13b . - Two larger coding/reasoning models: gpt-oss-20b https://huggingface.co/ggml-org/gpt-oss-20b-GGUF and Devstral Small 2 24B https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512 . I specifically included the LLM Compiler models from the Meta paper https://ai.meta.com/research/publications/meta-large-language-model-compiler-foundation-models-of-compiler-optimization/ I mentioned earlier. These models are based on Code Llama and were further pretrained on LLVM IR and x86-64, ARM and CUDA assembly. Meta also released FTD variants, further fine-tuned to predict optimization sequences to reduce code size. I tested the foundation variants, hoping their specialized training would make them better at processing LLVM IR than the more general models. However, in my experiment, neither produced a valid optimization candidate. If you managed to get better results out of these models, I’d be interested to hear how. These are the results by model: Results results We’ve already seen the models don’t always produce correct results. But when the generated code is valid, it certainly can outperform the code optimized by the compiler: Also when I benchmark the fastest LLVM optimization against the fastest LLM candidate, the LLM wins: What did the LLM actually do? what-did-the-llm-actually-do All candidates that produced a meaningful 2% speedup came from the C++ strategies. These mostly target the deliberate inefficiencies present in the benchmark programs. The most interesting of them, is probably format list , where the LLM attempts optimizations that are closer to the kinds of things a compiler might attempt. The LLMs try things like reserving the output string, estimating the required size or manually converting integers https://github.com/Blandinium/llmO/blob/main/results/guided-cpp/20260807 full run-guided/devstral-small-2-24b-q4km/format list cpp/artifacts/guided-cpp devstral-small-2-24b-q4km format list final/optimized format list.cpp . But most of these candidates fail to outperform -O3 . And format list is also the benchmark where the best LLM result was closest to conventional compiler performance. Against the fastest LLVM configuration, the difference was around 2%. Failure modes failure-modes Failures can be roughly grouped into five categories: Response truncation : With 71 cases, this is the larges group. The LLM burns through the token budget, without producing a useful result. For instance this Gemma 4 run https://github.com/Blandinium/llmO/blob/main/results/naive-cpp/20260807 full run-naive/gemma-4-12b-it-qat-udq4xl/fibonacci cpp/optimization response.json . The model keeps reasoning. 4096 Tokens long the reasoning content field in the JSON . And it never generates the actual response. Context too large : There are 33 cases where the model context was simply insufficient. This was mainly an LLVM IR problem, where sometimes even the prompt alone did not fit in the context. This top words from file run https://github.com/Blandinium/llmO/blob/main/results/llm-ir/20260807 full run-ir/qwen2-5-coder-14b-q4km/top words from file cpp/summary.json is an extreme example. The context was around 32k tokens, but the prompt alone was 107k tokens long. Compile/assemble failures : In 20 cases, the generated code did not compile. These are sometimes relatively small mistakes. In this format list example https://github.com/Blandinium/llmO/blob/main/results/naive-cpp/20260807 full run-naive/qwen3-14b-q4km/format list cpp/optimized.cpp , Qwen3 used memcpy , but forgot to include