{"slug": "performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200", "title": "Performance portable LLM inference in Fortran (beating llama.cpp/vllm on H200)", "summary": "A hand-written Fortran implementation of Qwen3.8-27B (MLX 8-bit) inference, compiled with LFortran and offloaded to an NVIDIA H200 GPU via pure Fortran do concurrent, reached 102 tok/s, beating vllm at 92 tok/s and llama.cpp at 72 tok/s, according to benchmarks published by developer certik on GitHub. A hand-optimized C + CUDA version of the same model hit 120 tok/s, while nvfortran from NVHPC 26.5 failed to compile the code and managed only 5 tok/s after an agent rewrite, so it was excluded from the table. The unmodified Fortran code also runs on CPUs via any Fortran compiler and on Apple GPUs via LFortran at 3.4 tok/s, down from nearly 6 tok/s previously, against a peak of about 17 tok/s; the author reports the kernels span about 2,000 lines versus a C+CUDA version roughly 3x shorter and 20% faster.", "body_md": "I think Fortran might be a very solid language for AI, but it has not been used much in it yet. I think it has a lot of potential, because it has arrays, fast numerics, it’s simple, parallelism in the language (both on-node and GPU via do concurrent as well as across-nodes via coarrays), has high-performance compilers and Fortran has always been used to run fast on any platform.\n\nSo let’s see how it can work. Let’s take one of the best latest local models, I picked Qwen3.8-27B-8bit (released 3 months ago), but any other recent model can be used also.\n\nLet’s write the inference for this model in Fortran: [GitHub - certik/qwen3.8-27B: Qwen3.8-27B (MLX 8-bit) inference: C reference, Fortran (LFortran GPU) and CUDA · GitHub](https://github.com/certik/qwen3.8-27B).\n\nLet’s use do concurrent for GPU offloading, but no other directive/pragmas (so no OpenACC, no OpenMP, nothing), just pure Fortran, as it’s meant to be used.\n\nLet’s start with H200 from NVIDIA. On the machine that I have access to, the inference is limited by the available bandwidth to about 170 tok/s based on theoretical bandwidth, but based on measured memory speed more like 150 tok/s. Let’s write our own C + CUDA version (available at the link above), optimize it. Then let’s benchmark llama.cpp, vllm and finally LFortran-compiled Fortran inference above with CUDA GPU offloading.\n\n| Code | tok/s | \n|---|---|\n| C + CUDA | 120 | \n| LFortran | 102 | \n| vllm | 92 | \n| llama.cpp | 72 | \n\nThere are various options for vllm and llama.cpp, I let AI agent iterate with both for quite a while to extract as much performance as possible, for llama it used flash attention, vllm uses cutlass, etc. Other combinations were slower. Maybe it’s possible to make them run even faster, but the above was the best I was able to get.\n\nI also tried nvfortran that I had access to from NVHPC 26.5 (May 2026 release), but it can’t compile the code, so I had an agent rewrite it so that nvfortran can compile it, but it only ran at 5 tok/s on the GPU, so I didn’t include it in the table.\n\nRegarding performance portability, the above code, unmodified, runs on any CPU via any Fortran compiler. And it also runs on Apple GPU via LFortran, right now at 3.4 tok/s, but I had it running at almost 6 tok/s, so the performance regressed a little bit, but I’ll fix it. The performance peak is about 17 tok/s. So not a bad start.\n\nSome thoughts:\n\nI expected that we will have to extend Fortran with some additional real kinds (bf16, and various other reduced precision), but we didn’t, just standard Fortran is enough.\n\nI expected that we need some directives to handle memory movement, but it looks like the compiler is able to figure everything out by itself. [@sumseq](https://fortran-lang.discourse.group/u/sumseq) is that what you would also expect? If this translates to other codes, then we’ll be in great shape with Fortran.\n\nThe Fortran code is not too bad, about 2,000 lines of kernels, but they can be simplified, I had to do some of the optimizations by hand in Fortran, and we should build them into the compiler instead. The C+CUDA version has fewer kernels and is about 3x shorter as a result, and 20% faster. So we should do the same in Fortran.\n\nSo there is opportunity for improvement for sure, this is just the start. But hopefully it shows that it is possible to use Fortran for LLM inference and run on GPUs and run faster than some of the established inference frameworks.", "url": "https://wpnews.pro/news/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200", "canonical_source": "https://fortran-lang.discourse.group/t/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200/11309#post_1", "published_at": "2026-11-16 05:06:51+00:00", "updated_at": "2026-11-16 05:50:28.669152+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools", "ai-research"], "entities": ["Fortran", "LFortran", "Qwen3.8-27B", "llama.cpp", "vllm", "NVIDIA H200", "nvfortran", "NVHPC 26.5"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200", "markdown": "https://wpnews.pro/news/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200.md", "text": "https://wpnews.pro/news/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200.txt", "jsonld": "https://wpnews.pro/news/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200.jsonld"}}