TornadoVM 7 Ships jitLLM: Beat llama.cpp From Java

TornadoVM 7.3 shipped this month from the University of Manchester and Red Hat, and the headline feature is jitLLM v2.0 — a pure-Java LLM inference engine that JIT-compiles transformer kernels to CUDA, OpenCL, or Apple Metal at runtime. No Python, no native toolchain, no per-GPU recompile. On an A100, the team’s benchmarks show jitLLM outperforming llama.cpp by 18–34% on prompt prefill. Java developers have spent years watching the GPU AI inference space get built entirely in Python. jitLLM is the first credible Java answer to that.

What jitLLM Actually Does #

jitLLM ships as a single JAR. Load a GGUF-format model, call onGPU(true), and TornadoVM’s runtime JIT-compiles the hot transformer kernels — attention, matrix multiply — to CUDA PTX, OpenCL, or Metal shaders on first run. Subsequent calls hit the compiled kernel cache. No manual memory management, no JNI glue, no native rebuild when hardware changes. The supported model list covers the practical majority of open-weight options: Llama 3, Mistral, Qwen 2.5 and 3, Phi-3, IBM Granite 3.3/4.0, DeepSeek-R1-Distill, and Gemma 4.

The fastest on-ramp requires nothing except JBang:

jbang jitllm@beehive-lab -m model.gguf -p "Your prompt here"

For Spring or Quarkus integration, the LangChain4j provider (official since v1.7.1) reduces the entire integration to a builder pattern:

JitLLMChatModel model = JitLLMChatModel.builder()
    .modelPath(Paths.get("/models/llama-3.2-1B-Q8.gguf"))
    .temperature(0.9)
    .onGPU(true)
    .build();

jitLLM also includes an OpenAI-compatible server mode — ./jitllm serve --gpu --model model.gguf --port 8080 — so existing client code that targets the OpenAI API connects without modification. The Quarkus CDI extension injects a model as a bean, treating it like any other dependency in your application context.

The Performance Story and Its Caveats #

On an A100 80GB with 512-token batch prompt prefill, InfoQ’s TornadoVM 7.3 coverage documents the team’s figures: Llama-3.2-1B Q8₀ at 29,004 tokens/s versus llama.cpp’s 23,550 (+23%), and Qwen3-0.6B F16 at 54,355 versus 40,522 (+34%). The CUDA advantage comes from TornadoVM 7’s direct integration with cuBLAS and CUTLASS — it can swap JIT-generated kernels for CUDA library calls or chain them together, bypassing kernel launch overhead for operations where NVIDIA’s own libraries are optimal.

However, these are the team’s own benchmarks. Independent verification has not been published. llama.cpp is a heavily hand-tuned C++/CUDA project with years of micro-optimization, and the DEV Community analysis puts jitLLM’s general performance at roughly 90% of llama.cpp across a broader range of workloads. On Apple Metal with an M4 Pro, MLX wins clearly: 4,042 versus 3,257 tok/s for Qwen3-0.6B F16. If your development or production hardware is Apple Silicon, skip jitLLM for now. MLX is the right tool there, and the jitLLM team says so explicitly.

The FFM Migration and What It Means for Your JDK #

TornadoVM 6.0 in September 2026 replaced JNI with the Foreign Function and Memory (FFM) API and dropped the JVMCI dependency. TornadoVM 7.3 extends that foundation. This matters because JDK 27 removes JVMCI from OpenJDK entirely — any production TornadoVM code on the old architecture would have broken next year. The new path sources compiler metadata through a reflection + ASM + Unsafe layer, giving one uniform path across JDK 21, 25, 26, and 27. As a bonus, the FFM build decodes 3.2% faster and prefills 1.8% faster than the old JNI build. For developers already on JDK 21 or later, this migration is transparent — no API changes. JDK 21 is the hard minimum for TornadoVM 7.x.

Where It Fits Against Ollama and the Python Stack #

jitLLM is not replacing vLLM for high-concurrency batch serving — it is single-stream optimized, and the team makes that explicit. It is not out-DXing Ollama for local developer environments. The specific problem it solves is embedding GPU inference inside a JVM process: for backend services with data residency constraints, for Java shops that cannot run a Python subprocess alongside a production service, or for teams adding LLM features to an existing Quarkus or Spring application without new infrastructure. Memory requirements are substantial — plan for 14GB GPU RAM for 1B models and 20GB+ for 8B — so this is not a laptop-friendly deployment for larger models.

Related: DeepSeek Open-Sources DeepInfer: The Production Inference Engine

Key Takeaways #

• jitLLM v2.0 ships with TornadoVM 7.3 as a single JAR supporting Llama 3, Mistral, Qwen 3, Phi-3, IBM Granite, and DeepSeek models in GGUF format on CUDA, OpenCL, and Apple Metal
• On CUDA, the team’s benchmarks show jitLLM beating llama.cpp by 18–34% on prompt prefill — take these as promising but unverified until independent benchmarks publish
• TornadoVM 7 requires JDK 21 minimum; the FFM API migration future-proofs the stack against JDK 27’s removal of JVMCI
• LangChain4j, Quarkus, and Spring AI integrations make adoption practical for Java teams without architectural rewrites
• Apple Metal is not jitLLM’s strength yet — use it for NVIDIA or OpenCL targets, and reach for MLX on Apple Silicon instead