{"slug": "rust-inference-engine-how-to-run-ai-models-in-rust", "title": "Rust Inference Engine: How to Run AI Models in Rust", "summary": "A pure-Rust LLM inference engine called Frink, built by developer Antonello F since the summer, decodes faster than llama.cpp on an M2 Pro (v0.20) but was 25x to 43x slower at prefill on an RTX 3090 (v0.21), according to the project's published benchmark results. In the post, the author argues that Rust's speed advantage comes from the underlying SIMD, Metal or CUDA kernels rather than the language itself, while its memory overhead and concurrency benefits are genuine wins for serving, and that Rust's safety guarantees stop at the FFI boundary, leaving `ort`, `tch-rs` and llama.cpp bindings able to segfault.", "body_md": "# Rust Inference Engine: How to Run AI Models in Rust\n\nA **Rust inference engine** runs a trained model from Rust code: an LLM, an embedding model, a classifier or a recommender. You give it weights and input, and it gives you tokens, vectors or scores back. (This post is about machine learning inference. If you came here for the compiler’s type inference, that is a different topic.)\n\nRust has become a common choice for this layer. Most teams still train in Python, then look for something faster, smaller and easier to deploy to serve the result. I spend most of my time on that serving side, and I have been building [Frink](https://github.com/antonellof/frink), a pure-Rust LLM inference engine, since the summer. Building one taught me which of the usual claims about Rust hold up and which need a footnote.\n\nThis post covers what an inference engine does, where Rust helps, the main ways to run models in Rust today, and how to choose between them.\n\n## What an inference engine does\n\nWhatever the model, a call goes through the same stages. For an LLM behind an API, it looks like this:\n\nThe engine is the part in the middle. It loads the weights, runs the forward pass, and for an LLM manages the KV cache and the sampling loop. A serious one also handles what happens when many requests arrive at once: batching, queueing and memory. If you want the mechanics of a single LLM call, I wrote them up in the [LLM inference roadmap](https://www.fratepietro.com/2026/llm-inference-learning-roadmap/).\n\n## Python for training, Rust for inference\n\nThe typical split looks like this:\n\nTraining depends on an ecosystem that Rust won’t replace any time soon: PyTorch, TensorFlow, CUDA tooling, and years of research code. Serving has different needs. You care about latency, throughput per dollar, RAM, startup time, and how easily the thing deploys, and Rust is strong on all of those.\n\n## What Rust buys you, and what it doesn’t\n\nThe usual list says Rust is fast, memory efficient, safe, concurrent and easy to deploy. All of that is broadly true, but each item has conditions.\n\n**Speed comes from the kernels.** Writing an engine in Rust doesn’t make it fast. Most inference time is spent in tight loops over matrices, and those run as fast as the SIMD, Metal or CUDA code behind them. PyTorch’s kernels aren’t written in Python either. Frink shows this well. On an M2 Pro (v0.20) it [decodes faster than llama.cpp](https://github.com/antonellof/frink/blob/main/benchmarks/RESULTS.md) on the same file, because Metal is where the kernel work went first. On CUDA, in the last run on an RTX 3090 (v0.21), it was 25x to 43x slower at prefill, in the same language.\n\n**Memory and overhead are real wins.** There is no garbage collector, no interpreter and no large runtime sitting beside the model. The weights still take what they take, but everything around them shrinks. That matters most on edge devices and in services that scale to many small replicas.\n\n**Concurrency is where serving gets better.** Async Rust handles thousands of open streams on a few threads, and the type system rejects data races at compile time. Python’s GIL is the usual limit on the other side. Native code can release it during a forward pass, so the limit mostly hits the request handling around the model. Python 3.14 also made the [free-threaded build officially supported](https://peps.python.org/pep-0779/), although it is optional and the GIL is still the default. One Rust-specific trap: inference blocks the CPU, so keep it off the async workers, with `spawn_blocking` or a dedicated thread, or your other streams will stall.\n\n**Safety stops at the FFI boundary.** Safe Rust rules out use-after-free and data races. It can’t protect you inside a C++ library you call, so `ort`, `tch-rs` and llama.cpp bindings can all still segfault. GPU kernels also need `unsafe` code in any language, and in a pure-Rust engine that code is yours to get right. Rust won’t catch a model that computes the wrong answer either, and that was the bug I worried about most in Frink. Each architecture there is checked against llama.cpp’s own logits, and anything without that evidence refuses to load.\n\n**The single binary is a big deal, with one condition.** Copying one file to a server and running it beats matching Python wheels against CUDA drivers. Frink ships as a 23 MB binary on macOS arm64, including the server. PyTorch alone is 402 MB. That only holds if you avoid large native dependencies, though. `tch-rs` still needs libtorch, and `ort` needs ONNX Runtime.\n\n## How to run AI models in Rust\n\nThere are six main routes, and the first thing to compare is what each one brings along at runtime.\n\n### ONNX inference in Rust with ort\n\nFor anything that isn’t a large LLM, this is my default. You export the model once and run it from Rust:\n\n[ONNX Runtime](https://onnxruntime.ai) is mature, and `ort` wraps it with execution providers for CUDA, CoreML and others. It fits embeddings, BERT, classifiers and rerankers, where the graph is fixed and the job is a single forward pass. Make sure the GPU provider is really enabled, because a missing Cargo feature can leave you on the CPU without an obvious error. Also note that `ort` 2.0 is still published as a release candidate.\n\nIf you want ONNX with no native dependency at all, [tract](https://github.com/sonos/tract) is the small, self-contained option.\n\n### Candle: pure Rust from Hugging Face\n\n[Candle](https://github.com/huggingface/candle) is Hugging Face’s minimalist ML framework, with native Rust tensors and no Python dependency. It is lightweight, it pairs naturally with the `tokenizers` crate, and it runs Llama, Mistral, Phi and Gemma models. Choose it when you want to write the model code yourself in Rust, or embed a transformer in an application. It is a framework more than a server, so batching and an API are up to you.\n\nIf you also want training in Rust, look at [Burn](https://github.com/tracel-ai/burn), which covers both.\n\n### tch-rs: PyTorch bindings\n\n[tch-rs](https://github.com/LaurentMazare/tch-rs) gives you Rust bindings to PyTorch’s C++ API. It makes sense when a model only exists in PyTorch form and you want it behind a Rust service without re-implementing it. The price is libtorch, which is exactly the dependency many teams are trying to leave behind.\n\n### llama.cpp bindings\n\n[llama.cpp](https://github.com/ggml-org/llama.cpp) has the widest model and hardware coverage of any local engine, and crates like [llama-cpp-rs](https://github.com/utilityai/llama-cpp-rs) bring it into Rust. This is the low-risk way to run local LLMs today: you get a proven engine and keep your application in Rust. You also keep a C++ build and its FFI boundary. I compared it with vLLM and SGLang in [three approaches to LLM inference](https://www.fratepietro.com/2026/llama-cpp-vllm-sglang-inference-tools/).\n\n### Pure-Rust LLM inference engines\n\nThe last group reimplements the whole engine in Rust. [mistral.rs](https://github.com/EricLBuehler/mistral.rs) is the biggest project here by GitHub stars, with broad multimodal support and an OpenAI-compatible server. Google results for “Rust inference engine” also turn up newer projects like [Atlas](https://github.com/Avarok-Cybersecurity/atlas), [xinfer](https://github.com/guoqingbao/xinfer) and [Crane](https://github.com/lucasjinreal/Crane), so this space is moving quickly.\n\nOne item you will still see in older tutorials: the [`llm` crate from rustformers](https://github.com/rustformers/llm) was archived in June 2024 and is unmaintained. Skip it for new work.\n\n## Frink: a Rust inference engine for GGUF models\n\n[Frink](https://github.com/antonellof/frink) (first released as Ferrox) is my entry in that last group. It loads GGUF checkpoints and runs them on CPU, Apple Metal or CUDA, with no bindings to llama.cpp and no ggml wrapper. I want it to be what you run instead of llama.cpp, with the same models and the same commands.\n\n```\ncurl -fsSL https://raw.githubusercontent.com/antonellof/frink/main/scripts/install.sh | bash\n\n# Fetch a model and serve an OpenAI-compatible API\nfrink serve -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -c 8192 --alias local\n\n# Compare it with llama.cpp on your own machine\nfrink bench -m model.gguf -p 512 -n 128 -r 3 --compare\n```\n\nYour existing OpenAI client can point at it unchanged. The server also answers Anthropic Messages and the Responses API, does continuous batching, and shares a prefix cache across conversations. On an M2 Pro, a shared 757-token system prompt reuses 736 tokens from that cache, which took time to first answer from 938 ms to 410 ms. If you want the engine as a library, it is on crates.io as [`frink-inference`](https://crates.io/crates/frink-inference).\n\nIt isn’t the right pick for everything. As of the [latest benchmarks](https://github.com/antonellof/frink/blob/main/benchmarks/RESULTS.md), Metal is where it is strongest, while CPU trails llama.cpp and CUDA trails it by a lot, and those CUDA rows are overdue for a re-run. There is no Vulkan backend yet, so AMD and Intel GPUs are out. 101 architectures run with evidence behind them, and any other architecture stops with an error saying what is missing. The [full comparison with llama.cpp](https://www.fratepietro.com/2026/frink-rust-alternative-to-llama-cpp/) has the details, and [the first post](https://www.fratepietro.com/2026/frink-rust-gguf-inference-engine/) covers why I started it.\n\nBuilding an engine yourself is a good way to learn, but most teams shouldn’t. Pick an existing one unless you have a problem they can’t solve. Mine was running mixture-of-experts models that don’t fit in memory.\n\n## RAG architecture in Rust\n\nRAG is one of the best fits for Rust, because most of the request is concurrent I/O around a few inference calls:\n\nThe pieces already exist. [Axum](https://github.com/tokio-rs/axum) for the API, [tokenizers](https://github.com/huggingface/tokenizers) for tokenization, and [fastembed](https://github.com/Anush008/fastembed-rs) for local embeddings. [Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference) covers embeddings as a separate service, and [qdrant-client](https://github.com/qdrant/rust-client) talks to [Qdrant](https://github.com/qdrant/qdrant), which is written in Rust as well.\n\nFrink can cover the model side of this too. Point it at a BGE, E5 or GTE checkpoint and `/v1/embeddings` serves it, and a cross-encoder answers `/v1/rerank`.\n\n## Rust vs Python for inference\n\nThe kernels are native on both sides, so Python is often not the bottleneck during the forward pass. [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang) are mostly Python, and both are built for large GPU fleets. Rust earns its place in the layers around the model, and on hardware where a Python stack is too heavy.\n\n## A good stack for production RAG\n\nIf I were starting a RAG product today:\n\nThe GPU fleet row stays outside Rust on purpose. At that scale what matters is scheduling and KV cache management, and the mature engines have a long lead there. Everything around that model is a good candidate for Rust.\n\n## When to use a Rust inference engine\n\nRust is a strong fit for:\n\n- RAG pipelines and embedding APIs\n- AI gateways and streaming APIs\n- High-QPS inference and low-latency systems\n- Edge AI, and anywhere a small binary matters\n- Apps that embed a model, like desktop tools and CLIs\n\nI’d stay with Python if you change models every week, depend on research code, or have a small team that doesn’t already know Rust.\n\n## What big companies do\n\nMany production AI systems follow the same split: Python for training, and C++, Rust or Go for serving. When you serve millions of requests, efficiency starts to matter more than training ergonomics.\n\nYou can see this in the projects themselves. [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) describes itself as “Built in Rust for performance, Python for extensibility”. Hugging Face’s `tokenizers` has a Rust core under its Python package, and Text Embeddings Inference has [Candle and ONNX Runtime backends](https://github.com/huggingface/text-embeddings-inference/tree/main/backends). Rust is usually one layer of the stack, working next to Python and C++, and that seems to me the sensible way to adopt it.\n\n## FAQ\n\n### Is Rust good for LLM inference?\n\nYes, particularly for the server, scheduling and application layers, and for local inference on consumer hardware. The speed depends on the engine’s kernels, so benchmark the engine itself on your hardware.\n\n### What is the best Rust inference engine?\n\nIt depends on the workload. I’d use `ort` for embeddings and classifiers, Candle to write model code in Rust, llama.cpp bindings for the widest local coverage, mistral.rs for multimodal serving, and Frink for GGUF models on Apple Silicon or a llama.cpp replacement with no C++.\n\n### Can Rust run GGUF models?\n\nYes. llama.cpp bindings, Candle, mistral.rs and Frink all load GGUF. Frink is built around it, and uses llama.cpp’s flags, so commands from a model card work unchanged.\n\n### Can you run LLM inference in Rust without Python?\n\nYes. Candle, mistral.rs and Frink all run without a Python runtime. You may still use Python once to export or convert the model.\n\n### Should you rewrite your Python inference service in Rust?\n\nUsually not all of it. Measure first. If the time goes to the model, a rewrite won’t help much. If it goes to request handling, tokenization or memory overhead, move those parts first.\n\n## Where this is heading\n\nThe AI stack may settle into something like this:\n\nRust is becoming one of the most important languages in AI infrastructure, mostly in that middle line. If you want to try it on a model you already use, [Frink](https://github.com/antonellof/frink) installs in one command, and `frink bench --compare` will show you how it does on your own machine.", "url": "https://wpnews.pro/news/rust-inference-engine-how-to-run-ai-models-in-rust", "canonical_source": "https://www.fratepietro.com/2026/rust-inference-engine/", "published_at": "2026-11-06 23:00:00+00:00", "updated_at": "2026-11-09 01:38:53.645520+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Frink", "Rust", "llama.cpp", "PyTorch", "TensorFlow", "CUDA", "Metal", "Antonello F"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rust-inference-engine-how-to-run-ai-models-in-rust", "markdown": "https://wpnews.pro/news/rust-inference-engine-how-to-run-ai-models-in-rust.md", "text": "https://wpnews.pro/news/rust-inference-engine-how-to-run-ai-models-in-rust.txt", "jsonld": "https://wpnews.pro/news/rust-inference-engine-how-to-run-ai-models-in-rust.jsonld"}}