# Llama.cpp vs vLLM vs SGLang > Source: > Published: 2026-09-15 16:00:00+00:00 Type “llama.cpp vs” into Google and it finishes the sentence for you: vllm, sglang. That’s not us guessing at what people care about – that’s autocomplete rank one, which means enough people are typing this exact comparison that Google’s learned to expect it. Good. Because we went through this exact decision ourselves a few months back, and the honest answer surprised us. Short version up front: these three tools are not competing for the same job. llama.cpp is the lightweight, run-anywhere engine. vLLM and SGLang are built for serving lots of people at once. Picking between them isn’t a benchmark exercise – it’s a question about who’s actually hitting your server. ## What each one actually is **llama.cpp** is the C++ inference engine that started the whole local-LLM movement. It’s the thing Ollama wraps under the hood. It runs GGUF-format quantized models on almost anything – CPU, a single consumer GPU, Apple Silicon, a Raspberry Pi if you’re patient. The project lives on [GitHub](https://github.com/ggerganov/llama.cpp), and its defining feature is flexibility: it supports variable-bit quantization, which lets you trade a little quality for a lot of memory headroom. That one knob – how aggressively you quantize – matters more on a consumer card than most benchmark charts let on. **vLLM** exists to answer a different question: how do you serve one model to a hundred people at once without falling over? It does this with continuous batching – instead of processing requests one at a time, it interleaves them, filling GPU compute that would otherwise sit idle waiting on the next token. That’s its whole reason for existing, and it’s a real one, if you have the load to justify it. **SGLang** sits next to vLLM in the same “serious concurrent serving” category, but it leans harder into structured generation – constrained decoding, complex multi-turn programs, cases where you’re not just asking one question but running a whole decision tree through the model. Think of it as the tool you reach for when your workload looks more like a pipeline of prompts than a chat window. None of these three is “the best.” That’s not a cop-out – the right choice depends on model format, hardware, and concurrency, not a tokens-per-second chart you found on Reddit. ## The question we actually asked ourselves Here’s where this stops being theoretical. A few months into running Glad Labs on a self-hosted [ASUS ROG Astral RTX 5090](https://www.gladlabs.io/go/asus-rog-astral-nvidia-geforce-rtx) with 64GB of system RAM, we asked the obvious question: should we switch our inference stack from Ollama to vLLM? The pitch made sense on paper. We’re an AI-operated content shop – writer agent, reviser agent, topic researcher, QA rails, a voice agent, video generation prompts, all hammering the same box. Surely that adds up to real concurrency, right? Surely vLLM’s batching wins there? The honest answer, after looking at our own call logs, is no. Don’t switch. The reason comes down to load shape, not raw capability. vLLM’s advantage shows up when many simultaneous requests hit the same model – that’s where continuous batching starts filling GPU cycles that would otherwise sit empty. Our actual traffic looks nothing like that. It’s one to three concurrent calls at most – the writer, maybe a voice session, maybe a background worker – against roughly fifty calls a day total. At that load there is nothing for batching to amortize, so a heavier serving layer is overhead we would pay on every single call. That chart is the whole argument in one picture. A model’s raw decode speed – tokens per second, measured in isolation – is not the number that reaches your application. What reaches you is throughput after batching overhead, queuing, and scheduling take their cut. vLLM’s batching machinery is designed to close that gap when you’ve got a stack of concurrent requests to amortize it across. When you don’t, that machinery is pure overhead sitting between you and the model. This is the trap in a lot of “vLLM vs llama.cpp” content: the benchmarks are almost always run at high concurrency, because that’s where vLLM was built to win. If your actual production traffic is a handful of sequential calls a day, you’re reading a benchmark for a workload you don’t have. ## So where does SGLang fit for us We haven’t put SGLang into production, and we’re not going to pretend otherwise. But the audit made its niche clear by contrast. SGLang’s structured-generation strengths – enforcing a JSON schema across a long multi-step prompt chain, running RadixAttention-style prefix caching across a family of related requests – matter most when your workload is a pipeline of many related generations sharing common context, not a single chat turn. Our content pipeline does chain prompts – the writer feeds into the reviser, the reviser feeds into the QA rails – but those stages run through different models and different providers, not repeated calls into one SGLang server holding shared context. If we ever build something that looks like SGLang’s ideal case – a fixed template hammered thousands of times a day with heavy prefix overlap – it goes back on the table. Right now it’s a tool without a job on our stack. The [dev.to guide to 2026 local inference tools](https://dev.to/sreeraj-sreenivasan/the-complete-guide-to-local-llm-inference-tools-in-july-2026-llamacpp-ollama-vllm-sglang-and-4mh1) frames this whole ecosystem as three layers – desktop tools, lightweight servers, and heavy concurrent-serving engines – and that framing matches what we found in practice. llama.cpp and Ollama are layer one and two. vLLM and SGLang are layer three. You don’t “upgrade” from one layer to the next just because a new tool released. You move up a layer when your traffic pattern actually changes shape. ## The hardware wrinkle nobody mentions in the comparison charts Here’s a detail that doesn’t show up in most “vLLM vs llama.cpp” writeups but bit us directly: neither Ollama nor the llama.cpp backend underneath it can pool GPUs from different vendors into one shared VRAM space. If you’ve got an RTX 5090 sitting next to an AMD card, you cannot combine them into a single bigger memory pool for one model. Ollama picks one compute backend per model load – CUDA or ROCm, never both – so a model either lives entirely on the NVIDIA card or entirely on the AMD one. Multi-GPU support in this stack assumes a matched set: multiple NVIDIA cards, or multiple AMD cards, not a mixed pair. That constraint matters more than people expect when they’re planning hardware. It’s part of why we’ve written before about [VRAM as the real currency in local LLM work](https://www.gladlabs.io/posts/the-vram-currency-problem-bb10de87) – the bottleneck usually isn’t compute, it’s whether the model and its context fit in one coherent memory space, on one vendor’s stack, at once. Our [piece on the RTX 5090’s 32GB threshold](https://www.gladlabs.io/posts/the-32gb-threshold-how-the-rtx-5090-redefines-loca-433d67bd) goes into what that extra headroom buys you specifically, and this GPU-pooling limitation is exactly why that headroom on a single card matters more than assembling a mismatched multi-card rig. vLLM and SGLang, by contrast, are built with distributed multi-GPU serving as a first-class feature – tensor parallelism across a matched cluster of NVIDIA cards is a core use case, not an afterthought. If your actual plan is a rack of identical GPUs serving real concurrent traffic, that’s a genuine point in their favor. It’s just not our situation, and it’s probably not most solo developers’ situation either. ## What actually changed for us instead The audit didn’t end with “stay on Ollama and move on.” It ended somewhere more useful: the real problem wasn’t which inference engine we were running, it was that our application code was wired directly to Ollama’s client. Switching engines meant rewriting call sites, not swapping a config value. So the fix wasn’t picking a different serving engine. It was putting a router in front of all of them. We run [LiteLLM](https://github.com/BerriAI/litellm) as the provider layer now, and we’re actively working it toward being the default path for standard-tier generation, with the old hand-rolled Ollama client getting retired from the generation path over time. LiteLLM speaks the same interface whether the backend is Ollama, a cloud model, or – if the load profile ever genuinely calls for it – vLLM or SGLang sitting behind an OpenAI-compatible endpoint. That’s the actual lesson from this whole exercise: the llama.cpp-vs-vLLM-vs-SGLang question stops being a one-time architecture bet once you decouple the application from the engine. You get to answer it per-workload instead of once for the whole company. Our writer agent can sit on a local Ollama instance running a quantized GGUF model. A future high-concurrency customer-facing endpoint, if we ever build one, can sit behind vLLM without touching a single call site upstream. Right now, every generation call in our pipeline – writer, reviser, topic researcher, voice agent – routes through that same layer regardless of which engine answers it, which is the only way this decision doesn’t have to be relitigated every time traffic shape changes. We’re not there yet with SGLang specifically. But the point of putting LiteLLM in the middle is that adding it later is a config change, not a rewrite. ## A decision framework that isn’t a chart Skip the tokens-per-second leaderboard. Ask three questions instead. **How many concurrent requests actually hit this model, at the same time, in a typical hour?** If the honest answer is one to three, llama.cpp – via Ollama or bare – wins on latency and simplicity. If it’s routinely far higher, vLLM’s continuous batching starts paying for itself. That is the actual fork in the road, not raw throughput numbers. **Is the workload a chat turn, or a pipeline?** A single question in, single answer out – that’s llama.cpp territory, full stop. A chain of structured, related generations sharing heavy context – schema-constrained extraction, multi-step agent programs – that’s the shape SGLang was built for. **What GPUs do you actually own, and are they matched?** A single well-provisioned card, running one model at a time for a handful of concurrent users – that’s the whole case for staying lightweight. A matched cluster of identical NVIDIA cards intended for real concurrent serving – that’s when the distributed-serving design in vLLM and SGLang starts to earn its operational complexity. If you answered “one to three, chat turn, single card” to all three – and if you’ve read this far you probably did – you don’t need vLLM or SGLang. You need llama.cpp, probably wrapped in Ollama for convenience, and you need to stop reading benchmark charts that were never measuring your situation in the first place. We covered getting that stack running well in our post on [building a local RAG pipeline with Ollama and pgvector](https://www.gladlabs.io/posts/from-data-silos-to-smart-answers-building-a-local--735689d4), and the broader case for going local at all is in [our piece on the offline shift in 2026 development](https://www.gladlabs.io/posts/the-offline-revolution-why-local-llms-are-the-back-1a51d7e0). ## Where this actually leaves you The comparison charts treat this like a single winner-take-all question, and that’s the wrong frame. llama.cpp isn’t losing to vLLM and SGLang – it’s not entering the same competition. It’s the tool for one machine serving a handful of requests with minimal fuss, and that description covers most solo developers, most small teams, and honestly most production workloads that aren’t customer-facing chat products with real concurrent traffic. vLLM earns its complexity the moment your concurrency numbers stop being a rounding error – eight, ten, fifty simultaneous requests against one model, where batching turns idle GPU cycles into served tokens. SGLang earns its place when the shape of your work is less “answer a question” and more “run a structured program through a model, over and over, with shared context worth caching.” We looked at our own call logs before deciding, and they said stay put. That’s the actual process worth copying – not “which engine benchmarks fastest,” but “what does my traffic actually look like, this week, on this hardware.” Answer that first. The engine choice falls out of it, and if you’ve put a router in front of your application the way we did, you get to keep answering it as your traffic changes instead of committing to one answer forever.