# Same Model, Two Speeds: A Friendly Tour of LLM Inference Engines

> Source: <https://dev.to/lovestaco/same-model-two-speeds-a-friendly-tour-of-llm-inference-engines-2ccj>
> Published: 2026-08-22 18:00:44+00:00

*Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.*

Quick experiment. Open ChatGPT, ask it something, and watch the words spill out.

Now do the same in Gemini.

Notice anything?

The text does not always arrive at the same pace.

One feels like it is sprinting, the other feels like it is thinking out loud over coffee.

Here is the part that trips people up: that speed is not really about which model is "smarter."

A big chunk of it comes from the thing running *underneath* the model.

The plumbing. The engine. And once you see how that plumbing works, a lot of mysterious LLM behavior suddenly makes sense.

Now let's walk through it.

We are going to talk about inference, why the same model can run at wildly different speeds, and the clever operating-system trick that made modern LLM serving actually affordable.

When people say training, they mean teaching the model.

Inference is the other half: the model is done learning, and now it has to actually *do the job* and generate tokens for you.

Every time you hit send, you are paying for inference.

We measure how fast this goes in **tokens per second**.

That is the speedometer for the whole conversation.

Higher number, snappier feel.

But generation is not one big magic step.

It happens in two phases, and understanding them is the key to everything else.

See that loop? That is where most of the time goes. And that little box called the **KV cache** is the secret villain of this whole story.

Every token the model has seen so far gets summarized into two vectors, a Key and a Value.

Storing them means the model does not have to recompute all that work for every single new token.

Without the cache, generating token number 500 would mean redoing the math for the previous 499 tokens. Ouch.

So the cache is great.

The catch is that it is *hungry* and it *grows*.

The longer the conversation, the bigger it gets, and it lives on the GPU where memory is precious and expensive.

For a 70B model, the KV cache for a single long request can run into tens of gigabytes.

Multiply that by a room full of users and you see the problem.

Now here is where older serving systems fell flat.

Because nobody knows in advance how long your answer will be, the naive approach was to reserve memory for the worst case, the maximum possible length, for *every* request.

Ask for a haiku? Congrats, the system still booked you the memory of a novel, just in case.

That leads to two flavors of waste, and both have proper names worth knowing:

How bad was it? The vLLM team measured that older systems wasted roughly [60 to 80 percent of KV cache memory](https://blog.vllm.ai/2023/06/20/vllm.html). That is not a rounding error. That is most of your very expensive GPU sitting idle while you pay for it.

Here is the "aha" that makes this fun. The vLLM folks at UC Berkeley looked at this mess and realized it was not a new problem at all.

Operating systems solved almost exactly this decades ago with **virtual memory and paging**.

Think about how your OS handles RAM.

Your program thinks it has one long, tidy stretch of memory.

In reality, the OS chops memory into fixed-size pages scattered all over the place, and a **page table** keeps track of which logical page maps to which physical spot.

Your program never notices. It just sees a clean, continuous view.

[PagedAttention](https://arxiv.org/abs/2309.06180) does the same thing to the KV cache.

Instead of one giant contiguous block per request, it slices the cache into small fixed-size **blocks** (16 tokens each by default).

Blocks get handed out on demand as the answer grows, and they can live *anywhere* in the memory pool.

A **block table** plays the role of the page table, mapping each logical block of a sequence to whatever physical block it landed in.

The payoff is huge.

Since blocks are tiny and allocated only when needed, the only waste left is the half-empty last block of each sequence, at most 15 unused token slots.

Memory waste drops from that ugly 60 to 80 percent down to [under 4 percent](https://blog.vllm.ai/2023/06/20/vllm.html).

Pack the memory tighter, fit more requests at once, and your throughput jumps 2 to 4 times on the exact same hardware.

No new GPUs required.

That is basically free money, which in GPU-land is a rare and beautiful thing.

Paging Dr. Memory, indeed.

Efficient memory is only useful if you actually put it to work.

This is where **continuous batching** comes in, and it pairs with PagedAttention like chips and salsa.

Old-school batching would group a bunch of requests, run them together, and wait for the whole batch to finish before starting the next one.

Problem: requests finish at different times.

The short ones sit there twiddling their thumbs while the GPU waits on the one guy asking for a 2000-word essay.

GPU utilization tanks.

Continuous batching is smarter.

The moment one request in the batch finishes, its slot is freed and a new waiting request slides right in.

The GPU never gets to nap.

Because PagedAttention makes it cheap to add and drop sequences on the fly (just update a block table, no giant memory shuffles), the two techniques together are what let a single box serve a genuinely useful number of concurrent users.

Okay, so vLLM sounds like the hero. And it is a fantastic default.

But if you walk away thinking "just use vLLM for everything," you have missed the real lesson.

vLLM is optimized for one specific thing: **high throughput when you are serving many users at once**.

That is a great goal for a public API.

It is not the *only* goal, and other engines exist because other goals matter.

Let me introduce the rest of the lineup, because this is a genuine ecosystem and each member earned its seat:

The honest takeaway is that the workload picks the engine, not a star rating on some leaderboard infographic.

Ask where it runs, how many users hit it, what your latency target is, and whether your requests look alike.

If you take three things away from our little chat, make it these:

The next time an LLM streams an answer at you, you will know there is a whole storage-Tetris tournament happening behind the curtain, packing blocks and juggling batches so your tokens show up fast.

Not bad for something you never see.

If you want to go deeper, the [vLLM project on GitHub](https://github.com/vllm-project/vllm) is very readable, and the original [PagedAttention paper](https://arxiv.org/abs/2309.06180) is worth a slow read on a rainy afternoon.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star it on GitHub:

| [🇩🇰 Dansk](https://github.com/HexmosTech/git-lrc/readme/README.da.md) | [🇪🇸 Español](https://github.com/HexmosTech/git-lrc/readme/README.es.md) | [🇮🇷 Farsi](https://github.com/HexmosTech/git-lrc/readme/README.fa.md) | [🇫🇮 Suomi](https://github.com/HexmosTech/git-lrc/readme/README.fi.md) | [🇯🇵 日本語](https://github.com/HexmosTech/git-lrc/readme/README.ja.md) | [🇳🇴 Norsk](https://github.com/HexmosTech/git-lrc/readme/README.nn.md) | [🇵🇹 Português](https://github.com/HexmosTech/git-lrc/readme/README.pt.md) | [🇷🇺 Русский](https://github.com/HexmosTech/git-lrc/readme/README.ru.md) | [🇦🇱 Shqip](https://github.com/HexmosTech/git-lrc/readme/README.sq.md) | [🇨🇳 中文](https://github.com/HexmosTech/git-lrc/readme/README.zh.md) | [🇮🇳 हिन्दी](https://github.com/HexmosTech/git-lrc/readme/README.hi.md) |

GenAI today is a **race car without brakes**. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents *silently break things*: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.

** git-lrc is your braking system.** It hooks into

`git commit`

and runs an AI review on every diff In short, git-lrc helps **Prevent Outages, Breaches, and Technical Debt Before They Happen**

**At a glance:** [10 risk categories](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · [100+ failure patterns tracked](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · every commit…