Inference Optimization for the Rest of Us — KV Cache, Quantization, and Latency Tradeoffs

Most inference optimization content is written for people running fleets of H100s. If you're serving models out of a single GPU box, a modest EC2/ECS setup, or even CPU inference for smaller models, a lot of that advice doesn't transfer — the constraints are different, and the "just add more GPUs" answer isn't available to you.

This post covers the three levers that actually move the needle at small-to-mid scale: KV cache management, quantization, and the latency tradeoffs that come with both. No cluster-scale assumptions.

Inference cost and latency don't show up as a line item until they do — usually right around the point where a side project or an MVP gets real traffic. At that point you have three options: spend more on hardware, spend more on hosted API calls, or actually understand what's eating your latency and memory budget. The third option is the only one that scales your understanding along with your infrastructure.

Every token a transformer generates attends back to all previous tokens. Naively, that means recomputing attention over the whole sequence at every step — wildly wasteful. The KV cache stores the key and value tensors from previous tokens so each new token only needs one forward pass, not a full replay.

The tradeoff: that cache grows linearly with sequence length, and its memory footprint is often the actual bottleneck — not the model weights themselves — once you're serving multiple concurrent requests with long contexts.

A rough sizing formula, per request:

kv_cache_bytes = 2 * num_layers * num_heads * head_dim * seq_len * bytes_per_element

The 2

is for both K and V. For a 7B-class model with 32 layers, 32 heads, head_dim 128, at fp16 (2 bytes) and a 4096-token context, that's already several hundred MB — per concurrent request. Multiply by your target concurrency and you'll see why KV cache, not weights, is usually what determines how many simultaneous users a single GPU can serve.

If you're rolling your own serving layer rather than using vLLM/TGI, this is the single highest-leverage thing to borrow from those projects even if you don't adopt them wholesale.

Quantization reduces the numeric precision of model weights (and sometimes activations), shrinking both memory footprint and, often, latency.

The practical decision isn't "should I quantize" — for anything outside a hyperscaler budget, you almost certainly should. It's which quantization scheme, and that depends on your bottleneck:

A pattern worth adopting: don't guess — benchmark your actual task against 2–3 quantization levels before committing. A model that quantizes cleanly on general chat can degrade noticeably on structured extraction or tool-calling tasks.

// Rough shape of a quant-level config in a gateway/router setup
#[derive(Debug, Clone)]
struct ModelVariant {
    name: String,
    quant: QuantLevel,      // Fp16, Int8, Int4Awq, GgufQ4KM
    vram_gb: f32,
    avg_tokens_per_sec: f32,
}

fn select_variant(task: TaskProfile, variants: &[ModelVariant]) -> &ModelVariant {
    // route structured/tool-calling tasks to higher precision,
    // route casual chat/summarization to the cheapest variant that fits
    variants.iter()
        .filter(|v| task.quality_floor <= v.quant.quality_score())
        .min_by(|a, b| a.vram_gb.partial_cmp(&b.vram_gb).unwrap())
        .unwrap_or(&variants[0])
}

This is the same instinct as multi-provider routing in a gateway — route by task requirements, not a single fixed model for everything.

Once cache and quantization are handled, the remaining latency budget splits into pieces that respond to different fixes:

A concrete tradeoff worth naming explicitly: batching improves throughput but can hurt per-request latency under load, since a request may wait briefly for a batch slot. For a single interactive assistant, that's rarely visible. For a shared production endpoint under real concurrency, it's the difference between p50 and p95 latency looking very different — and it's worth deciding up front which one your use case actually cares about.

For a small-to-mid scale deployment, the order that gets you the most benefit per hour invested:

None of this requires hyperscaler infrastructure. It requires knowing which of these five levers you're actually constrained by before spending money on the wrong fix.

The next post in this series moves back up the stack — open-weight versus closed API models, and a practical cost/control framework for deciding which to run yourself versus call through a provider.

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.