# What Is Inference Engineering? The Layer Doing 80% of Your LLM Bill.

> Source: <https://pub.towardsai.net/what-is-inference-engineering-the-layer-doing-80-of-your-llm-bill-9bb536ecc31d?source=rss----98111c9905da---4>
> Published: 2026-07-16 15:01:04+00:00

Member-only story

# What Is Inference Engineering? The Layer Doing 80% of Your LLM Bill.

## FP8 KV cache, prompt caching, quantization, speculative decoding, MoE routing. The five 2026 levers between your model and your invoice, ranked by which one to pull first.

Read the article for free

[.]here

There is a layer between your model and your invoice. It has five levers. Pull the wrong ones and your bill is 10x what it should be. Pull the right ones in the right order and it is 10x less.

Take a bill a five-engineer team on Sonnet 4.6 has seen. Around **$4,800** for one month of Claude Code sessions, agent turns, and tool-heavy calls.

The Sonnet line item is **$960** of it. The rest **$3,840 **lives in a layer most engineers do not touch.

The layer got a lot of new shape in the first half of 2026. In April, vLLM shipped FP8 KV cache, storing attention memory in 8-bit floats instead of 16, which adds about 15% throughput on `Llama-3.1-8B`

. In March, Ollama shipped MLX, Apple's tensor framework tuned for M-series Macs, which nearly doubled `Qwen3.5-35B`

token generation speed on the same M5 Max. DeepSeek's prompt cache, which reuses precomputed attention state when your prompt prefix repeats, started running at 120x cheaper per…