What Is Inference Engineering? The Layer Doing 80% of Your LLM Bill.

Inference engineering, the layer between an LLM and its invoice, accounts for 80% of costs, with five key levers—FP8 KV cache, prompt caching, quantization, speculative decoding, and MoE routing—that can reduce bills by 10x when pulled correctly. A five-engineer team on Sonnet 4.6 saw a $4,800 monthly bill, with only $960 from the model itself and the rest from inference overhead. Recent 2026 advances like vLLM's FP8 KV cache and DeepSeek's prompt caching are reshaping this layer.

Member-only story What Is Inference Engineering? The Layer Doing 80% of Your LLM Bill. FP8 KV cache, prompt caching, quantization, speculative decoding, MoE routing. The five 2026 levers between your model and your invoice, ranked by which one to pull first. Read the article for free . here There is a layer between your model and your invoice. It has five levers. Pull the wrong ones and your bill is 10x what it should be. Pull the right ones in the right order and it is 10x less. Take a bill a five-engineer team on Sonnet 4.6 has seen. Around $4,800 for one month of Claude Code sessions, agent turns, and tool-heavy calls. The Sonnet line item is $960 of it. The rest $3,840 lives in a layer most engineers do not touch. The layer got a lot of new shape in the first half of 2026. In April, vLLM shipped FP8 KV cache, storing attention memory in 8-bit floats instead of 16, which adds about 15% throughput on Llama-3.1-8B . In March, Ollama shipped MLX, Apple's tensor framework tuned for M-series Macs, which nearly doubled Qwen3.5-35B token generation speed on the same M5 Max. DeepSeek's prompt cache, which reuses precomputed attention state when your prompt prefix repeats, started running at 120x cheaper per…