Member-only story
FP8 KV cache, prompt caching, quantization, speculative decoding, MoE routing. The five 2026 levers between your model and your invoice, ranked by which one to pull first. #
Read the article for free
[.]here There is a layer between your model and your invoice. It has five levers. Pull the wrong ones and your bill is 10x what it should be. Pull the right ones in the right order and it is 10x less.
Take a bill a five-engineer team on Sonnet 4.6 has seen. Around $4,800 for one month of Claude Code sessions, agent turns, and tool-heavy calls.
The Sonnet line item is $960 of it. The rest **$3,840 **lives in a layer most engineers do not touch.
The layer got a lot of new shape in the first half of 2026. In April, vLLM shipped FP8 KV cache, storing attention memory in 8-bit floats instead of 16, which adds about 15% throughput on Llama-3.1-8B
. In March, Ollama shipped MLX, Apple's tensor framework tuned for M-series Macs, which nearly doubled Qwen3.5-35B
token generation speed on the same M5 Max. DeepSeek's prompt cache, which reuses precomputed attention state when your prompt prefix repeats, started running at 120x cheaper per…