# Inference and self-hosting

> Running models yourself: GPU memory, quantization, KV cache, serving engines, throughput and latency tuning, hardware choices.
> 6 recommendations, newest first. Canonical: https://wpnews.pro/playbooks/inference-serving

- Offload parked KV caches to CPU memory rather than recomputing prefill; the source measured reload over PCIe at about 50 GB/s versus 1.5-2.5 s to recompute a 32K-token prefill.  
  — [When the KV Cache Spills Over PCIe](https://wpnews.pro/news/when-the-kv-cache-spills-over-pcie), source: <https://pub.towardsai.net/when-the-kv-cache-spills-over-pcie-996781bd7658?source=rss----98111c9905da---4>, 2026-11-17
- Budget capacity with training as the dominant load: Anthropic put 74% of 2026 compute into training versus 26% into inference, so plan serving headroom accordingly.  
  — [74% Of Anthropic’s Compute Has Gone Towards Training In 2026, Only 26% To Inference: Brad Gerstner](https://wpnews.pro/news/74-of-anthropics-compute-has-gone-towards-training-in-2026-only-26-to-inference), source: <https://officechai.com/ai/74-of-anthropics-compute-has-gone-towards-training-in-2026-only-26-to-inference-brad-gerstner/>, 2026-11-17
- Budget VRAM for context growth: llama3.2:3b Q4_K_M held 351.22 tok/s at 4,096 context but dropped to 102.68 tok/s at 131,072 on a 16 GB RTX 5080.  
  — [The 16 GB Cliff: Measured LLM Memory and Speed on an RTX 5080](https://wpnews.pro/news/the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080), source: <https://techfuelhq.com/data/rtx-5080-16gb-vram-cliff/>, 2026-11-17
- Before downloading a model, check that parameters × 0.56 GB fits your fast memory and that bandwidth divided by active bytes per token clears your tokens-per-second floor, since KV cache and slower memory pools cut real speed to about a third of the ceiling.  
  — [How to Calculate VRAM and Speed Before Running an LLM Locally](https://wpnews.pro/news/how-to-calculate-vram-and-speed-before-running-an-llm-locally), source: <https://www.mindstudio.ai/blog/how-to-estimate-llm-vram-requirements/>, 2026-11-17
- Size local deployments by HBM, not total memory: budget ~0.56 GB per billion parameters at 4.5-bit and reserve KV-cache headroom, since overflow into slower unified memory cuts token speeds.  
  — [HP ZGX Fury Benchmarked: Local LLM Speeds for DeepSeek, GLM, Qwen](https://wpnews.pro/news/hp-zgx-fury-benchmarked-local-llm-speeds-for-deepseek-glm-qwen), source: <https://www.mindstudio.ai/blog/hp-zgx-fury-local-llm-benchmarks/>, 2026-11-17
- Size the VLLM context window to the actual task instead of defaulting to 256K; the test cut memory below the ~130GB full-context footprint, and most agentic workflows don't need the full window.  
  — [Maxwell V1.0 Flash Tested Locally: Does This 35B MoE Model Hold Up?](https://wpnews.pro/news/maxwell-v1-0-flash-tested-locally-does-this-35b-moe-model-hold-up), source: <https://www.mindstudio.ai/blog/maxwell-v1-flash-moe-local-test/>, 2026-11-17
