{"playbook": "inference-serving", "title": "Inference and self-hosting", "description": "Running models yourself: GPU memory, quantization, KV cache, serving engines, throughput and latency tuning, hardware choices.", "url": "https://wpnews.pro/playbooks/inference-serving", "count": 6, "recommendations": [{"advice": "Offload parked KV caches to CPU memory rather than recomputing prefill; the source measured reload over PCIe at about 50 GB/s versus 1.5-2.5 s to recompute a 32K-token prefill.", "takeaway": "For Llama 3.1 70B at 32K tokens, offloading the KV cache to CPU memory and reloading it over PCIe at about 50 GB/s is more than 50 times faster than recomputing prefill, and even a 6.2 GB/s SSD tier is at least 7 times faster.", "article_url": "https://wpnews.pro/news/when-the-kv-cache-spills-over-pcie", "title": "When the KV Cache Spills Over PCIe", "source_url": "https://pub.towardsai.net/when-the-kv-cache-spills-over-pcie-996781bd7658?source=rss----98111c9905da---4", "published_at": "2026-11-17T21:01:01+00:00"}, {"advice": "Budget capacity with training as the dominant load: Anthropic put 74% of 2026 compute into training versus 26% into inference, so plan serving headroom accordingly.", "takeaway": "Anthropic allocated 74% of its 2026 compute to training and 26% to inference, per Brad Gerstner, indicating training workloads still dominate its capacity over serving costs.", "article_url": "https://wpnews.pro/news/74-of-anthropics-compute-has-gone-towards-training-in-2026-only-26-to-inference", "title": "74% Of Anthropic’s Compute Has Gone Towards Training In 2026, Only 26% To Inference: Brad Gerstner", "source_url": "https://officechai.com/ai/74-of-anthropics-compute-has-gone-towards-training-in-2026-only-26-to-inference-brad-gerstner/", "published_at": "2026-11-17T20:12:43+00:00"}, {"advice": "Budget VRAM for context growth: llama3.2:3b Q4_K_M held 351.22 tok/s at 4,096 context but dropped to 102.68 tok/s at 131,072 on a 16 GB RTX 5080.", "takeaway": "On a 16 GB RTX 5080, llama3.2:3b Q4_K_M used 3,760 MiB of 16,303 MiB and decoded at 351.22 tok/s at 4,096 context but fell to 102.68 tok/s at 131,072 context.", "article_url": "https://wpnews.pro/news/the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080", "title": "The 16 GB Cliff: Measured LLM Memory and Speed on an RTX 5080", "source_url": "https://techfuelhq.com/data/rtx-5080-16gb-vram-cliff/", "published_at": "2026-11-17T17:00:00+00:00"}, {"advice": "Before downloading a model, check that parameters × 0.56 GB fits your fast memory and that bandwidth divided by active bytes per token clears your tokens-per-second floor, since KV cache and slower memory pools cut real speed to about a third of the ceiling.", "takeaway": "Estimate VRAM as parameters in billions times 0.56 GB at ~4.5 bits per weight, and estimate decode speed as memory bandwidth divided by active weight bytes per token, expecting real single-user throughput near a third of that ceiling.", "article_url": "https://wpnews.pro/news/how-to-calculate-vram-and-speed-before-running-an-llm-locally", "title": "How to Calculate VRAM and Speed Before Running an LLM Locally", "source_url": "https://www.mindstudio.ai/blog/how-to-estimate-llm-vram-requirements/", "published_at": "2026-11-17T00:00:00+00:00"}, {"advice": "Size local deployments by HBM, not total memory: budget ~0.56 GB per billion parameters at 4.5-bit and reserve KV-cache headroom, since overflow into slower unified memory cuts token speeds.", "takeaway": "On the HP ZGX Fury, keep models inside the 252 GB of HBM rather than the 496 GB unified memory, budget about 0.56 GB per billion parameters at 4.5-bit quantization, and expect GLM 5.3 Flash to serve 16 concurrent users at close to 80 tokens per second each.", "article_url": "https://wpnews.pro/news/hp-zgx-fury-benchmarked-local-llm-speeds-for-deepseek-glm-qwen", "title": "HP ZGX Fury Benchmarked: Local LLM Speeds for DeepSeek, GLM, Qwen", "source_url": "https://www.mindstudio.ai/blog/hp-zgx-fury-local-llm-benchmarks/", "published_at": "2026-11-17T00:00:00+00:00"}, {"advice": "Size the VLLM context window to the actual task instead of defaulting to 256K; the test cut memory below the ~130GB full-context footprint, and most agentic workflows don't need the full window.", "takeaway": "Maxwell V1.0 Flash needs about 130GB of VRAM at its full 256K context under VLLM, so size the context window to the task to cut memory; it fixed a hidden Flask bug unaided but failed a creative front-end code task and a live web research job with mismatched citations.", "article_url": "https://wpnews.pro/news/maxwell-v1-0-flash-tested-locally-does-this-35b-moe-model-hold-up", "title": "Maxwell V1.0 Flash Tested Locally: Does This 35B MoE Model Hold Up?", "source_url": "https://www.mindstudio.ai/blog/maxwell-v1-flash-moe-local-test/", "published_at": "2026-11-17T00:00:00+00:00"}]}