{"slug": "inference-internals-go-public-as-benchmarks-keep-deflating-the-more-effort", "title": "Inference internals go public as benchmarks keep deflating the \"more effort\" reflex", "summary": "Three independent benchmarks show that expensive AI defaults are rarely optimal: medium reasoning effort captures nearly all of Claude Opus 5's bug-fix gains, a three-model open-weight jury (GPT-OSS 120B, DeepSeek-V4 Flash, Gemma-4 31B) matches frontier judges on IMO-GradingBench at up to 100x lower cost, and base LLMs outperform instruction-tuned ones as VLM backbones. Production inference internals were detailed by Baseten, Cloudflare, and a teardown of Kimi K3's linear attention, highlighting KV cache quantization, disaggregated prefill/decode, and cache-aware routing. However, cheaper and more autonomous systems are not uniformly safe: role steering degrades on 14% of personas, and multimodal models fail on raw satellite streams in disaster scenarios.", "body_md": "Today's intel converges on a single engineering posture: the expensive default is rarely the right one. Three independent benchmarks argue the point from different angles — medium reasoning effort captures nearly all of Opus 5's bug-fix gains, a three-model open-weight jury grades math proofs about as well as frontier judges at up to 100x less cost, and base LLMs turn out to make better VLM backbones than instruction-tuned ones, predictable in advance from a textual capability score. Underneath that, the serving layer opened up unusually far, with Baseten, Cloudflare, and a deep read of Kimi K3's linear attention all describing the same set of levers — KV cache quantization, disaggregated prefill/decode, cache-aware routing — and the same emerging friction between linear attention and prefix caching. The counterweight is that cheaper and more autonomous does not mean uniformly safe: role steering degrades on 14% of personas regardless of tuning, and multimodal models still fail on raw, unprocessed satellite streams in disaster scenarios.\nMethod: Reasoning-effort dials are hitting diminishing returns — Medium captures nearly all of Claude Opus 5's resolve-rate gains, and Opus 5 at XHigh loses on every metric to Grok 4.5, GPT-5.6 Sol, and Fable 5.\nMethod: A unanimous three-judge ensemble of open-weight models (GPT-OSS 120B, DeepSeek-V4 Flash, Gemma-4 31B) matches frontier judges on IMO-GradingBench at up to 100x lower cost, making cheap eval pipelines a defensible default.\nRelease: A capability-driven scaling law trained across 150+ VLMs and 34 LLMs predicts multimodal accuracy from a PCA-derived textual capability score — and finds base LLMs outperform instruction-tuned ones as VLM backbones.\nTooling: Production inference internals got unusually legible in one day: Baseten's walkthrough of cache-aware routing and disaggregated prefill/decode, Cloudflare's KV-cache and weight-compression levers for Kimi and GLM, and a teardown of Kimi K3's KDA linear attention and its awkward interaction with prefix caching in vLLM.\nDebate: Agent memory has no free lunch — AgentMemBench finds external key-value retrieval beats windowing, graph episodic memory, and summarization on long-range recall, but at roughly 17x the memory footprint.\nWatch: Two papers argue against uniform agent configuration: 38 of 275 roles degrade under activation steering at any coefficient, and Obshazard-bench shows MLLMs still fail to reason over raw satellite streams in time-sensitive disaster response.\nTooling: The build-vs-adopt question keeps resolving toward open harnesses — Stripe stood up its company-wide agent Kai on Deep Agents in a week, GitHub published measured results for cross-repo agentic doc workflows, and an undocumented path lets Claude Code reach existing Claude Connectors directly.", "url": "https://wpnews.pro/news/inference-internals-go-public-as-benchmarks-keep-deflating-the-more-effort", "canonical_source": "https://www.vibeleaderboard.ai/intel/brief/2026-08-04", "published_at": "2026-08-04 04:57:08+00:00", "updated_at": "2026-08-20 14:44:04.485369+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "ai-safety"], "entities": ["Claude Opus 5", "GPT-OSS 120B", "DeepSeek-V4 Flash", "Gemma-4 31B", "Baseten", "Cloudflare", "Kimi K3", "Stripe"], "alternates": {"html": "https://wpnews.pro/news/inference-internals-go-public-as-benchmarks-keep-deflating-the-more-effort", "markdown": "https://wpnews.pro/news/inference-internals-go-public-as-benchmarks-keep-deflating-the-more-effort.md", "text": "https://wpnews.pro/news/inference-internals-go-public-as-benchmarks-keep-deflating-the-more-effort.txt", "jsonld": "https://wpnews.pro/news/inference-internals-go-public-as-benchmarks-keep-deflating-the-more-effort.jsonld"}}