LLM Inference Efficient Frontier: Baseten Latency vs Throughput Map Baseten published an updated "efficient frontier of LLM inference" article on November 2, 2026, authored by Philip Kiely, that separates inference optimizations into moves along the latency–throughput curve (batch size, TP vs EP vs ADP parallelism, aggressive quantization) and moves that push the whole curve outward (kernel/runtime improvements, speculative decoding, P/D disaggregation). Kiely assumes agentic coding traffic on models such as GLM-5.3 or Kimi K3 with KV cache reuse and KV-aware routing already enabled, and notes that two independent 2x improvements, one hardware and one software, compound to roughly 4x. The framing warns that vendor benchmarks often mix a kernel win with tiny batch settings, so buyers compare incomparable configurations. Baseten's efficient frontier of LLM inference https://www.baseten.co/blog/the-efficient-frontier-of-llm-inference/?ref=explainx article — updated November 2, 2026 — gives serving teams a vocabulary wall chart: some tricks slide you along the latency–throughput curve, while others push the whole curve outward so you can spend new headroom on speed, cost, or both. Philip Kiely assumes agentic coding traffic on models like GLM-5.3 or Kimi K3 with KV cache reuse and KV-aware routing already enabled — the same buyer persona explainx.ai covered in Baseten DeepSeek Ultrafast https://explainx.ai/blog/baseten-deepseek-ultrafast-2060-tps-rust-inference-engine-2026 and Grid Router cost claims https://explainx.ai/blog/baseten-grid-router-72-percent-coding-cost-cut-opus-5-5-claimed-vs-verified-2026 . If you operate vLLM , SGLang , or a vendor API, this post is the conceptual layer above raw tokens-per-second brags — the map behind Qwen drafter speculative decoding https://explainx.ai/blog/qwen3-8-27b-dspine-b16-open-drafter-sglang-speculative-decoding-2026 and Luminal compiler fundraising https://explainx.ai/blog/luminal-25m-series-a-inference-compiler-multi-chip-2026 . TL;DR | Technique class | Effect on frontier | Examples per Baseten | |---|---|---| | Trade along curve | Swap latency ↔ throughput or quality ↔ cost | Batch size , TP vs EP vs ADP , aggressive quant | | Push curve outward | More efficiency to allocate | Kernel/runtime , speculative decoding , P/D disaggregation | | Reality check | Empirical, jagged | Sweeps beat intuition; small config changes jump latency cliffs | Three tradeoffs behind the frontier Baseten's article names three separate tradeoffs, not one. Latency versus throughput drives cost per token. Quality versus throughput covers quantization, distillation, and pruning. Intelligence versus speed covers how much reasoning effort a request spends. Keep the three apart when you read a benchmark, because a single headline number can hide a move on any of them. Baseten also notes that gains compound. Two independent 2x improvements, one from hardware and one from software, give roughly a 4x gain overall. That is why stacking a kernel win, a quantization step, and speculative decoding can look dramatic even when each step is modest. Two kinds of inference engineering moves Economists use efficient frontiers to show best achievable blends of two goods under constraints. Baseten maps that to LLM serving : 1. Targeting a point on the frontier — accept worse per-user latency to save money on batch jobs, or sacrifice throughput to make a copilot feel snappy. 2. Expanding the frontier — make each token cheaper in GPU-seconds , then decide whether to bank savings as lower price or faster decode . That distinction matters because vendor benchmarks often mix both: a kernel win true frontier push gets marketed with tiny batch settings a point pick , and buyers compare incomparable apples. LLM inference efficient frontier chart maps latency against throughput for agent serving teams Moves along the frontier tradeoffs Batch sizing Continuous batching removed wait-for-batch-start latency, but configured batch caps still set how many sequences share a forward pass. Baseten's framing: - Small batches → great per-user latency , high cost per token . - Large batches → worse interactive latency, better throughput per dollar . Agent products split here: IDE inline complete wants small effective batches; nightly repo audits want large ones on the same cluster if you partition queues . Parallelism strategy Large models shard across GPUs. Baseten summarizes: | Mode | Baseten guidance latency-sensitive | Throughput angle | |---|---|---| | Tensor Parallel TP | Favors low latency on NVLink | Costly all-to-all but fast at small batch | | Expert Parallel EP | Moderate EP can help latency; wide EP boosts throughput racks | | | Attention Data Parallel ADP | Replicates attention for throughput, hurts per-request speed | | This is why two DGX tensor-parallel guides https://explainx.ai/blog/glm-5-3-two-dgx-stations-vllm-tensor-parallel-753b-nvfp4-607-tokens-per-second-guide-2026 obsess over NVLink topology while neocloud ads sell wide EP SKUs. Quantization quality tradeoffs Quantization can push the serving frontier because FP8/MXFP4/NVFP4 cuts memory bandwidth. But it also introduces a second axis : quality vs efficiency . Baseten warns that curve is jagged — some precision drops barely move MMLU-class scores https://explainx.ai/blog/goodharts-law-ai-benchmark-contamination-gsm1k-mmlu-arena-2026 while others crater coding reliability. Operators should A/B on their prompts, not trust a vendor slide for Tetris HTML emit only. Moves that push the frontier outward Kernel and runtime optimization CUDA kernels and full forward-pass schedules that shave microseconds per layer compound across thousands of tokens . Baseten points readers to intern Brian Li's kernel writeup https://www.baseten.co/blog/agentic-kernels-in-production/?ref=explainx linked from the article — the same family of work as DeepSeek Ultrafast Rust servers . When Together compiler agents https://explainx.ai/blog/together-compiler-agents-optimize-inference-1111-tps-minimax-1205-tps-deepseek-claimed-vs-verified-2026 claim automated kernel search, they are selling frontier pushes , not just batch retuning. Speculative decoding Early speculation traded extra forward passes for uncertain wins. Baseten's November 2026 update emphasizes EAGLE-3 https://www.baseten.co/blog/how-to-train-custom-eagle-3-heads-for-speculative-decoding/?ref=explainx , DSpark , and DFlash https://www.baseten.co/blog/dflash-faster-llm-inference/?ref=explainx on code generation — long, somewhat predictable emits — where acceptance rates fund both user-visible tok/s and GPU utilization . Match that mental model to SGLang consortium https://explainx.ai/blog/sglang-consortium-lmsys-inference-summit-november-2026 releases and open drafter weights https://explainx.ai/blog/qwen3-8-27b-dspine-b16-open-drafter-sglang-speculative-decoding-2026 you can self-host. Prefill/decode disaggregation P/D disaggregation splits prefill and decode workers so each pool sizes for input-heavy vs output-heavy traffic. Baseten notes it often raises throughput while holding latency flat or slightly better — ideal when agent reads huge repos but emits moderate patches . Hyperscalers bake similar ideas into Snowflake Cortex training previews https://explainx.ai/blog/snowflake-cortex-training-preview-serverless-sft-rl-open-weight-models-aws-what-we-know-2026 adjacent to serving SKUs. Baseten efficient frontier inference techniques separate batch tradeoffs from kernel and speculative decoding wins How to apply the map this week 1. Label your traffic modes — interactive Claude Code-class sessions vs batch eval ; never tune one knob for both without separate deployments . 2. Run sweeps, not vibes — Baseten stresses jagged frontiers ; log p50/p95 latency and $/1M tokens across batch 1, 4, 16, 64 before you promise SLAs. 3. Separate vendor claims — ask whether a benchmark moved along the curve bigger batch or pushed it new kernel . Our claimed-vs-verified Grid Router piece https://explainx.ai/blog/baseten-grid-router-72-percent-coding-cost-cut-opus-5-5-claimed-vs-verified-2026 shows why. 4. Pair with economics posts — if Nvidia financing tightens https://explainx.ai/blog/nvidia-open-checkbook-kress-financing-pullback-claimed-vs-verified-2026 , efficiency per dollar matters more than peak tok/s . Worked example: agentic coding session shapes the frontier pick Imagine a Claude Code loop reading a 40k-token repo then emitting a 2k-token patch : | Phase | Dominant cost | Frontier knob | |---|---|---| | Prefill on huge context | Attention memory bandwidth | Quant , kernel , disaggregated prefill pool | | Decode on patch | Per-step latency | Small batch , higher TP , speculative drafting | | Follow-up turns | KV hits | KV-aware routing , RDMA pull patterns like Ultrafast | Baseten assumes KV reuse already works. If your router sends every tool call to a cold GPU , you are off the frontier entirely — no batch tuning fixes 100% cache miss . That is why MCP-heavy fitness connectors https://explainx.ai/blog/connect-myfitnesspal-apple-health-strava-to-claude-chatgpt-mcp-2026 and long-context science blogs https://explainx.ai/blog/openai-gpt-6-astra-standard-model-lattice-nathaniel-craig-science-blog-2026 should be modeled as different SLAs on one invoice. Common misreads when shopping vendors | Marketing phrase | Often means | Ask for | |---|---|---| | 2,000 tok/s | Single benchmark prompt, thinking off | Your tool-loop trace replayed | | 70% cheaper than Opus | Router sends easy hops to small models | Quality eval on your repo | | FP8 everywhere | Weights quant, activations maybe not | Per-layer loss on your JSON schema tasks | | Disaggregated | Separate autoscaling groups | Queue depth metrics per phase | explainx.ai's Together compiler claims https://explainx.ai/blog/together-compiler-agents-optimize-inference-1111-tps-minimax-1205-tps-deepseek-claimed-vs-verified-2026 post showed the same parsing discipline — frontier language is meaningless without workload labels . When not to chase the frontier Pushing kernels yields diminishing returns if: - Egress to your vector DB dominates wall clock, not decode . - Human review gates every agent PR — time-to-merge is people-bound . - Compliance mandates specific regions with older GPUs — you may be constraint-bound , not optimization-bound . In those cases, move along the frontier: accept higher latency on nightly jobs and spend engineering on data governance https://explainx.ai/blog/fed-november-2026-financial-stability-report-ai-bank-exposure-trillion-2026 instead of MXFP4 . Quick operator checklist printable Before the next capacity reservation renewal, walk this list tied to Baseten's categories: 1. Label traffic — interactive agent vs batch eval vs embedding refresh ; never one autoscaler for all three. 2. Measure jagged cliffs — step batch caps \{1,2,4,8,16,32\} on a frozen prompt set ; plot p95 decode vs $/1M tokens . 3. Log speculation — acceptance rate , draft model cost , wasted forwards ; kill speculation when tool JSON is unpredictable. 4. Audit quant — run schema-heavy evals after MXFP4 ; coding agents care about bracket matching , not MMLU . 5. Revisit monthly — frontier kernels ship weekly in late 2026; Inkling https://explainx.ai/blog/inkling-2-small-thinking-machines-594b-closed-weights-apache-next-week-2026 -class models change memory footprints overnight. Checklist complete means you know whether you are sliding along or pushing outward — the vocabulary Baseten wants normalize. Glossary tie-in for staff meetings Use these phrases so finance and infra align: - Frontier move — after a change, both p95 latency and throughput/$ improve at the same batch target . - Slide move — one metric improves only because you cranked batch or cut quant bits . - Second-axis trade — quality eval falls when you win $/token — common on JSON tool tasks. When Snowflake Cortex training https://explainx.ai/blog/snowflake-cortex-training-preview-serverless-sft-rl-open-weight-models-aws-what-we-know-2026 and inference share a single FinOps dashboard , mislabeled moves create bad capacity forecasts — the hidden cost of hype-grade benchmarks . For platform engineers shipping internal model APIs , publish a one-page frontier diagram per quarter in your internal wiki — link kernel release notes , batch policy , and quality regression dashboards so product managers do not confuse cheaper with smarter . Invite finance to the review: latencies and $/token belong on the same slide as gross margin , especially when Nvidia financing tightens https://explainx.ai/blog/nvidia-open-checkbook-kress-financing-pullback-claimed-vs-verified-2026 and reserved GPUs stop acting like free depreciation on someone's balance sheet. Baseten Inference Engineering book The article closes by pointing to Kiely's free Inference Engineering book https://www.baseten.co/inference-engineering/?ref=explainx for depth on each bullet. Treat the blog as index card ; use the book when you implement disaggregation routing rules or speculative acceptance telemetry. Related on explainx.ai Technique summaries follow Baseten's post as updated November 2, 2026. Measure your own frontier before production cutovers.