# Nvidia's latest demo proves the inference stack matters more > Source: > Published: 2026-08-21 21:26:19+00:00 # Nvidia's latest demo proves the inference stack matters more ## What the demo actually showed They ran Llama-3-70B on identical H100 hardware — once through a baseline HF generate loop, once through TensorRT-LLM with in-flight batching, paged attention, and FP8 quantization. The optimized path delivered 4.2× higher throughput at half the latency. Same weights, same silicon. The harness extracted performance the raw model couldn't reach on its own. This isn't a one-off. vLLM's continuous batching, SGLang's radix cache, and TensorRT-LLM's kernel fusion all solve the same problem: the gap between what a model *can* do theoretically and what it *does* in production. That gap is now the entire ballgame. ## Why the model became a commodity Open weights caught up. Llama-3, Qwen2, Nemotron — pick your 70B, they're all within a few MMLU points. Fine-tunes for coding, [RAG](/en/tags/rag/), or function calling are abundant. The differentiation shifted upstream: who can serve the model cheapest, fastest, and with the longest context window without OOMing. The harness handles: **KV cache management**— paged attention, prefix caching, offloading to CPU/NVMe** Scheduling**— in-flight batching, chunked prefill, priority queues** Quantization**— FP8, AWQ, GPTQ calibration pipelines that don't tank accuracy** Kernel selection**— fused MHA, flash attention variants, custom cutlass kernels per GPU arch None of this lives in the model weights. All of it lives in the serving stack. ## The practical takeaway for deployment If you're still treating the model as the decision variable, you're optimizing the wrong layer. A 7B model on a tuned vLLM deployment often outperforms a 70B on a naive HF pipeline for real-world latency budgets. The harness lets you run smaller models faster, or larger models at all. Start benchmarking runtimes, not checkpoints. Profile your actual workload — batch sizes, sequence lengths, concurrency targets — against TensorRT-LLM, vLLM, SGLang, and TGI. The winner changes per use case. That's the new evaluation loop. The model is table stakes. The harness is the moat. [Nvidia AVO hits 100% on ARC-AGI-3 and the benchmark might be 7h ago](/en/news/7169/) [Nvidia's compute asset class push hits $500 billion — here's why 23h ago](/en/news/7102/) [Cerebras WSE-3 smokes H100 on Llama 3 70B inference at a 2d ago](/en/news/6962/) [Nvidia's buyback binge looks smart until you check the 2d ago](/en/news/6956/) [PantheonGPU proves that telemetry alone is a lie for GPU health 3d ago](/en/news/6849/) [Groq is spending billions to poach Nvidia engineers 3d ago](/en/news/6810/) [Next Built a visual canvas that lets you branch AI conversations like →](/en/news/7214/) [these AI tool field notes](https://tanyan888.com/), with plenty of directly applicable cases.