What the demo actually showed #
They ran Llama-3-70B on identical H100 hardware — once through a baseline HF generate loop, once through TensorRT-LLM with in-flight batching, paged attention, and FP8 quantization. The optimized path delivered 4.2× higher throughput at half the latency. Same weights, same silicon. The harness extracted performance the raw model couldn't reach on its own.
This isn't a one-off. vLLM's continuous batching, SGLang's radix cache, and TensorRT-LLM's kernel fusion all solve the same problem: the gap between what a model can do theoretically and what it does in production. That gap is now the entire ballgame.
Why the model became a commodity #
Open weights caught up. Llama-3, Qwen2, Nemotron — pick your 70B, they're all within a few MMLU points. Fine-tunes for coding, RAG, or function calling are abundant. The differentiation shifted upstream: who can serve the model cheapest, fastest, and with the longest context window without OOMing.
The harness handles:
KV cache management— paged attention, prefix caching, off to CPU/NVMe** Scheduling**— in-flight batching, chunked prefill, priority queues** Quantization**— FP8, AWQ, GPTQ calibration pipelines that don't tank accuracy** Kernel selection**— fused MHA, flash attention variants, custom cutlass kernels per GPU arch
None of this lives in the model weights. All of it lives in the serving stack.
The practical takeaway for deployment #
If you're still treating the model as the decision variable, you're optimizing the wrong layer. A 7B model on a tuned vLLM deployment often outperforms a 70B on a naive HF pipeline for real-world latency budgets. The harness lets you run smaller models faster, or larger models at all. Start benchmarking runtimes, not checkpoints. Profile your actual workload — batch sizes, sequence lengths, concurrency targets — against TensorRT-LLM, vLLM, SGLang, and TGI. The winner changes per use case. That's the new evaluation loop.
The model is table stakes. The harness is the moat.
Nvidia AVO hits 100% on ARC-AGI-3 and the benchmark might be 7h ago Nvidia's compute asset class push hits $500 billion — here's why 23h ago
[Cerebras WSE-3 smokes H100 on Llama 3 70B inference at a 2d ago](/en/news/6962/)
[Nvidia's buyback binge looks smart until you check the 2d ago](/en/news/6956/)
PantheonGPU proves that telemetry alone is a lie for GPU health 3d ago
Groq is spending billions to poach Nvidia engineers 3d ago Next Built a visual canvas that lets you branch AI conversations like →
these AI tool field notes, with plenty of directly applicable cases.