Your Context Length Decides What a Kernel Is Worth

A 2× faster attention kernel yields only 0.66% end-to-end speedup on a 1,024-token prompt with a 128-token answer under vLLM's --goodput ttft:500 tpot:50 promise, according to a seven-part technical series by an unnamed author. The analysis, based on NVIDIA's blog post showing attention's share of prefill time rising from 18% at 4K context to 85% at 128K on DeepSeek-R1, demonstrates that the kernel's value depends heavily on context length, model architecture, and phase (prefill vs. decode), with Amdahl's law capping returns at 1.10× for 4K and 1.74× for 128K prefill.

Part seven of seven. What is left of a kernel win by the time it reaches the top of the stack. Somebody hands you an attention kernel that is twice as fast. Whether that is worth 9% of a prefill or 42.5% is not a fact about the kernel. Six parts of this series climbed the stack. This one follows a single win back up it, to see how much arrives. Amdahl’s law. The 1 − p term never moves. That sets a ceiling of 1/ 1−p no matter how large s gets, so a part worth 18% of the time can never return more than 1.22×, and a part worth 85% can return 6.67×. Five times the difference, from one number that has nothing to do with the code. NVIDIA’s developer blog on long-context attention publishes a breakdown in which attention’s share rises from 18% to 85% as context grows from 4K to 128K . It is a blog post, not a paper https://developer.nvidia.com/blog/co-designing-ai-model-attention-for-fast-interactive-long-context-inference . The conditions: prompt-reading phase only , on DeepSeek-R1 , with FP8 attention arithmetic and an FP8 KV cache. No batch size and no hardware. Two of those do more work than the rest. The measurement is prefill, so it says nothing about generation, where part one showed the arithmetic runs the other way. And DeepSeek-R1 attends over the whole sequence on every layer, with no sliding window to cap it, so at 128K the cost that grows with context lands on all of them. gpt-oss-120b, the model this series has computed on since part two, is built differently. Grouped-query, sliding-window attention on half its layers. A window caps the growth on the layers that have one, which should pull attention’s share down. It also activates fewer parameters per token, which makes the work attention is measured against cheaper, and pulls the share back up. Nobody has published which effect wins. So “attention is 85% of inference time” is not a fact about inference. It is a fact about one model, in one phase, measured once. At 4K, p = 0.18 and a 2× kernel returns 1 / 0.82 + 0.09 , or 1.10× . At 128K, p = 0.85 and the same kernel returns 1 / 0.15 + 0.425 , or 1.74× . Prefill only, and the ceiling column is what no kernel can pass. Gold is a 2× kernel. The dashed line is a kernel of infinite speed. Decode is a different regime. Part one put decode attention at an arithmetic intensity of about 8 against a ridge of 296, deeply bandwidth-bound, which means the card is waiting on memory rather than on arithmetic. A kernel that computes twice as fast waits just as long. Two things do help. Moving fewer bytes, which is part two's --kv-cache-dtype fp8. Or getting nearer the card's peak rate, which is what a split-KV decode kernel does when a small batch leaves most of the card idle. Raw arithmetic is the one thing that does not. The kernel is fixed at 2× for every frame. Only the workload moves. Prefill is part of a request, not the whole of it, so the 1.74× has one more dilution to survive. Part five tells you what to promise: --goodput ttft:500 tpot:50, a first token inside 500 ms and every token after it inside 50. Take part five's own default shape, a 1,024-token prompt and a 128-token answer. A request that exactly meets that promise gets its first token at 500 ms and spends 127 × 50 = 6,350 ms on the rest. vLLM measures the per-token promise across the gaps between tokens, so 128 tokens leave 127 gaps. That 500 ms is not all prefill. Time to first token also covers queueing, tokenization and the first decode step, so prefill is one slice of it. Call the whole thing prefill anyway and it is 500 of 6,850 ms. Seven percent of the request, and the honest number is smaller. Run the kernel win through that. The only attention share anybody publishes is the 18%, and it came off a different model at four times this prompt length. Both of those flatter the kernel. Borrow it regardless: it takes 9% off the 500 ms, which is 45 ms, and against 6,850 ms that is 0.66% . The wedge is the same 500 ms in both bars. A benchmark says 2×. The prefill arithmetic says 1.10×. The request says 0.66%. That same arithmetic cannot be run at 128K, because it needs a prefill time nobody publishes. So the 42.5% stops where prefill stops. What it leaves in a whole request is a number nobody has put in public. A tokens-per-second figure is not comparable on its own. Ten things decide what it means. Prefix caching and the token count are the two that go missing quietly. The last pair is what turns throughput into goodput , which part five called the only number that maps to money: a latency target, and the fraction of requests that met it. A server at full throughput and 40% goodput is failing while looking busy. The quietest condition is which tokens got counted. Prompt tokens are processed in parallel and generated tokens are not, so a figure counting both can dwarf one counting only what the model generated. On part five’s default shape that multiplier is 1,152 over 128, or nine times, with nobody lying anywhere in the process. Concurrency is on the list for part one's reason, that batch size moves a workload along the roofline. Precision is there for part four's, prefix caching for part five's, engine version because defaults move between releases. I have yet to read a published figure that carries all ten. Prefix caching deserves its own warning. A fixed system prompt with repeated inputs measures cache hits rather than computation. Not the model. The cache. A benchmark measures the top of the stack from outside. Tokens per second, time to first token, tail latency: all of them say a number moved, none of them say which of the six layers moved it. Profiling reads downward, and two kinds answer different halves of the question. A timeline profiler shows where the time went across a whole request. vLLM emits PyTorch traces if the server is started with them enabled: vllm serve <model --profiler-config '{"profiler": "torch", "torch profiler dir": "./vllm profile"}' That flag only says where traces land. Bracket the requests worth looking at with POST /start profile and POST /stop profile, or llm.start profile offline. Nsight Systems goes deeper through nsys profile, and vLLM's docs suggest --trace-fork-before-exec=true --cuda-graph-trace=node with VLLM WORKER MULTIPROC METHOD=spawn, because the default fork method confuses the tracer. Check those names against the version you are using. This interface moved recently, and VLLM TORCH PROFILER DIR, still the first hit on most forums, is gone from v0.27.1 entirely. A healthy trace is dense. The bad ones have a shape that is visible before any number is read: gaps between kernels where the GPU idled waiting on the scheduler, rows of launches too short to be worth their own launch overhead, and long flat bars where one card waited on another. Attention’s total against the step’s total is p, and it is the only number in the trace that decides what to do next. A kernel profiler answers the second question. Nsight Compute takes one kernel and places it against the memory roof from part one, drawing that roofline rather than leaving it to be computed. Worth running once p says the kernel matters, and not before. Say p comes back at 0.2. That ends the argument: the ceiling is 1.25×, a doubling returns 1.11×, and the request dilutes that again. The work is at another layer. One profiler run bought you that, and it is the difference between rewriting a kernel and leaving it alone. The speedup you were quoted was measured somewhere you do not run. That number is probably fine. Somebody ran it and wrote it down. What they could not write down was p, because p was never a property of the kernel. It belongs to the context lengths and the traffic, and whoever wrote the kernel saw neither. Six parts of this series have hit the same wall. Part two sized a cache at 128K, a context length I picked to make a point rather than one your traffic runs at. The accuracy deltas in part four came off benchmarks picked by whoever ran them. And part five tuned its defaults on traffic belonging to somebody else. Real numbers, all of them, and every one stopped being true somewhere between where it came from and where you are. So check where a number came from before repeating it. Your own included. Most of the figures in this series came off hardware that is not mine either, which is why every part carries a note at the bottom saying so. The attention-share figures come from NVIDIA’s developer blog on long-context attention. They cover the prompt-reading phase only. Your Context Length Decides What a Kernel Is Worth https://pub.towardsai.net/your-context-length-decides-what-a-kernel-is-worth-3011334ef830 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.