That knob still works. But it's not the interesting one anymore.
The real question teams are fighting with now is: given a model you already have, how much compute should it spend per question? Not per training run. Per question. And the answer turns out to break almost every assumption your inference stack was built on.
Here's the thing nobody says out loud enough: a model's first answer is often just a noisy sample. The right answer was in there somewhere — it just wasn't the loudest one. You don't fix that with more pretraining tokens. You fix it by sampling more, checking the work, or letting the model take a second pass.
That's the whole idea behind test-time compute. And there are three ways to spend that extra compute, and they are not interchangeable:
Most real systems end up doing all three — cheap by default, escalating only when a confidence check says "this one's hard." That escalation decision is its own piece of architecture. Most teams bolt it on as an afterthought. It shouldn't be one.
Continuous batching, paged KV-cache, speculative decoding — all of that infra assumes requests are independent, similar in length, and finish in one pass.
Test-time compute violates every one of those assumptions:
If you've built speculative decoding before, forking KV-cache across search branches will feel familiar. It's the same "shared prefix, divergent suffix" problem — just stretched over a much longer horizon, with a lot more at stake if you get the cache management wrong.
Scaling laws told us how to spend money before a model ever met a user. Test-time compute asks the harder question: how do you spend money per question, in real time, without quietly teaching your search process to fool its own judge?
That's not a prompting trick. It's inference architecture with an alignment problem sitting inside it — and it deserves the same rigor we've spent five years pouring into pretraining.