Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

Reasoning language models fail to strategically allocate a shared test-time compute budget across multiple questions, according to a new arXiv preprint (arXiv:2608.07968v1). The study introduces an exam-style framework and finds that models act as greedy sequential solvers, prioritizing questions by presentation order and ignoring difficulty and point value, with the issue worsening as question count grows. This global budget allocation capability is not captured by per-question evaluations and remains a challenge for current reasoning models.

arXiv:2608.07968v1 Announce Type: new Abstract: Reasoning language models increasingly use test-time compute to improve performance, but existing evaluations typically study this compute one question at a time. Yet when multiple problems share an end-to-end cost or latency constraint, models must decide how to divide limited inference compute among them. We introduce an exam-style evaluation framework for studying this setting, in which a model must distribute one shared token budget across questions with different difficulty and point values to maximize its total score. Across several open and frontier reasoning models, we find that models fail to allocate a shared budget strategically across questions of varying difficulties and values. Models behave largely as greedy sequential solvers: they prioritize questions by presentation order, front-load effort on early questions, and remain insensitive to value, with these tendencies becoming more pronounced as the number of questions grows. Explicit planning prompts spread compute more evenly but do not produce value- or difficulty-aware prioritization. The same behavioral pattern extends from mathematical to code reasoning. These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation and remains a challenge for current reasoning models.