AIArticle Effort levels won because reasoning depth is trained, not dialed — and your cost model needs to catch up.
Mariana Souza
Sometime in the last eighteen months, without ever announcing it as a coordinated decision, every major LLM vendor killed the numeric thinking budget. Anthropic deprecated budget_tokens
on the Claude 4.6 generation and now rejects it outright — a 400 error — on Opus 4.7/4.8, Sonnet 5, and Fable 5, replacing it with adaptive thinking plus a five-level effort
setting. Google's Gemini 2.5 shipped a literal token dial (thinkingBudget
, 0–24,576 tokens); Gemini 3 replaced it with a discrete thinking_level
, and sending both parameters in one request is a 400. OpenAI never shipped a token knob at all — reasoning_effort started at minimal/low/medium/high on GPT-5 and has since sprouted
xhigh
(first on the codex-max variant) and a full none-through-max range on the newest models.That's three companies with very different API philosophies converging on the same abstraction: a small enum instead of a number. The interesting question is why — and Sebastian Raschka's recent survey of how labs actually train these controls supplies the answer. The effort knob isn't an inference-time setting that meters tokens. It's a trained behavior, and once you understand that, several things about the current API landscape stop being annoying and start being legible.
The knob is a string in the chat template #
Reasoning models get their "thinking" from reinforcement learning with verifiable rewards: sample a chain of thought, check the final answer against a math checker or a test suite, reward accordingly. Crucially, the reasoning trace itself isn't graded — only the answer and the format. Left alone, this training recipe produces models that think a lot, because longer traces raise the odds of a correct answer and nothing in the reward penalizes verbosity.
Effort control is bolted on during that same training, not afterwards. The recipe, common across the open-weight models where you can actually inspect it: put a plain-text tag like Reasoning effort: low
in the system prompt, and apply a per-token length penalty during RL whose harshness depends on the tag. High effort gets a gentle penalty, low effort a steep one. The model learns three or four distinct reasoning styles keyed to a string. In gpt-oss you can see this directly — the "API parameter" is literally a line in the Harmony system prompt. Qwen3 does a variant with /think
and /no_think
soft switches learned during fine-tuning, backed by a hard switch where the tokenizer force-closes an empty <think></think>
block when enable_thinking=False
. DeepSeek-V4 trains separate non-think, think-high, and think-max modes with distinct length penalties per mode. This explains the discreteness. The model only knows the effort levels it was trained on; there is no meaningful "0.7" between medium and high because no gradient ever taught it one. (Research systems with continuously conditioned effort exist — Raschka covers one trained across a 0.2–0.99 range — but nothing shipping commercially works that way, and I'd treat that claim as promising-but-unreplicated for now.) It also explains why vendors dropped the token number: a budget cap was always a lie of precision. The model didn't know about your 8,192-token budget mid-trace; the runtime just truncated or the model overshot. An effort level is honest about what's actually controllable.
What you lost, and where it went #
Be clear-eyed about the trade: a numeric budget was a cap, and an effort level is a disposition. You've exchanged hard cost control for better-calibrated behavior. The compensating machinery moved elsewhere — max_tokens
remains the enforced ceiling everywhere, and Anthropic added a separate task-budget primitive for agentic loops where the model sees a countdown and paces itself. If you were using budget_tokens
as a spend guardrail, that's the migration path: effort for quality, output caps for cost.
Meanwhile the labs are attacking token spend on their side. Moonshot's Kimi K2.5 trains with a "Toggle" heuristic — alternating unconstrained RL with phases where the model must solve problems inside a per-problem budget estimated from the token counts of known-correct solutions — and reports 25–30% fewer output tokens at negligible quality cost. Expect this everywhere: overthinking is now a training bug, not your problem to prompt around.
What to actually do with this #
Three practical consequences for anyone shipping on these APIs:
Treat effort as per-route config, and sweep it. Defaults diverge wildly: Claude defaults to high
(Claude Code runs xhigh
), GPT-5 defaults to medium
, Gemini defaults to dynamic thinking that decides per request. The same "just swap the model string" migration can double your latency or quietly under-think a hard route. Put effort in your eval harness as a swept dimension — the cost/quality curve is not monotonic in your favor, and every vendor's own guidance now admits diminishing returns at the top setting.
Recalibrate your intuition downward. Because effort levels are trained against each model's own capability, low
on a current-generation model frequently matches high
on the previous one. The reflexive "max everything for production" setting mostly buys you latency and overthinking on routine work. The pattern that holds up: one level down for subagents and mechanical tasks, top levels reserved for genuinely intelligence-bound routes.
Mind the max_tokens interaction. Thinking tokens count against your output ceiling on every one of these APIs. A limit tuned for a non-reasoning model will truncate a high-effort response mid-answer — you'll see a response that's mostly reasoning followed by a cut-off conclusion. If you raise effort, raise the ceiling.
If you self-host, you get the whole mechanism for free: the knob is in the chat template, so chat_template_kwargs={"thinking": False}
in vLLM or SGLang, or an effort line in the system prompt for gpt-oss, does exactly what the hosted APIs do — because it's the same string the model was trained on.
My read: this is a genuine and durable shift, not API fashion. Token budgets were an implementation detail that leaked into the interface during the first year of reasoning models, and the enum is the correct abstraction for a control that's fundamentally behavioral. The cost is that "how much will this request cost me" got fuzzier at exactly the moment reasoning tokens became the dominant line item — which is why the effort knob belongs in your config file and your eval matrix, not hardcoded next to the model name.
Sources & further reading #
[Controlling Reasoning Effort in LLMs](https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms)— magazine.sebastianraschka.com -
[Reasoning models](https://developers.openai.com/api/docs/guides/reasoning)— developers.openai.com -
[Gemini thinking](https://ai.google.dev/gemini-api/docs/thinking)— ai.google.dev -
[Effort parameter](https://platform.claude.com/docs/en/build-with-claude/effort)— platform.claude.com -
[Kimi K2.5 model card](https://huggingface.co/moonshotai/Kimi-K2.5)— huggingface.co -
[DeepSeek-V4-Pro model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)— huggingface.co
[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.