What objective is reasoning solving?
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Reinforcement learning maximizes the probability of a model finding any path to a correct answer, not the probability of individual paths converging to a correct answer, which means longer chain generation doesn't necessarily improve accuracy. This matters because it implies that models may not improve with more computation per chain, limiting the effectiveness of simply increasing chain length. Models may still improve with more samples, as they can find alternative paths to the correct answer.
Reasoning RL is better understood as shifting probability mass toward token paths that end in verifiably correct answers, not making any single chain-of-thought converge as it gets longer. For production systems, forcing longer reasoning traces is a weak control knob and can degrade outputs; invest instead in sampling multiple paths, using verifiers/rerankers, and treating CoT text as an unreliable artifact rather than a faithful execution trace.