FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

Researchers introduced FlowBalance, a method that improves reasoning models by combining sparse terminal verifier supervision with on-policy experience, addressing the fragility of self-improvement loops. The approach aims to prevent overconcentration on narrow solution modes and false confidence reinforcement.

A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance,