On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

A paper posted to arXiv (2610.32770v1) presents "on-policy self-training," a framework in which a foundation model improves itself using only coarse, high-level topic descriptions, generating its own question-answer pairs and privileged reasoning demonstrations without human-curated prompts, answers, or ground-truth data. The authors report consistent self-improvement on standard language benchmarks and Visual Question Answering across text and multimodal domains, while the model retained capabilities on domains it was not self-trained on. The work claims foundation models hold latent capabilities that can be amplified without external task supervision or curated problem distributions.

arXiv:2610.32770v1 Announce Type: new Abstract: On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.