RL Contributes to Illegible Reasoning Traces in Hard Problems

A study of 14 reasoning models found that outcome-based reinforcement learning (RL) often produces illegible chain-of-thought (CoT) reasoning, with accuracy dropping by 53% when models are forced to use only legible portions, and legibility degrading on harder questions. The researchers, who submitted the paper to arXiv on 31 Oct 2025, found no correlation between legibility and performance when resampling, suggesting the relationship is nuanced, and they propose hypotheses including steganography, training artifacts, and vestigial tokens. The findings indicate that without explicit optimization for legibility, RL naturally yields models with increasingly opaque reasoning, potentially undermining monitoring approaches.

Computer Science Machine Learning Submitted on 31 Oct 2025 Title:Reasoning Models Sometimes Output Illegible Chains of Thought View PDF /pdf/2510.27338 HTML experimental https://arxiv.org/html/2510.27338v1 Abstract:Language models trained via outcome-based reinforcement learning RL to reason using chain-of-thought CoT have shown remarkable performance. Monitoring such a model's CoT may allow us to understand its intentions and detect potential malicious behavior. However, to be effective, this requires that CoTs are legible and faithful. We study CoT legibility across 14 reasoning models, finding that RL often causes reasoning to become illegible to both humans and AI monitors, with reasoning models except Claude generating illegible CoTs while returning to perfectly readable final answers. We show that models use illegible reasoning to reach correct answers accuracy dropping by 53\% when forced to use only legible portions , yet find no correlation between legibility and performance when resampling - suggesting the relationship is more nuanced. We also find that legibility degrades on harder questions. We discuss potential hypotheses for these results, including steganography, training artifacts, and vestigial tokens. These results suggest that without explicit optimization for legibility, outcome-based RL naturally produces models with increasingly opaque reasoning processes, potentially undermining monitoring approaches. References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender IArxiv Recommender What is IArxiv? https://iarxiv.org/about arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .