Does Pretraining Teach Chemical Language Models Chemistry?

A study by Yoel Zimmermann posted to arXiv on 13 October 2026 finds that pretraining chemical language models (CLMs) does not scale and that downstream performance saturates below gradient-boosting baselines across 64 molecular property prediction datasets. Only supervised pretraining on traditional physico-chemical descriptors consistently improved downstream performance for the CLMs tested, while some forms of pretraining produced models worse than a randomly initialized encoder. The authors conclude that CLMs resolve molecular structure no better than a randomly initialized model and learn little chemistry beyond what classical molecular features already capture.

Computer Science Machine Learning Submitted on 13 Oct 2026 v1 https://arxiv.org/abs/2610.18651v1 , last revised 14 Oct 2026 this version, v2 Title:Does Pretraining Teach Chemical Language Models Chemistry? View PDF https://arxiv.org/pdf/2610.18651 HTML experimental https://arxiv.org/html/2610.18651v2 Abstract:Chemical language models CLMs are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture. Submission history From: Yoel Zimmermann view email https://arxiv.org/show-email/c3eb3f64/2610.18651 Tue, 13 Oct 2026 16:14:35 UTC 555 KB \ v1\ https://arxiv.org/abs/2610.18651v1 v2 Wed, 14 Oct 2026 17:58:29 UTC 555 KB Additional Features Current browse context: cs.LG Change to browse by: References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender IArxiv Recommender What is IArxiv? https://iarxiv.org/about arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .