[Submitted on 13 Oct 2026 (
[v1](https://arxiv.org/abs/2610.18651v1)), last revised 14 Oct 2026 (this version, v2)]
[View PDF](https://arxiv.org/pdf/2610.18651)
[HTML (experimental)](https://arxiv.org/html/2610.18651v2)
Abstract:Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.
Submission history #
From: Yoel Zimmermann [
[view email](https://arxiv.org/show-email/c3eb3f64/2610.18651)]
**Tue, 13 Oct 2026 16:14:35 UTC (555 KB)**
[\[v1\]](https://arxiv.org/abs/2610.18651v1)
**[v2]** Wed, 14 Oct 2026 17:58:29 UTC (555 KB)
Additional Features
Current browse context:
cs.LG
Change to browse by:
References & Citations
...
Bibliographic Explorer
(What is the Explorer?) Connected Papers
(What is Connected Papers?) Litmaps
(What is Litmaps?) scite Smart Citations
(What are Smart Citations?) alphaXiv
(What is alphaXiv?) CatalyzeX Code Finder for Papers
(What is CatalyzeX?) DagsHub
(What is DagsHub?) Gotit.pub
(What is GotitPub?) Hugging Face
(What is Huggingface?) ScienceCast
(What is ScienceCast?) Influence Flower
(What are Influence Flowers?) CORE Recommender
(What is CORE?) IArxiv Recommender
(What is IArxiv?) arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.