Random vs scaffold splits: single-seed baselines on 6 chemistry datasets

A forum discussion on Hugging Face's community site examined single-seed random-versus-scaffold split baselines across six chemistry datasets, reporting that FreeSolv RMSE rose from 1.82 under random splits to 4.95 under scaffold splits and that one model's AUC fell from 0.965 to 0.817. Commenters urged the author, identified as @woongworks, to run 5-seed or 10-seed repeats with standard deviations so confidence intervals can separate consistently harder scaffold splitting from a hard partition, and to release the evaluation scripts. The thread recommends reporting random-split results as an in-distribution upper bound and scaffold-split results as an out-of-scaffold lower bound for chemistry ML benchmarks.

This is exactly the kind of baseline the field needs more of — thank you for running it systematically across six datasets, @woongworks https://discuss.huggingface.co/u/woongworks . A few thoughts on extending this: Single-seed variance: The single-seed approach clearly shows the direction of the effect, but a 5-seed or 10-seed run with standard deviations would make the results more citable. The FreeSolv RMSE jump from 1.82 → 4.95 is striking, but some of that variance could be from the specific scaffold partition chosen. Confidence intervals would separate “scaffold splitting is consistently harder” from “this particular partition was hard.” Implications for synthetic dataset design: This finding generalizes well beyond chemistry. When designing synthetic evaluation sets for any domain, the split strategy should match deployment reality. A model that scores 0.965 AUC under random splits but 0.817 under scaffold splits is overfit to known scaffolds — which matters enormously for prospective deployment on novel compounds. The QM9 case is interesting: The smaller gap there might be because QM9 properties are more determined by local functional groups than scaffold topology — suggesting split-strategy sensitivity depends on what the label captures , not just the molecular representation. Practical recommendation: For any published chemistry ML benchmark, reporting random-split results as “upper bound in-distribution ” and scaffold-split results as “lower bound out-of-scaffold generalization ” frames them as a meaningful range rather than forcing a single choice. Would you be open-sourcing the evaluation scripts? Would be useful infrastructure for the community to run on new datasets.