This is exactly the kind of baseline the field needs more of — thank you for running it systematically across six datasets, @woongworks.
A few thoughts on extending this:
Single-seed variance: The single-seed approach clearly shows the direction of the effect, but a 5-seed or 10-seed run with standard deviations would make the results more citable. The FreeSolv RMSE jump from 1.82 → 4.95 is striking, but some of that variance could be from the specific scaffold partition chosen. Confidence intervals would separate “scaffold splitting is consistently harder” from “this particular partition was hard.”
Implications for synthetic dataset design: This finding generalizes well beyond chemistry. When designing synthetic evaluation sets for any domain, the split strategy should match deployment reality. A model that scores 0.965 AUC under random splits but 0.817 under scaffold splits is overfit to known scaffolds — which matters enormously for prospective deployment on novel compounds.
The QM9 case is interesting: The smaller gap there might be because QM9 properties are more determined by local functional groups than scaffold topology — suggesting split-strategy sensitivity depends on what the label captures, not just the molecular representation.
Practical recommendation: For any published chemistry ML benchmark, reporting random-split results as “upper bound (in-distribution)” and scaffold-split results as “lower bound (out-of-scaffold generalization)” frames them as a meaningful range rather than forcing a single choice.
Would you be open-sourcing the evaluation scripts? Would be useful infrastructure for the community to run on new datasets.