# FINAL-Bench finds a 0.21 AUROC gap between time and random splits > Source: > Published: 2026-08-22 16:46:31+00:00 # FINAL-Bench finds a 0.21 AUROC gap between time and random splits **VIDRAFT's new leaderboard uses hidden labels, harder test sets and assay noise floors to make drug-prediction scores less flattering and more useful.** By [RuntimeWire Staff](/author/runtimewire-staff) ยท Published Primary source: [Hugging Face Community](https://huggingface.co/blog/FINAL-Bench/leadboard-drug) ## Why it matters Drug-prediction gains can disappear when test sets resemble training data or assay noise exceeds the score gap. FINAL-Bench makes both problems visible before declaring a winner. VIDRAFT CEO Min-sik Kim has released [FINAL-Bench's LEADBOARD](https://huggingface.co/spaces/FINAL-Bench/leadboard?ref=runtimewire), a 21-board test for drug-property prediction tools built around a result that surprised the project: changing how the data was divided moved the same baseline model from 0.606 to 0.818 AUROC. [FINAL-Bench's Hugging Face announcement](https://huggingface.co/blog/FINAL-Bench/leadboard-drug?ref=runtimewire) documents the comparison. VIDRAFT described the result in an [August 22nd community article](https://huggingface.co/blog/FINAL-Bench/leadboard-drug?ref=runtimewire) published through Hugging Face. Hugging Face hosts the article and leaderboard; LEADBOARD is a FINAL-Bench and VIDRAFT project. The benchmark covers seven disciplines, with 212,670 training compounds and 18,382 held-out compounds whose labels are withheld from participants. ([huggingface.co](https://huggingface.co/blog/FINAL-Bench/leadboard-drug?ref=runtimewire)) Kim is an unusual public face for a scientific benchmarking project. On [VIDRAFT's website](https://vidraft.net/index.html?lang=en&ref=runtimewire), he says he is 28 and paused a computer-engineering degree at Korea Cyber University to build a scientific research company. The site credits him as a named inventor on 12 patent applications and a co-author of four papers, claims that come from VIDRAFT. His stated ambition is larger than a leaderboard: he wants commercial products and intellectual property to finance a Korean scientific institution modeled on the Max Planck Society. ([vidraft.net](https://vidraft.net/index.html?lang=en&ref=runtimewire)) LEADBOARD gives that ambition a practical starting point. Drug-discovery AI has plenty of reported benchmark wins, while differences in datasets, splits and laboratory measurements can make those wins difficult to compare. Kim's project is trying to standardize the ruler before building more systems that claim to beat it. ### The flattering split FINAL-Bench began with hERG, a potassium-channel liability screened early in drug development because blocking it can create dangerous heart-rhythm problems. FINAL-Bench extracted 12,021 hERG records with document years from ChEMBL 37, retained 11,972 after unit checks and parsing, and reduced those to 9,788 unique compounds. The first experiment trained a default LightGBM model on Morgan fingerprints from compounds reported before 2022, then tested it on compounds reported later. It scored 0.606 AUROC. When FINAL-Bench randomly split the same dataset, the model averaged 0.818 across five seeds, with individual results ranging from 0.803 to 0.830. The gap was 0.211 AUROC. ([huggingface.co](https://huggingface.co/blog/FINAL-Bench/leadboard-drug?ref=runtimewire)) The model, fingerprints and hyperparameters stayed fixed. The split changed what the model was being asked to do. Medicinal chemistry produces families of closely related compounds as researchers modify a promising molecular scaffold. A random split can scatter those analogues across the training and test sets, allowing the model to evaluate compounds that closely resemble examples it has already seen. A chronological split asks the model to predict later measurements from an earlier snapshot of available knowledge. FINAL-Bench does not claim that published drug benchmarks universally use random splits. Scaffold splits are common and create a different level of difficulty. Its narrower finding is still consequential: a leaderboard score becomes difficult to interpret when the underlying split is unspecified. Every open LEADBOARD task therefore uses a time split or a scaffold split. Random-split boards are excluded. ### The assay can disagree with itself The second design choice came from the labels. FINAL-Bench compared hERG measurements for the same compounds when those measurements appeared in different publications. Across 5,185 pairs covering 857 compounds, the median absolute difference was 0.148 log units and the mean was 0.475. At the 90th percentile, measurements differed by 1.338 log units, or more than 20-fold. FINAL-Bench converted that disagreement into an estimated single-measurement standard deviation of 0.421 log units and calls it the board's noise floor. That number appears beside model scores so users can see when a small ranking difference is narrower than the experimental uncertainty underneath it. ([huggingface.co](https://huggingface.co/blog/FINAL-Bench/leadboard-drug?ref=runtimewire)) The noise also weakens the hERG classification task. FINAL-Bench says 696 of 1,338 test compounds sit within one noise floor of the pIC50 = 5.0 cutoff. A new measurement could move many of those compounds across the class boundary, so LEADBOARD treats regression as the primary metric and classification as secondary. The same check produced an uncomfortable result across the broader benchmark. Among 19 regression boards, a constant predictor produced the lowest mean absolute error on seven. On the chronological hERG split, the default LightGBM model recorded an error of 0.599, slightly worse than the constant baseline's 0.589. Under the random split, LightGBM's error improved to 0.457 while the constant baseline measured 0.671. That comparison suggests the fingerprint model retained useful ranking information while struggling to estimate absolute potency for newer chemistry. It also explains why FINAL-Bench publishes three baselines before submissions open: a constant prediction, nearest-neighbor lookup and Morgan fingerprints with default LightGBM settings. ### A leaderboard designed to resist its users According to [FINAL-Bench's August 22 article](https://huggingface.co/blog/FINAL-Bench/leadboard-drug?ref=runtimewire), participants download test sets containing molecular structures, generate predictions and upload a two-column CSV containing `compound_id` and `prediction` . The project says scoring runs off-platform on hardware holding the labels, which are absent from the public Hugging Face Space. Every score carries the SHA of the test file and scoring code used for that evaluation, according to the same article. FINAL-Bench also limits the information revealed through repeated submissions. Its design draws on Avrim Blum and Moritz Hardt's [Ladder method](https://arxiv.org/abs/1502.04585?ref=runtimewire), which addresses the risk that teams gradually overfit a hidden test set by submitting many versions and learning from each score. [FINAL-Bench says](https://huggingface.co/blog/FINAL-Bench/leadboard-drug?ref=runtimewire) LEADBOARD reveals a new result only when it beats the participant's previous best by more than the board's noise floor; otherwise, it shows the prior best again. The surrounding field already includes mature resources. [Therapeutics Data Commons](https://tdcommons.ai/?ref=runtimewire) provides therapeutic datasets, evaluation tools, splits and leaderboards through an open Python library. [Polaris](https://polarishub.io/?ref=runtimewire) distributes drug-discovery datasets and benchmarks developed with contributors from pharmaceutical and biotechnology companies. FINAL-Bench's contribution is its combination of hidden labels, time or scaffold splits, trivial published baselines, assay-specific noise floors and restricted score feedback. ([tdcommons.ai](https://tdcommons.ai/?ref=runtimewire)) The central 0.211 result still rests on FINAL-Bench's own analysis. A strong independent replication will need the complete preprocessing path, the precise split implementation, the time-split seed treatment and uncertainty estimates around the reported scores. The announcement provides five random-seed results and one chronological result, enough to expose the size of the issue without settling how broadly the exact gap generalizes. That restraint suits Kim's larger bet. VIDRAFT's website repeatedly frames verification as part of the product rather than paperwork added after a model ships. LEADBOARD applies that view to drug prediction by making impressive scores earn their context. For a young research company trying to span AI and scientific discovery, building the measuring instrument first is a credible place to begin.