{"slug": "do-language-models-know-what-compelling-scientific-evidence-looks-like", "title": "Do Language Models Know What Compelling Scientific Evidence Looks Like?", "summary": "A new arXiv paper (2610.41015v1) introduces EDB-Bio, a scientific reasoning benchmark of 96,000 episodes drawn from real data in drug-perturbation transcriptomics, CRISPR screens and human genome-wide association studies, asking models to predict the probability that a target hypothesis is true. Out-of-the-box open-weight and frontier models showed systematic underconfidence and Brier scores comparable to a trivial base-rate forecaster, but post-training against strictly proper scoring rules improved calibration and discrimination, and the resulting Ibis0 model was 2.5 times more efficient than the best frontier model in a sequential experiment selection task. The authors release all code, data and trained models, framing the work as extending calibration training and uncertainty analysis to realistic scientific forecasting with applications to human cancer biology.", "body_md": "arXiv:2610.41015v1 Announce Type: cross \nAbstract: Expert scientists excel at weighing empirical findings from multiple sources that vary in both statistical reliability and epistemic relevance. To what extent are language models, which are increasingly driving autonomous discovery systems, capable of such evidence deliberation? We introduce EDB-Bio, a scientific reasoning benchmark curated from real-world data in three biological domains: drug-perturbation transcriptomics, CRISPR screens, and human genome-wide association studies (GWAS). Each of EDB-Bio's 96k episodes presents a sequence of experimental results from published studies and asks the model to predict the probability that a target hypothesis is true. Out-of-the-box, open-weight and frontier models alike demonstrate systematic underconfidence and attain Brier scores comparable to a trivial base-rate forecaster. Post-training against strictly proper scoring rules significantly improves both calibration and discrimination. This calibration pays off downstream: in a sequential experiment selection task, our Ibis0 model is 2.5 times more efficient than the best frontier model. Throughout, quantifying evidence weights as Bayes factors reveals how models update their beliefs over time, whether they agree with expert biologists on relevance, and where they exhibit inconsistencies and biases. Our work extends calibration training and uncertainty analysis to a realistic scientific forecasting setting with applications to human cancer biology. We release all code, data, and trained models.", "url": "https://wpnews.pro/news/do-language-models-know-what-compelling-scientific-evidence-looks-like", "canonical_source": "https://www.machinebrief.com/news/do-language-models-know-what-compelling-scientific-evidence-k3hu", "published_at": "2026-11-02 05:00:00+00:00", "updated_at": "2026-11-02 06:20:53.688018+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "large-language-models", "ai-safety"], "entities": ["EDB-Bio", "Ibis0", "arXiv", "CRISPR", "GWAS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/do-language-models-know-what-compelling-scientific-evidence-looks-like", "markdown": "https://wpnews.pro/news/do-language-models-know-what-compelling-scientific-evidence-looks-like.md", "text": "https://wpnews.pro/news/do-language-models-know-what-compelling-scientific-evidence-looks-like.txt", "jsonld": "https://wpnews.pro/news/do-language-models-know-what-compelling-scientific-evidence-looks-like.jsonld"}}