{"slug": "expert-validated-stem-qa", "title": "Expert-validated STEM QA", "summary": "Researchers introduced 'Expert-validated STEM QA', a dataset of 398 expert-created questions in physics, chemistry, biology, and mathematics, built by 241 domain experts to address gaps in existing STEM benchmarks. Frontier AI models scored below 25% on the dataset, while post-training on a private 2,000-question version improved an open-source model's performance by 15% relative to baseline on the HLE-verified STEM subset (p=0.045). The team open-sourced part of the dataset for the AI research community.", "body_md": "arXiv:2608.28591v1 Announce Type: new\nAbstract: Recent advancements in AI are helping scientists achieve breakthroughs in fields such as mathematics, medicine, and materials sciences. New evaluation datasets for AI models contribute to such advancement in AI. In the STEM domain, frontier models have consumed most of the available online data, creating the need for human-created datasets that codify the knowledge of leading experts in the domain. There are several STEM datasets available for the research community in this field. However, there are some gaps in these datasets, leaving room for improvement. Examples of gaps include (1) saturation in model performance on these datasets, leaving no head-room for meaningful evaluations, (2) skewed taxonomy distributions, (3) multiple choice question format that is misaligned with how scientists use AI in the real world, and (4) inaccurate answers and rationales partially led by a contest-based data collection and a time-bound review process. In this study, we present 'Expert-validated STEM QA', a high-quality, expert-validated STEM dataset (N=398) in Physics, Chemistry, Biology, and Mathematics, created by 241 domain experts. We (1) carefully designed a taxonomy with balanced distribution, (2) vetted question contributors with quality-driven incentive, (3) conducted multiple rounds of reviews with revisions validated by domain experts based on consensus, and (4) created the dataset in verifiable question and answer format. Our study demonstrated low performance ($<25\\%$) of frontier AI models on the dataset as a benchmark. Post-training on a separate, private version of the dataset (N=2,000) increased performance of the open source model by $15\\%$ relative to the baseline model (p=0.045) on the STEM subset of HLE-verified dataset, indicating potential utility of the dataset for model training. We have open-sourced a portion of our dataset for the AI research community.", "url": "https://wpnews.pro/news/expert-validated-stem-qa", "canonical_source": "https://arxiv.org/abs/2608.28591", "published_at": "2026-09-01 04:00:00+00:00", "updated_at": "2026-09-01 04:26:22.430921+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research"], "entities": ["Expert-validated STEM QA", "HLE-verified dataset"], "alternates": {"html": "https://wpnews.pro/news/expert-validated-stem-qa", "markdown": "https://wpnews.pro/news/expert-validated-stem-qa.md", "text": "https://wpnews.pro/news/expert-validated-stem-qa.txt", "jsonld": "https://wpnews.pro/news/expert-validated-stem-qa.jsonld"}}