{"slug": "bioeval-a-global-multi-institutional-benchmark-of-large-language-and-multimodal", "title": "BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering", "summary": "BioEVAL, a global multi-institutional benchmark assembled by 22 research groups, evaluated large language and multimodal models on PhD-level bioengineering tasks and found top accuracy of up to 90% on multiple-choice questions, a 0.72 similarity score on literature synthesis, and 80% accuracy on a small sample of multimodal reasoning questions. The benchmark comprises 608 evaluation items across 11 major bioengineering subfields plus uncategorized items, including 380 multiple-choice questions (359 retained after audit), 218 literature synthesis tasks, and 10 multimodal problems with experimental image interpretation. A blinded cross-group consensus audit flagged 21 of the highest- and lowest-accuracy MCQ items for revision or removal, and all reported MCQ results are computed on the 359 retained items, with substantial performance variation across subfields.", "body_md": "arXiv:2609.30489v1 Announce Type: new \nAbstract: Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing LLM benchmarking emphasizes factual recall, offering limited insight into model performance on frontier and multimodal tasks. We assembled BioEVAL (BioEngineering Validation of AI and LLMs), a global, multi-institutional initiative designed to assess experimental reasoning capability across bioengineering (BE) subfields. BioEVAL spans 11 major BE subfields plus a set of uncategorized items, bringing together 22 research groups to create a PhD-level benchmark comprising 608 evaluation items: 1) 380 multiple-choice questions (MCQs, 359 retained after audit), 2) 218 literature synthesis tasks, and 3) 10 multimodal problems with experimental image interpretation. Benchmark items underwent authoring-group expert review and centralized quality control before evaluation. Following evaluation, a blinded cross-group consensus audit of the highest- and lowest-accuracy MCQ items flagged 21 questions for revision or removal; these were withheld, and all reported MCQ results are computed on the 359 retained items. We evaluated diverse cloud-scale foundation/multimodal models (e.g., ChatGPT, Gemini, and Grok) and locally deployable models suitable for inference on consumer-grade GPUs. Models achieved the highest accuracy of up to 90% on MCQs, similarity score of 0.72 on literature synthesis, and accuracy of 80% on a small sample of multimodal reasoning questions, with substantial performance variation across subfields. Leaderboard rankings characterize current capabilities, limitations, and development priorities across the evaluated BE task categories. BioEVAL is maintained as an extensible benchmark with standardized protocols for continuing expert item contribution and model evaluation.", "url": "https://wpnews.pro/news/bioeval-a-global-multi-institutional-benchmark-of-large-language-and-multimodal", "canonical_source": "https://arxiv.org/abs/2609.30489", "published_at": "2026-09-28 04:00:00+00:00", "updated_at": "2026-09-28 04:19:56.259830+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["BioEVAL", "ChatGPT", "Gemini", "Grok", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/bioeval-a-global-multi-institutional-benchmark-of-large-language-and-multimodal", "markdown": "https://wpnews.pro/news/bioeval-a-global-multi-institutional-benchmark-of-large-language-and-multimodal.md", "text": "https://wpnews.pro/news/bioeval-a-global-multi-institutional-benchmark-of-large-language-and-multimodal.txt", "jsonld": "https://wpnews.pro/news/bioeval-a-global-multi-institutional-benchmark-of-large-language-and-multimodal.jsonld"}}