{"slug": "openais-healthbench-in-action-evaluating-an-llm-based-medical-assistant-on", "title": "OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries", "summary": "OpenAI's HealthBench benchmark shows DR. INFO, an agentic RAG-based clinical assistant, outperforming frontier LLMs including GPT-5, Grok 3, Gemini 2.5 Pro, and Claude 3.7 Sonnet on realistic clinical queries, achieving a HealthBench Hard score of 0.68 versus GPT-5's 0.46. In a separate evaluation against similar assistants OpenEvidence and DoxGPT, DR. INFO scored 0.72, demonstrating strengths in communication and accuracy while highlighting areas for improvement in context awareness.", "body_md": "arXiv:2509.02594v3 Announce Type: replace-cross\nAbstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios. Traditional evaluations are often limited to multiple-choice questions that fail to capture essential competencies such as contextual reasoning, contextual awareness, and uncertainty handling.\nTo address these limitations, we evaluate our agentic RAG-based clinical support assistant, DR. INFO, using HealthBench, a rubric-driven benchmark composed of open-ended, expert-annotated health conversations. On the Hard subset of 1,000 challenging examples, DR. INFO achieves a HealthBench Hard score of 0.68, outperforming leading frontier LLMs including the GPT-5 model family (GPT-5: 0.46, GPT-5.2: 0.42, GPT-5.1: 0.40), Grok 3 (0.23), Gemini 2.5 Pro (0.19), and Claude 3.7 Sonnet (0.02) across all behavioral axes (accuracy, completeness, instruction following, etc.). In a separate 100-sample evaluation against similar agentic RAG assistants (OpenEvidence and Pathway.md, now DoxGPT by Doximity), it maintains a performance lead with a HealthBench Hard score of 0.72.\nThese results highlight the strengths of DR. INFO in communication, instruction following, and accuracy, while also revealing areas for improvement in context awareness and response completeness. Overall, the findings underscore the utility of behavior-level, rubric-based evaluation for building reliable and trustworthy AI-enabled clinical support systems.", "url": "https://wpnews.pro/news/openais-healthbench-in-action-evaluating-an-llm-based-medical-assistant-on", "canonical_source": "https://www.machinebrief.com/news/openais-healthbench-in-action-evaluating-an-llm-based-medica-tifx", "published_at": "2026-07-28 04:00:00+00:00", "updated_at": "2026-07-28 06:58:39.271198+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-agents", "ai-research"], "entities": ["OpenAI", "DR. INFO", "GPT-5", "Grok 3", "Gemini 2.5 Pro", "Claude 3.7 Sonnet", "OpenEvidence", "Doximity"], "alternates": {"html": "https://wpnews.pro/news/openais-healthbench-in-action-evaluating-an-llm-based-medical-assistant-on", "markdown": "https://wpnews.pro/news/openais-healthbench-in-action-evaluating-an-llm-based-medical-assistant-on.md", "text": "https://wpnews.pro/news/openais-healthbench-in-action-evaluating-an-llm-based-medical-assistant-on.txt", "jsonld": "https://wpnews.pro/news/openais-healthbench-in-action-evaluating-an-llm-based-medical-assistant-on.jsonld"}}