{"slug": "designing-a-robust-llm-based-evaluation-system-for-agentic-ai-in-drug-discovery", "title": "Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment", "summary": "AstraZeneca researchers developed an LLM-as-a-Judge evaluation framework for ChatInvent, an agentic drug discovery assistant, achieving human-aligned scoring with a majority-vote agreement of 0.86 after few-shot optimization. The framework defines four quality dimensions and validated Gemini 3.1 Pro, Claude Opus 4.7, GPT-5, and Llama 3.1 70B as judges, finding that informal phrasings do not degrade output quality.", "body_md": "arXiv:2608.21057v1 Announce Type: new\nAbstract: Agentic large language model (LLM) systems are reshaping scientific workflows in chemistry and drug discovery, but evaluating their open-ended, tool-augmented outputs remains a fundamental bottleneck. Reference-based metrics such as BLEU and ROUGE fail to capture semantic correctness, while expert human evaluation does not scale to the iteration speed these systems demand. The LLM-as-a-Judge paradigm has emerged as a scalable alternative, but existing drug discovery benchmarks deploy LLM judges without validating their alignment with human experts. In this work, we present an LLM-as-a-Judge evaluation framework for ChatInvent, an agentic drug discovery assistant deployed at AstraZeneca, with four contributions. First, we define four output-quality evaluation dimensions---Completeness, Relevancy, Structural Clarity, and Scope Adherence---alongside deterministic Tool Call Correctness checks. Second, we validate the judge through a human alignment study with five expert annotators, comparing Gemini 3.1 Pro, Claude Opus 4.7, GPT-5, and Llama 3.1 70B as candidate judges. Third, we optimize the best-performing judge using few-shot demonstrations of human-annotated examples, improving alignment with the human majority vote from 0.80 to 0.86. Fourth, applying the optimized judge to 70 held-out questions, we surface concrete limitations and find that informal phrasings do not systematically degrade output quality; if anything, it is helpful to have the LLM rewrite the original question before querying the agent. Our framework provides a reusable template for human-aligned evaluation of agentic systems in scientific domains.", "url": "https://wpnews.pro/news/designing-a-robust-llm-based-evaluation-system-for-agentic-ai-in-drug-discovery", "canonical_source": "https://www.machinebrief.com/news/designing-a-robust-llm-based-evaluation-system-for-agentic-a-ph70", "published_at": "2026-08-24 04:00:00+00:00", "updated_at": "2026-08-24 05:13:57.700705+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents"], "entities": ["AstraZeneca", "ChatInvent", "Gemini 3.1 Pro", "Claude Opus 4.7", "GPT-5", "Llama 3.1 70B"], "alternates": {"html": "https://wpnews.pro/news/designing-a-robust-llm-based-evaluation-system-for-agentic-ai-in-drug-discovery", "markdown": "https://wpnews.pro/news/designing-a-robust-llm-based-evaluation-system-for-agentic-ai-in-drug-discovery.md", "text": "https://wpnews.pro/news/designing-a-robust-llm-based-evaluation-system-for-agentic-ai-in-drug-discovery.txt", "jsonld": "https://wpnews.pro/news/designing-a-robust-llm-based-evaluation-system-for-agentic-ai-in-drug-discovery.jsonld"}}