{"slug": "efficient-benchmarking-in-production-a-study-of-an-evolving-llm-agent", "title": "Efficient Benchmarking in Production: A Study of an Evolving LLM Agent", "summary": "A study of a production analytics LLM agent serving tens of thousands of monthly active users found that multidimensional 2PL adaptive testing delivered the best score fidelity, executing 200 questions — 38.5% of a full run — for 1.03 percentage points of mean absolute error, according to an arXiv paper (2609.21267v1) using 574 historical benchmark runs split chronologically into calibration and held-out periods. The team nonetheless deployed difficulty-stratified fixed subsets for their operational simplicity, showing they transfer without recalibration to five other agent families and remain stable across calibration windows as short as one day. The paper offers practical recommendations for recurring production-agent evaluation.", "body_md": "arXiv:2609.21267v1 Announce Type: new \nAbstract: Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation for a production analytics agent serving tens of thousands of monthly active users and report first-hand deployment experience. Using 574 historical runs of the production benchmark, split chronologically into calibration and held-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The results show that multidimensional 2PL adaptive testing achieves the best overall score fidelity: executing 200 questions, 38.5% of a full run, yields 1.03 pp of MAE. We nevertheless deployed difficulty-stratified fixed subsets because of their operational simplicity, and show they transfer without recalibration to five other agent families and remain stable across calibration windows as short as one day. Drawing on this deployment experience, we report practical recommendations for recurring production-agent evaluation.", "url": "https://wpnews.pro/news/efficient-benchmarking-in-production-a-study-of-an-evolving-llm-agent", "canonical_source": "https://arxiv.org/abs/2609.21267", "published_at": "2026-09-21 04:00:00+00:00", "updated_at": "2026-09-21 04:26:26.142320+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research", "mlops"], "entities": ["arXiv", "2PL adaptive testing", "IRT"], "alternates": {"html": "https://wpnews.pro/news/efficient-benchmarking-in-production-a-study-of-an-evolving-llm-agent", "markdown": "https://wpnews.pro/news/efficient-benchmarking-in-production-a-study-of-an-evolving-llm-agent.md", "text": "https://wpnews.pro/news/efficient-benchmarking-in-production-a-study-of-an-evolving-llm-agent.txt", "jsonld": "https://wpnews.pro/news/efficient-benchmarking-in-production-a-study-of-an-evolving-llm-agent.jsonld"}}