{"slug": "cartbench-rubric-based-evaluation-of-shopping-advice-for-complex-and-ambiguous", "title": "CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines", "summary": "A new benchmark, CARTBench, finds that 3 of 10 evaluated models exceed 90% on Core criteria for LLM shopping advice, but none exceeds 45% on Excellence, according to the arXiv paper 2611.05862v1. CARTBench pairs timestamped dialogue with 1,603 product records across 17 retail domains and scores models against 1,328 expert-annotated criteria in Core, Excellence and Safety, using case-specific rubrics distilled from manually selected successful replies. The authors evaluated 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, plus open-weight models of 35B+ parameters, and identify an \"understanding-to-execution gap\" tied to shared failures in product evidence and unresolved needs.", "body_md": "arXiv:2611.05862v1 Announce Type: new \nAbstract: Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. For Difficulty Standardizing Criteria, we develop case-specific rubrics guided by Ideal Response Principles, distilled from excellent replies manually selected from successfully completed orders. These principles also guide Principle-Guided CART Prompting, our structured recommendation strategy. The experimental benchmark contains 1,328 expert-annotated criteria across Core, Excellence, and Safety. To answer the CART question, we evaluate 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, and 35B+ parameters open-weight models. In the first full-suite evaluation under the frozen rubrics, 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice. To examine these limitations, we diagnose shared failures in product evidence and unresolved needs, linked to changing requirements and decisive unknowns. Code and data are available at https://github.com/Da1yuqin/CARTBench.", "url": "https://wpnews.pro/news/cartbench-rubric-based-evaluation-of-shopping-advice-for-complex-and-ambiguous", "canonical_source": "https://arxiv.org/abs/2611.05862", "published_at": "2026-11-05 05:00:00+00:00", "updated_at": "2026-11-05 05:19:51.827141+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-products", "ai-agents"], "entities": ["CARTBench", "GPT-6.1-Sol", "Claude-Opus-5", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cartbench-rubric-based-evaluation-of-shopping-advice-for-complex-and-ambiguous", "markdown": "https://wpnews.pro/news/cartbench-rubric-based-evaluation-of-shopping-advice-for-complex-and-ambiguous.md", "text": "https://wpnews.pro/news/cartbench-rubric-based-evaluation-of-shopping-advice-for-complex-and-ambiguous.txt", "jsonld": "https://wpnews.pro/news/cartbench-rubric-based-evaluation-of-shopping-advice-for-complex-and-ambiguous.jsonld"}}