{"slug": "conv-to-bench-evaluating-language-models-via-user-assistant-dialogues-in-code", "title": "Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks", "summary": "Researchers have developed Conv-to-Bench, a framework that automatically converts real-world user-assistant dialogues into structured evaluation benchmarks for large language models. In programming tasks, the system achieved near-perfect alignment with human-authored standards, reaching Spearman correlations of up to 1.000 while reducing computational overhead. The approach addresses the scalability bottleneck of traditional evaluation methods by leveraging authentic conversational logs to create verifiable requirement checklists.", "body_md": "arXiv:2605.26440v1 Announce Type: new\nAbstract: The rapid advancement of Large Language Models (LLMs) has outpaced the scalability of traditional evaluation benchmarks, which remain heavily dependent on labor-intensive expert curation. We address this bottleneck with Conv-to-Bench, a multi-stage framework that automatically transforms authentic multi-turn user-assistant dialogues into structured, verifiable requirement checklists. By leveraging the \"instructional evolution\" found in real-world conversational logs, our approach deconstructs fragmented user intent into consolidated instructions and binary evaluation criteria. Applied to the programming domain, Conv-to-Bench produces evaluation sets that demonstrate near-perfect alignment with human-authored standards like BigCodeBench, achieving Spearman correlations of up to $\\rho$ = 1.000 with significantly lower computational overhead. Validation of the LLM-as-a-judge framework further confirms its reliability, with the primary evaluator achieving substantial agreement with human-verified ground truth ($\\kappa$ = 0.705). Our comprehensive ablation studies reveal that while multi-turn interactions capture the iterative evolution of user intent, instruction-centric extraction provides a more robust foundation. Ultimately, Conv-to-Bench provides a scalable, cost-effective paradigm for maintaining high-fidelity evaluation standards as user-centric AI applications continue to diversify.", "url": "https://wpnews.pro/news/conv-to-bench-evaluating-language-models-via-user-assistant-dialogues-in-code", "canonical_source": "https://arxiv.org/abs/2605.26440", "published_at": "2026-05-27 04:00:00+00:00", "updated_at": "2026-05-27 04:35:11.837436+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "machine-learning", "natural-language-processing", "ai-research"], "entities": ["Conv-to-Bench", "BigCodeBench", "LLM-as-a-judge"], "alternates": {"html": "https://wpnews.pro/news/conv-to-bench-evaluating-language-models-via-user-assistant-dialogues-in-code", "markdown": "https://wpnews.pro/news/conv-to-bench-evaluating-language-models-via-user-assistant-dialogues-in-code.md", "text": "https://wpnews.pro/news/conv-to-bench-evaluating-language-models-via-user-assistant-dialogues-in-code.txt", "jsonld": "https://wpnews.pro/news/conv-to-bench-evaluating-language-models-via-user-assistant-dialogues-in-code.jsonld"}}