{"slug": "hero-harnessing-evidence-for-test-time-refinement-and-prompt-optimization-in", "title": "HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation", "summary": "Researchers introduced HERO (Harnessing Evidence for Refinement and Optimization), a harness optimization method for program generation that combines offline prompt learning with test-time refinement, according to an arXiv paper (arXiv:2610.19184v1). Across four SQL, Cypher, and Python benchmarks, HERO improved pass@1 by 7.6 points over the initial prompt, with test-time refinement adding another 2.0 points and improving execution efficiency. HERO validates candidate prompt revisions using execution-verified successes and failures before adoption, then treats generated code as an instance-specific harness refined with execution feedback.", "body_md": "arXiv:2610.19184v1 Announce Type: new \nAbstract: Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce \\textsc{HERO} (\\textbf{H}arnessing \\textbf{E}vidence for \\textbf{R}efinement and \\textbf{O}ptimization), which combines offline and test-time harness optimization. At the first stage, \\method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, \\method{} achieves the best offline performance, improving pass@1 by $7.6$ points over the initial prompt, while test-time refinement adds another $2.0$ points and improves execution efficiency.", "url": "https://wpnews.pro/news/hero-harnessing-evidence-for-test-time-refinement-and-prompt-optimization-in", "canonical_source": "https://arxiv.org/abs/2610.19184", "published_at": "2026-10-15 04:00:00+00:00", "updated_at": "2026-10-15 05:55:22.303308+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "natural-language-processing", "ai-tools"], "entities": ["HERO", "arXiv", "Harnessing Evidence for Refinement and Optimization"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/hero-harnessing-evidence-for-test-time-refinement-and-prompt-optimization-in", "markdown": "https://wpnews.pro/news/hero-harnessing-evidence-for-test-time-refinement-and-prompt-optimization-in.md", "text": "https://wpnews.pro/news/hero-harnessing-evidence-for-test-time-refinement-and-prompt-optimization-in.txt", "jsonld": "https://wpnews.pro/news/hero-harnessing-evidence-for-test-time-refinement-and-prompt-optimization-in.jsonld"}}