HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

Researchers introduced HERO (Harnessing Evidence for Refinement and Optimization), a harness optimization method for program generation that combines offline prompt learning with test-time refinement, according to an arXiv paper (arXiv:2610.19184v1). Across four SQL, Cypher, and Python benchmarks, HERO improved pass@1 by 7.6 points over the initial prompt, with test-time refinement adding another 2.0 points and improving execution efficiency. HERO validates candidate prompt revisions using execution-verified successes and failures before adoption, then treats generated code as an instance-specific harness refined with execution feedback.

arXiv:2610.19184v1 Announce Type: new Abstract: Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce \textsc{HERO} \textbf{H}arnessing \textbf{E}vidence for \textbf{R}efinement and \textbf{O}ptimization , which combines offline and test-time harness optimization. At the first stage, \method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, \method{} achieves the best offline performance, improving pass@1 by $7.6$ points over the initial prompt, while test-time refinement adds another $2.0$ points and improves execution efficiency.