VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

A benchmark called VerifierBench has been introduced to measure how well LLM agents generate executable verifiers from a task's instruction and environment, where each verifier must accept every valid solution. The benchmark targets two uses: validating agent outputs during task completion and filtering synthetic training data. The work frames verifier generation as a distinct capability to be tested rather than assumed.

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution an