arXiv:2610.36328v1 Announce Type: new Abstract: Benchmarks for verifiable code generation usually test a proof against one fixed implementation. We ask whether the same proof remains valid after a program refactoring that preserves behavior. We introduce RefactorProof, which derives 458 Lean-certified variants from the 185 VERINA tasks that compile in our pinned environment while holding the specification and evaluated proof fixed byte-for-byte. VERINA supplies reference proofs for 46 tasks, all from its basic split; the primary model-versus-reference comparisons further match on 21-25 short, non-recursive tasks. Structural refactorings expose substantial brittleness across proof sources: task-normalized structural PSR is 4.3% for the supplied references and 13.3-26.8% for the specialized provers. On the matched tasks, specialized prover proofs nevertheless have higher overall unchanged-proof survival than the supplied reference artifacts: +15.6 points for DeepSeek-Prover-V2-7B, +18.8 for Goedel-Prover-V2-32B, and +20.3 for Goedel-Prover-V2-8B; all three 95% CIs exclude zero. The ordering persists after excluding universally survived T1 variants, under family-balanced aggregation, and under alternative deployable valid-proof selection rules. However, unchanged-proof failure is not the same as high repair cost: predefined one-identifier edits repair 51/54 eligible reference structural failures and 58/85 eligible prover failures. RefactorProof therefore measures unchanged-proof stability under program refactoring, not the cost of maintaining a proof after it breaks.
How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation
A new benchmark, RefactorProof, finds that LLM-generated Lean proofs for verifiable code generation are brittle under behavior-preserving program refactoring, with task-normalized structural proof survival (PSR) of just 4.3% for VERINA's supplied reference proofs and 13.3-26.8% for specialized provers. The benchmark derives 458 Lean-certified variants from the 185 VERINA tasks that compile in a pinned environment while holding the specification and evaluated proof fixed byte-for-byte; on 21-25 matched short, non-recursive tasks, specialized prover proofs survived refactoring better than the references, by +15.6 points for DeepSeek-Prover-V2-7B, +18.8 for Goedel-Prover-V2-32B and +20.3 for Goedel-Prover-V2-8B, with all three 95% CIs excluding zero. The authors note that unchanged-proof failure is not the same as repair cost: predefined one-identifier edits repaired 51/54 eligible reference structural failures and 58/85 eligible prover failures.
Run your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.