cd /news/large-language-models/how-stable-are-llm-generated-proofs-… · home › topics › large-language-models › article
[ARTICLE · art-159236] src=arxiv.org ↗ pub= topic=large-language-models verified=true sentiment=· neutral

How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation

A new benchmark, RefactorProof, finds that LLM-generated Lean proofs for verifiable code generation are brittle under behavior-preserving program refactoring, with task-normalized structural proof survival (PSR) of just 4.3% for VERINA's supplied reference proofs and 13.3-26.8% for specialized provers. The benchmark derives 458 Lean-certified variants from the 185 VERINA tasks that compile in a pinned environment while holding the specification and evaluated proof fixed byte-for-byte; on 21-25 matched short, non-recursive tasks, specialized prover proofs survived refactoring better than the references, by +15.6 points for DeepSeek-Prover-V2-7B, +18.8 for Goedel-Prover-V2-32B and +20.3 for Goedel-Prover-V2-8B, with all three 95% CIs excluding zero. The authors note that unchanged-proof failure is not the same as repair cost: predefined one-identifier edits repaired 51/54 eligible reference structural failures and 58/85 eligible prover failures.

by read1 min views1 publishedOct 28, 2026

arXiv:2610.36328v1 Announce Type: new Abstract: Benchmarks for verifiable code generation usually test a proof against one fixed implementation. We ask whether the same proof remains valid after a program refactoring that preserves behavior. We introduce RefactorProof, which derives 458 Lean-certified variants from the 185 VERINA tasks that compile in our pinned environment while holding the specification and evaluated proof fixed byte-for-byte. VERINA supplies reference proofs for 46 tasks, all from its basic split; the primary model-versus-reference comparisons further match on 21-25 short, non-recursive tasks. Structural refactorings expose substantial brittleness across proof sources: task-normalized structural PSR is 4.3% for the supplied references and 13.3-26.8% for the specialized provers. On the matched tasks, specialized prover proofs nevertheless have higher overall unchanged-proof survival than the supplied reference artifacts: +15.6 points for DeepSeek-Prover-V2-7B, +18.8 for Goedel-Prover-V2-32B, and +20.3 for Goedel-Prover-V2-8B; all three 95% CIs exclude zero. The ordering persists after excluding universally survived T1 variants, under family-balanced aggregation, and under alternative deployable valid-proof selection rules. However, unchanged-proof failure is not the same as high repair cost: predefined one-identifier edits repair 51/54 eligible reference structural failures and 58/85 eligible prover failures. RefactorProof therefore measures unchanged-proof stability under program refactoring, not the cost of maintaining a proof after it breaks.

── more in #large-language-models 4 stories · sorted by recency
── more on @refactorproof 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-stable-are-llm-g…] indexed:0 read:1min 2026-10-28 · —