{"slug": "how-stable-are-llm-generated-proofs-under-program-refactoring-a-stress-test-for", "title": "How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation", "summary": "A new benchmark, RefactorProof, finds that LLM-generated Lean proofs for verifiable code generation are brittle under behavior-preserving program refactoring, with task-normalized structural proof survival (PSR) of just 4.3% for VERINA's supplied reference proofs and 13.3-26.8% for specialized provers. The benchmark derives 458 Lean-certified variants from the 185 VERINA tasks that compile in a pinned environment while holding the specification and evaluated proof fixed byte-for-byte; on 21-25 matched short, non-recursive tasks, specialized prover proofs survived refactoring better than the references, by +15.6 points for DeepSeek-Prover-V2-7B, +18.8 for Goedel-Prover-V2-32B and +20.3 for Goedel-Prover-V2-8B, with all three 95% CIs excluding zero. The authors note that unchanged-proof failure is not the same as repair cost: predefined one-identifier edits repaired 51/54 eligible reference structural failures and 58/85 eligible prover failures.", "body_md": "arXiv:2610.36328v1 Announce Type: new \nAbstract: Benchmarks for verifiable code generation usually test a proof against one fixed implementation. We ask whether the same proof remains valid after a program refactoring that preserves behavior. We introduce RefactorProof, which derives 458 Lean-certified variants from the 185 VERINA tasks that compile in our pinned environment while holding the specification and evaluated proof fixed byte-for-byte. VERINA supplies reference proofs for 46 tasks, all from its basic split; the primary model-versus-reference comparisons further match on 21-25 short, non-recursive tasks. Structural refactorings expose substantial brittleness across proof sources: task-normalized structural PSR is 4.3% for the supplied references and 13.3-26.8% for the specialized provers. On the matched tasks, specialized prover proofs nevertheless have higher overall unchanged-proof survival than the supplied reference artifacts: +15.6 points for DeepSeek-Prover-V2-7B, +18.8 for Goedel-Prover-V2-32B, and +20.3 for Goedel-Prover-V2-8B; all three 95% CIs exclude zero. The ordering persists after excluding universally survived T1 variants, under family-balanced aggregation, and under alternative deployable valid-proof selection rules. However, unchanged-proof failure is not the same as high repair cost: predefined one-identifier edits repair 51/54 eligible reference structural failures and 58/85 eligible prover failures. RefactorProof therefore measures unchanged-proof stability under program refactoring, not the cost of maintaining a proof after it breaks.", "url": "https://wpnews.pro/news/how-stable-are-llm-generated-proofs-under-program-refactoring-a-stress-test-for", "canonical_source": "https://arxiv.org/abs/2610.36328", "published_at": "2026-10-28 04:00:00+00:00", "updated_at": "2026-10-28 04:11:52.866055+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-safety", "generative-ai"], "entities": ["RefactorProof", "VERINA", "Lean", "DeepSeek-Prover-V2-7B", "Goedel-Prover-V2-32B", "Goedel-Prover-V2-8B", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-stable-are-llm-generated-proofs-under-program-refactoring-a-stress-test-for", "markdown": "https://wpnews.pro/news/how-stable-are-llm-generated-proofs-under-program-refactoring-a-stress-test-for.md", "text": "https://wpnews.pro/news/how-stable-are-llm-generated-proofs-under-program-refactoring-a-stress-test-for.txt", "jsonld": "https://wpnews.pro/news/how-stable-are-llm-generated-proofs-under-program-refactoring-a-stress-test-for.jsonld"}}