Does Fixing Break Security? An Empirical Study of LLM Security Degradation

A study of 5,968 scenarios from the IaC-Eval benchmark found that iterative LLM-driven Infrastructure-as-Code repair introduces security regressions in 3.3% of scenarios under strict detection (5.2% of 4,440 transitions), with resource restructuring accounting for 79.0% of root causes. The researchers, who submitted the paper to arXiv on 13 Aug 2026, recommend iteration 3 as the optimal stopping point and call for security-aware feedback-loop design.

Computer Science Software Engineering Submitted on 13 Aug 2026 Title:Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair View PDF /pdf/2608.13404 HTML experimental https://arxiv.org/html/2608.13404v1 Abstract:Background: Iterative feedback loops are the dominant paradigm for improving LLM-generated Infrastructure-as-Code IaC : validators such as Checkov and terraform validate feed error signals back for successive repair attempts. Prior work reports cumulative-best metrics, which are non-decreasing by construction, so the raw per-iteration security trajectory has never been examined for IaC. Aims: We study security regression a previously-passing CIS Benchmark check that fails after a repair iteration to determine whether and how often iterative LLM repair degrades security while fixing other issues. Method: We analyze 5,968 scenario timelines from the IaC-Eval benchmark, each one scenario run through one configuration for up to 5 repair iterations. The 15 configurations six model-specific RAG, nine model-aggregated non-RAG, three temperatures each yield 4,440 iteration transitions with Checkov data on both sides. We track 30 individual CIS check IDs and classify root causes from code diffs, under two detection modes: standard inclusive and strict exclusive check failures only . Results: Under standard detection, 13.8% of scenarios 24.8% of transitions exhibit at least one regression. Under strict detection the rate falls to 3.3% of scenarios 5.2% of transitions , indicating most apparent regressions are multi-resource measurement artifacts. Resource restructuring 79.0% is the dominant root cause. Regression transitions show 2.6x more code churn Cohen's d=0.90 and 4.9x higher strict-mode check volatility d=1.49 . Of standard-mode regressions, 36.6% self-correct within an average of 1.2 iterations; iteration 3 is the optimal stopping point. Conclusions: Iterative IaC repair does introduce security regressions, but the conservative, defensible rate is about 3.3% of scenarios. Our findings motivate security-aware feedback-loop design and actionable iteration-budget guidance. References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .