Safety of LLM generated code

A discussion on the Haskell Discourse forum highlights that AI-assisted developers produce commits at three to four times the rate of their peers but introduce security findings at 10x the rate, according to empirical research across Fortune 50 enterprises. The debate centers on the safety of LLM-generated code, with participants arguing that verifying code for hidden destructive bugs is challenging, especially when LLMs may ignore instructions, and that securing 'vibe-coded' applications requires treating it as a workflow-level security problem.

philh https://discourse.haskell.org/u/philh 1 That seems like probably a very different failure mode to me? Like, the kind of bug that allows arbitrary file deletion in a program that’s intended to be able to delete specific files, is very different from the kind of bug that allows arbitrary file deletion in a program that’s not intended to be able to delete files. I’m imagining you were working on the first kind of program, and ychr is the second. I’m imagining that ychr has a handful of uses for IO: reading files, mutable variables IORef/MVar/STM/… , exceptions, printing to stdout/stderr. Maybe one or two I’m forgetting. But no writing to the filesystem, and no network access. Assuming that’s right which it may well not be then I think “ruling out hidden destructive bugs to sufficient degree of confidence” isn’t all that difficult. 1 Like I think our imagination is irrelevant when dealing with slop. And it doesn’t matter what you prompted. It’s known that LLMs sometimes ignore instructions and that unrelated code can make it into the implementation. I don’t think any amount of effects system or safe haskell would convince me. Either you own the output and understand it or you’re playing code roulette. philh https://discourse.haskell.org/u/philh 3 I’m not imagining details about what the LLM generated, I’m imagining details about how to check what the LLM generated. Like, if you know that a codebase shouldn’t have any IO at all, then checking it doesn’t is fairly straightforward. Do you agree with that? 1 Like gregbm https://discourse.haskell.org/u/gregbm 4 How do I know your program doesn’t delete my home directory? Same way you’d know for human-generated code on hackage, probably. Speaking of which, what defenses does hackage have against malicious packages? That’s a pretty strange way to dismiss the “lack of agency” issue. And I don’t think it can be taken seriously. - Empirical research across Fortune 50 enterprises found that AI-assisted developers produce commits at three to four times the rate of their peers but introduce security findings at 10x the rate, creating a security debt that accumulates faster than organizations can remediate it 7 . Our study suggests that securing vibe-coded appli- cations requires treating vibe coding as a workflow-level security problem more than a code-generation problem LLMs change the game here fundamentally sadly. They rework distributed trust because there is less of a hurdle of effort to get something compiling and plausibly working. I see this myself in industry with junior devs. Before, they would get stuck and be unable to cut a PR that looked reasonable. Now, the LLM can give them that by default. But the LLM PRs have some pretty silly issues that trace back to the juniors’ lack of understanding. gregbm https://discourse.haskell.org/u/gregbm 7 With regards to trusting code on hackage, I am more worried about intentionally-introduced malware than I am about vibecoded packages, so I would be interested in a verification mechanism that isn’t just scanning for signs of AI-generated code. 1 Like Models can very well be compromised by state actors.