Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks

A pre-registered, five-model paired evaluation tested a specification frame for LLM-generated code on money arithmetic, time handling, retry safety, and access control tasks, according to the study's authors. The evaluation addresses the finding that code generated by large language models passes security checks at a rate that has barely moved in four years, with teams responding via instruction files. The study frames specification as a prerequisite to generation for regulated backends.

Code generated by large language models passes security checks at a rate that has barely moved in four years. In regulated backends, the defect classes that matter most are money arithmetic, time handling, retry safety, and access control. Teams answer with instruction files, yet the largest control