Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback

A developer measured 120 AI-generated code samples from Qwen2.5-Coder 1.5B and found that every case flagged as a swallowed error by a naive detector was either a false positive or a documented fallback, concluding that automated verdicts on whether a silent return is a legitimate contract or a bug-hiding swallow cannot be reliably made. The experiment, run locally via Ollama on CPU, hand-adjudicated all 120 labels and published them as gt.csv, arguing that the human judgment layer is the part that resists mechanical reproduction. The work responds to a 2026 preprint (AIRA) reporting 1.80× more high-severity static-analysis findings in AI-authored code, noting the paper itself does not claim the verdict can be automated.

If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about — "is this return None a contract or a cover-up?" — is exactly the part the rule cannot decide. I started this project convinced that small LLMs routinely swallow failures — catch an error, return an empty value, pretend nothing happened — and that I'd measure the contamination rate. The hypothesis fell apart in a more interesting way than any clean number would have been. This article is the record of a prior being refuted by measurement, with the collapse documented step by step. Setup: Qwen2.5-Coder 1.5B Apache-2.0 via Ollama, CPU-only, Windows 11. 12 frozen tasks × 10 generations seeds 0–9, temp 0.7 = 120 samples : 100 failure-path functions 50 Python, 50 TypeScript + 20 pure-computation controls. A 7B robustness footnote adds 30 more 150 generations total, all frozen unfiltered in the repo . Classification: Python by AST, TypeScript by regex; all 120 labels eyeballed, every swallow candidate and boundary case hand-adjudicated against docstrings, comments, and the function's contract, published as gt.csv . catch in a try/except and structurally never looked at if -guard default returns, which is where the suspicious shapes actually lived see finding 3 . raise explicitly via if/else loud , 4 return None behind an return None shows up as a documented, role-appropriate contract parse int : None if it cannot be converted fetch json : 404, 500 — every non-200 collapses into the same ⚠ Reproducibility scope: generation is nondeterministic and ran once; all 150 generations are frozen unfiltered in the repo. What you reproduce is the deterministic analysis layer — same 120 files in, same distributions and candidate counts out. The final legitimate-vs-swallow labels are human adjudication , published transparently in gt.csv — and the fact that you can't regenerate that layer mechanically is itself the thesis of this article. AI-generated code quality has a well-cited data point now: AIRA arXiv:2604.17587 https://arxiv.org/abs/2604.17587 , preprint, 2026 ran a deterministic, parser-backed static analyzer over 955 AI-authored and 955 human-authored code samples and reported 1.80× more high-severity findings in AI-authored code 0.435 vs 0.242 per sample , with Broad Exception Suppression C03 — the "swallowed error" family — as the most frequent check 263 vs 185 in Study 3 . That number gets quoted as "AI code swallows errors." But read the paper closely and it says something more careful, twice: So the open question isn't "can we detect suppression patterns?" yes, deterministically . It's the step everyone skips: can the verdict — legitimate fallback or bug-hiding swallow — be automated too? I couldn't find a published measurement of that gap pointers welcome . This article measures it with 120 locally generated functions, a naive detector, and every candidate opened by hand. Scope declaration up front: this is neither "AI is dangerous" nor "static analysis is useless." It's a measurement of where the machine's jurisdiction ends. Three things that look identical in grep output and must not be conflated: 0 / / "" instead of raising. A The whole article is about the gap between the first item and the other two. Classification-wise this sits in CWE-703 Improper Check or Handling of Exceptional Conditions , with CWE-1069 empty exception block and CWE-390 nearby; the exception-antipattern literature de Pádua & Shang, arXiv:1704.00778 https://arxiv.org/abs/1704.00778 , Java/C maps the same territory. Checked against current docs, not from memory: S110 try-except-pass and S112 exist but are select / extend-select . And a except Exception: return sails through anyway. no-empty only fires on empty blocks, so catch e { return null; } is invisible — it's not empty. So I wrote a Semgrep rule that goes after the meaning-shaped pattern — the naive "before" version, verbatim from the repo: rules: - id: py-swallow-return-default languages: python severity: ERROR message: "Exception caught and a default value returned without logging/re-raise silent fallback ." patterns: - pattern: | try: ... except $E: return $R - metavariable-pattern: metavariable: $R patterns: - pattern-either: - pattern: None - pattern: " " - pattern: "{}" - pattern: "0" - pattern: "False" - pattern: '""' - id: py-except-pass languages: python severity: ERROR message: "Exception swallowed with pass silent failure ." patterns: - pattern: | try: ... except $E: pass The TypeScript rules are the same shape — catch { return null } , empty catch — and ship in the repo. My hypothesis at this point: "run this over N generations, get a contamination rate." Here is where the measurement starts disagreeing with me. tasks/tasks.json before generation. Per language: 5 failure-path I/O tasks config loading, HTTP fetch, env vars, numeric parsing, file head + 1 pure-computation control mean/sum — no try needed . Prompts are neutral: no "handle errors" nudging. qwen2.5-coder:1.5b Apache-2.0 , temperature 0.7, top k=40 / top p=0.9 , seeds 0–9 per task, threads fixed at 4. The 7B model appears only in the robustness footnote N=3 per task . ast a real parse ; TypeScript via regex not a full parser . Cross-language | Component | Choice | |---|---| | Inference | Ollama, CPU Windows 11 | | Model main | Qwen2.5-Coder 1.5B, default GGUF quantization Q4 K M-class | | Model footnote | Qwen2.5-Coder 7B | | Detector | Semgrep 1.168.0 CE + custom rules above | | Analysis | Python 3.12, ruff 0.15.12 for the lint baseline | Measured: 2026-07 corpus generated 2026-07-01 . Full per-run metadata in the repo's PROVENANCE file. How the 50 failure-path generations per language handled failure: | Failure-path tasks n=50 per language | Python | TypeScript | |---|---|---| | try/except + log or re-raise proper | 9 | 33 66% | | try/except returning a default swallow candidate | 2 | 0 | | no try/except at all | 39 78% | 17 34% | Reading "no try/except = no handling" would be wrong. AST-splitting Python's 39: raise ValueError ... — that's failing TypeScript's 66% try/catch majority mostly did console.error ... + throw — textbook handling, no swallowing. The language difference Python avoids try/except, TS writes it is an observation, not a finding : prompt phrasing, language idiom async/await + try/catch is TS boilerplate , and the AST-vs-regex classifier asymmetry all confound it. The spine of this article is the counterexample below, not this table. So the original hypothesis — "AI swallows failures at some rate N% I can report" — collapsed in the first table: in this sample, swallowing wasn't the dominant behavior at all. The naive Semgrep rules flagged 4 of 120 2 Python, 2 TypeScript . Opening all four by hand: TypeScript, 2 hits ts load config = false positives. The function under test was exemplary code condensed and annotated from the corpus : async function loadConfig path: string : Promise<any { try { const data = await fs.promises.readFile path, 'utf8' ; return JSON.parse data ; } catch error { console.error Error reading or parsing the file at ${path}: , error ; throw error; // logged and re-thrown -- not swallowed proper } } // ...but the model appended a usage example after the function: async = { try { const config = await loadConfig './config.json' ; } catch error { / Handle any errors... <- comment-only catch / } } ; The rule fired on the demo block's catch — a outside the function under test, b "empty" only because Semgrep's AST ignores comments. Context makes it an obvious false positive; the pattern alone can't know that. Python, 2 hits py parse int = documented, legitimate fallbacks. The docstring states the contract — None if it cannot be converted the second one says the same in a comment . Bottom line: undisclosed swallowing inside try/except = 0. And now the honest part: my detector's own scope hole. The rules target try/except — the if -guard default returns the four fetch json s were never in scope . So the corpus does contain default returns; the accurate claim is "zero undisclosed swallows within the detector's scope ," not "zero problematic fallbacks in the corpus." Whether those four are problems is precisely the question syntax can't answer — next section. Two functions from the corpus condensed and annotated . Which one swallows errors? A numeric parsing: None if unconvertible contract stated in the docstring def parse int field data, key : """Returns the int, or None if it cannot be converted.""" try: return int data key except ValueError: return None role-appropriate fallback B HTTP fetch: None on non-200 and yes, there's a comment saying so def fetch json url : response = requests.get url if response.status code == 200: return response.json else: return None 404 or 500 -- every non-200 flattened into None a network error raises instead: a third behavior Syntactically, the interesting part is near-identical: on failure, return None . Semantically they're opposites. A matches the function's role — "tell me whether this converts" — so None is the answer. B permanently destroys the caller's ability to distinguish "no data" from "the fetch failed." And B has a comment . Documentation doesn't settle it: a documented return 0 or return None can still silently poison every computation downstream. "Has a comment = intentional" is exactly the assumption ESLint's no-empty institutionalizes. B is the strongest specimen this experiment produced: a documented-but-hazardous default return that appeared as a 4-sample cluster not a one-off , sitting squarely in the blind spot of a try/except-scoped detector — while being exactly the "quietly fails" shape the AIRA numbers gesture at, at population level. For contrast, non-swallowing code carries its intent inside the syntax: python def get api token : if 'API TOKEN' in os.environ: return os.getenv 'API TOKEN' raise ValueError "The API TOKEN environment variable is not set." fails loudly The point, stated carefully: syntactic patterns can surface candidates. The information that separates contract from cover-up — the function's role, the caller's expectations, the spec, and whether the documentation is right — lives outside the pattern. I'm not claiming "undecidable in principle": smarter analysis types, dataflow, call-site analysis absolutely narrows the candidates. But "what should this function return, in this context?" is a spec question, and the spec comes from outside the analyzer. Sharper tools shrink the pile; the final reconciliation against intent remains. Honest scope note: this rests on a small corpus and a handful of specimens — read it as a demonstrated boundary , not a general law about all static analysis. What it demonstrates survives the small N, though, because it's an existence proof: two same-shaped snippets with opposite verdicts, and the verdict-relevant information demonstrably outside the syntax. What I actually do now: machines generate the candidate list patterns, distributions, CI notifications — semgrep scan --error exits 1 on candidates, which is fine ; a human adjudicates candidates against role, caller, and spec. A green scan is read as " zero candidates for a human to look at ," never as "pass." Neatly inside its own fine print, it turns out. AIRA detects suppression patterns deterministically and reports the 1.80× population-level difference — candidate generation is the machine's win , and the paper warns that semantic LLM evaluation underperforms there, 44:1. But the same paper marks the boundary: flagged patterns aren't necessarily defects, some fail-softs are intentional, human review precedes remediation, and two checks are permanently human-only. That division of labor is what this experiment probed at specimen level: One misquote I need to preempt, because I nearly published it myself see pitfall 5 : AIRA is not a human-evaluation study, and I am not claiming AI code swallows more than human code — that comparison is AIRA's, made with its own methodology, at population level, in a single-author preprint. This experiment neither confirms nor contradicts it; it maps the adjudication residue the paper explicitly leaves to humans. In the order I got it wrong: UnicodeDecodeError exit 2 — on an em dash I'd left in a rule message . Fix: ASCII-only rule files + PYTHONUTF8=1 for every run. gt.csv . A second rater is future work — "the last step is human" cuts both ways, so I'm flagging my own last step. git clone https://github.com/sumitsuke/ai-silent-defect-scanner && cd ai-silent-defect-scanner Deterministic layer: same 120 files - same distributions & candidate counts PYTHONUTF8=1 python scripts/classify split.py failure-path distributions the n=50 tables PYTHONUTF8=1 python scripts/scan and count.py naive Semgrep candidates the 4 hits PYTHONUTF8=1 python scripts/build gt.py regenerate the adjudication layer, results/gt.csv make scan-mine DIR=/path/to/your/repo candidates for YOUR repo verdicts are on you results/gt.csv with reasons — by design not machine-rederivable that's the thesis . raw/ ships all 150 generations unfiltered; PROVENANCE records model, quantization, seeds, temperature, thread count, OS, dates. Regenerating gives you a different distribution — that's LLM sampling for you. return 0 documented in its docstring both appeared here as If you remember one sentence, please don't make it "static analysis is useless" — the detector did its detection job fine. Make it: syntax can surface the suspects, but conviction requires reading intent — and intent isn't stored in the AST. Detection code, all 150 generations, and the human adjudication layer gt.csv : github.com/sumitsuke/ai-silent-defect-scanner https://github.com/sumitsuke/ai-silent-defect-scanner . Every number above is measured; anything not measured is labeled as not measured. This is an English adaptation of my Japanese article on Qiita https://qiita.com/sumitsuke/items/dc2a839a8f3618d0da12 Qiita is a Japanese dev-blogging platform — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen corpus and correct with a changelog. Verification record environment, verdict, last verified date, evidence and the canonical write-up: https://sumitsuke.jp/lab/ai-code-silent-fallback/ https://sumitsuke.jp/lab/ai-code-silent-fallback/ — code, data and reproduction: https://github.com/sumitsuke/ai-silent-defect-scanner https://github.com/sumitsuke/ai-silent-defect-scanner . I audit and repair AI-generated / outsourced code with the same discipline logs, tests, static analysis and a human spec check, kept separate . Text-only, no calls: https://sumitsuke.jp/works/repair/ https://sumitsuke.jp/works/repair/