{"slug": "does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-a", "title": "Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback", "summary": "A developer measured 120 AI-generated code samples from Qwen2.5-Coder 1.5B and found that every case flagged as a swallowed error by a naive detector was either a false positive or a documented fallback, concluding that automated verdicts on whether a silent return is a legitimate contract or a bug-hiding swallow cannot be reliably made. The experiment, run locally via Ollama on CPU, hand-adjudicated all 120 labels and published them as gt.csv, arguing that the human judgment layer is the part that resists mechanical reproduction. The work responds to a 2026 preprint (AIRA) reporting 1.80× more high-severity static-analysis findings in AI-authored code, noting the paper itself does not claim the verdict can be automated.", "body_md": "If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about — \"is this `return None` a contract or a cover-up?\" — is exactly the part the rule cannot decide.\n\nI started this project convinced that small LLMs routinely swallow failures — catch an error, return an empty value, pretend nothing happened — and that I'd measure the contamination rate. The hypothesis fell apart in a more interesting way than any clean number would have been. This article is the record of a prior being refuted by measurement, with the collapse documented step by step.\n\n*Setup: Qwen2.5-Coder 1.5B (Apache-2.0) via Ollama, CPU-only, Windows 11. 12 frozen tasks × 10 generations (seeds 0–9, temp 0.7) = **120 samples**: 100 failure-path functions (50 Python, 50 TypeScript) + 20 pure-computation controls. A 7B robustness footnote adds 30 more (150 generations total, all frozen unfiltered in the repo). Classification: Python by AST, TypeScript by regex; **all 120 labels eyeballed, every swallow candidate and boundary case hand-adjudicated** against docstrings, comments, and the function's contract, published as `gt.csv`.*\n\n`catch` in a `try/except` and structurally never looked at `if`-guard default returns, which is where the suspicious shapes actually lived (see finding 3).`raise` explicitly via `if/else` (loud), 4 return `None` behind an `return None` shows up as a documented, role-appropriate contract (`parse_int`: None if it cannot be converted) `fetch_json`: 404, 500 — every non-200 collapses into the same \n⚠ **Reproducibility scope:** generation is nondeterministic and ran once; all 150 generations are frozen unfiltered in the repo. What you reproduce is the deterministic analysis layer — same 120 files in, same distributions and candidate counts out. The final legitimate-vs-swallow labels are *human adjudication*, published transparently in `gt.csv` — and the fact that you can't regenerate that layer mechanically is itself the thesis of this article.\n\nAI-generated code quality has a well-cited data point now: **AIRA** ([arXiv:2604.17587](https://arxiv.org/abs/2604.17587), preprint, 2026) ran a deterministic, parser-backed static analyzer over 955 AI-authored and 955 human-authored code samples and reported **1.80× more high-severity findings in AI-authored code** (0.435 vs 0.242 per sample), with **Broad Exception Suppression (C03)** — the \"swallowed error\" family — as the most frequent check (263 vs 185 in Study 3).\n\nThat number gets quoted as \"AI code swallows errors.\" But read the paper closely and it says something more careful, twice:\n\nSo the open question isn't \"can we detect suppression patterns?\" (yes, deterministically). It's the step everyone skips: **can the *verdict* — legitimate fallback or bug-hiding swallow — be automated too?** I couldn't find a published measurement of that gap (pointers welcome). This article measures it with 120 locally generated functions, a naive detector, and every candidate opened by hand.\n\nScope declaration up front: this is neither \"AI is dangerous\" nor \"static analysis is useless.\" It's a measurement of where the machine's jurisdiction ends.\n\nThree things that look identical in grep output and must not be conflated:\n\n`0` / `[]` / `\"\"` instead of raising. A The whole article is about the gap between the first item and the other two. Classification-wise this sits in **CWE-703** (Improper Check or Handling of Exceptional Conditions), with CWE-1069 (empty exception block) and CWE-390 nearby; the exception-antipattern literature ([de Pádua & Shang, arXiv:1704.00778](https://arxiv.org/abs/1704.00778), Java/C#) maps the same territory.\n\nChecked against current docs, not from memory:\n\n`S110` (try-except-pass) and `S112` exist but are `select`/` extend-select`. And a `except Exception: return []` sails through anyway.`no-empty` only fires on empty blocks, so `catch (e) { return null; }` is invisible — it's not empty. So I wrote a Semgrep rule that goes after the *meaning-shaped* pattern — the naive \"before\" version, verbatim from the repo:\n\n```\nrules:\n  - id: py-swallow-return-default\n    languages: [python]\n    severity: ERROR\n    message: \"Exception caught and a default value returned without logging/re-raise (silent fallback).\"\n    patterns:\n      - pattern: |\n          try:\n            ...\n          except $E:\n            return $R\n      - metavariable-pattern:\n          metavariable: $R\n          patterns:\n            - pattern-either:\n                - pattern: None\n                - pattern: \"[]\"\n                - pattern: \"{}\"\n                - pattern: \"0\"\n                - pattern: \"False\"\n                - pattern: '\"\"'\n\n  - id: py-except-pass\n    languages: [python]\n    severity: ERROR\n    message: \"Exception swallowed with pass (silent failure).\"\n    patterns:\n      - pattern: |\n          try:\n            ...\n          except $E:\n            pass\n```\n\n(The TypeScript rules are the same shape — `catch { return null }`, empty catch — and ship in the repo.)\n\nMy hypothesis at this point: \"run this over N generations, get a contamination rate.\" Here is where the measurement starts disagreeing with me.\n\n`tasks/tasks.json` before generation. Per language: 5 failure-path I/O tasks (config loading, HTTP fetch, env vars, numeric parsing, file head) + 1 pure-computation control (mean/sum — no try needed). Prompts are neutral: no \"handle errors\" nudging.`qwen2.5-coder:1.5b` (Apache-2.0), temperature 0.7, `top_k=40 / top_p=0.9`, seeds 0–9 per task, threads fixed at 4. The 7B model appears only in the robustness footnote (N=3 per task).`ast` (a real parse); TypeScript via regex (not a full parser). Cross-language | Component | Choice | \n|---|---|\n| Inference | Ollama, CPU (Windows 11) | \n| Model (main) | Qwen2.5-Coder 1.5B, default GGUF quantization (Q4_K_M-class) | \n| Model (footnote) | Qwen2.5-Coder 7B | \n| Detector | Semgrep 1.168.0 (CE) + custom rules above | \n| Analysis | Python 3.12, ruff 0.15.12 for the lint baseline | \n\n*Measured: 2026-07 (corpus generated 2026-07-01). Full per-run metadata in the repo's PROVENANCE file.*\n\nHow the 50 failure-path generations per language handled failure:\n\n| Failure-path tasks (n=50 per language) | Python | TypeScript | \n|---|---|---|\n| try/except + log or re-raise (proper) | 9 | **33 (66%)** | \n| try/except returning a default (swallow *candidate* ) | 2 | 0 | \n| no try/except at all | **39 (78%)** | 17 (34%) | \n\nReading \"no try/except = no handling\" would be wrong. AST-splitting Python's 39:\n\n`raise ValueError(...)`) — that's failing TypeScript's 66% try/catch majority mostly did `console.error(...)` + `throw` — textbook handling, no swallowing.\n\nThe language difference (Python avoids try/except, TS writes it) is an **observation, not a finding**: prompt phrasing, language idiom (async/await + try/catch is TS boilerplate), and the AST-vs-regex classifier asymmetry all confound it. The spine of this article is the counterexample below, not this table.\n\nSo the original hypothesis — \"AI swallows failures at some rate N% I can report\" — collapsed in the first table: in this sample, swallowing wasn't the dominant behavior at all.\n\nThe naive Semgrep rules flagged **4 of 120** (2 Python, 2 TypeScript). Opening all four by hand:\n\n**TypeScript, 2 hits (`ts_load_config`) = false positives.** The function under test was exemplary (code condensed and annotated from the corpus):\n\n```\nasync function loadConfig(path: string): Promise<any> {\n    try {\n        const data = await fs.promises.readFile(path, 'utf8');\n        return JSON.parse(data);\n    } catch (error) {\n        console.error(`Error reading or parsing the file at ${path}:`, error);\n        throw error; // logged and re-thrown -- not swallowed (proper)\n    }\n}\n// ...but the model appended a usage example after the function:\n(async () => {\n    try { const config = await loadConfig('./config.json'); }\n    catch (error) { /* Handle any errors... <- comment-only catch */ }\n})();\n```\n\nThe rule fired on the *demo block's* catch — (a) outside the function under test, (b) \"empty\" only because Semgrep's AST ignores comments. Context makes it an obvious false positive; the pattern alone can't know that.\n\n**Python, 2 hits (`py_parse_int`) = documented, legitimate fallbacks.** The docstring states the contract — None if it cannot be converted (the second one says the same in a comment).\n\n**Bottom line: undisclosed swallowing inside try/except = 0.** And now the honest part: my detector's own scope hole. The rules target `try/except` — the **`if`-guard default returns (the four `fetch_json` s) were never in scope**. So the corpus *does* contain default returns; the accurate claim is \"zero undisclosed swallows *within the detector's scope*,\" not \"zero problematic fallbacks in the corpus.\" Whether those four are problems is precisely the question syntax can't answer — next section.\n\nTwo functions from the corpus (condensed and annotated). Which one swallows errors?\n\n```\n# (A) numeric parsing: None if unconvertible (contract stated in the docstring)\ndef parse_int_field(data, key):\n    \"\"\"Returns the int, or None if it cannot be converted.\"\"\"\n    try:\n        return int(data[key])\n    except ValueError:\n        return None            # role-appropriate fallback\n\n# (B) HTTP fetch: None on non-200 (and yes, there's a comment saying so)\ndef fetch_json(url):\n    response = requests.get(url)\n    if response.status_code == 200:\n        return response.json()\n    else:\n        return None            # 404 or 500 -- every non-200 flattened into None\n                               # (a network error raises instead: a third behavior)\n```\n\nSyntactically, the interesting part is near-identical: *on failure, return `None`*. Semantically they're opposites. (A) matches the function's role — \"tell me whether this converts\" — so `None` *is* the answer. (B) permanently destroys the caller's ability to distinguish \"no data\" from \"the fetch failed.\" And (B) **has a comment**. Documentation doesn't settle it: a documented `return 0` or `return None` can still silently poison every computation downstream. (\"Has a comment = intentional\" is exactly the assumption ESLint's `no-empty` institutionalizes.)\n\n(B) is the strongest specimen this experiment produced: a *documented-but-hazardous* default return that appeared as a **4-sample cluster** (not a one-off), sitting squarely in the blind spot of a try/except-scoped detector — while being exactly the \"quietly fails\" shape the AIRA numbers gesture at, at population level.\n\nFor contrast, non-swallowing code carries its intent *inside* the syntax:\n\n``` python\ndef get_api_token():\n    if 'API_TOKEN' in os.environ:\n        return os.getenv('API_TOKEN')\n    raise ValueError(\"The API_TOKEN environment variable is not set.\")  # fails loudly\n```\n\nThe point, stated carefully: **syntactic patterns can surface candidates. The information that separates contract from cover-up — the function's role, the caller's expectations, the spec, and whether the documentation is *right* — lives outside the pattern.** I'm not claiming \"undecidable in principle\": smarter analysis (types, dataflow, call-site analysis) absolutely narrows the candidates. But \"what should this function return, in this context?\" is a spec question, and the spec comes from outside the analyzer. Sharper tools shrink the pile; the final reconciliation against intent remains.\n\n**Honest scope note:** this rests on a small corpus and a handful of specimens — read it as a *demonstrated boundary*, not a general law about all static analysis. What it demonstrates survives the small N, though, because it's an existence proof: two same-shaped snippets with opposite verdicts, and the verdict-relevant information demonstrably outside the syntax.\n\n**What I actually do now:** machines generate the candidate list (patterns, distributions, CI notifications — `semgrep scan --error` exits 1 on candidates, which is fine); a human adjudicates candidates against role, caller, and spec. A green scan is read as \"**zero candidates for a human to look at**,\" never as \"pass.\"\n\nNeatly inside its own fine print, it turns out. AIRA detects suppression patterns deterministically and reports the 1.80× population-level difference — **candidate generation is the machine's win**, and the paper warns that semantic (LLM) evaluation *underperforms* there, 44:1. But the same paper marks the boundary: flagged patterns aren't necessarily defects, some fail-softs are intentional, human review precedes remediation, and two checks are permanently human-only.\n\nThat division of labor is what this experiment probed at specimen level:\n\nOne misquote I need to preempt, because I nearly published it myself (see pitfall 5): AIRA is **not** a human-evaluation study, and I am not claiming AI code swallows more than human code — that comparison is AIRA's, made with its own methodology, at population level, in a single-author preprint. This experiment neither confirms nor contradicts it; it maps the adjudication residue the paper explicitly leaves to humans.\n\nIn the order I got it wrong:\n\n`UnicodeDecodeError` (exit 2) — on an em dash I'd left in a rule `message`. Fix: ASCII-only rule files + `PYTHONUTF8=1` for every run. `gt.csv`. A second rater is future work — \"the last step is human\" cuts both ways, so I'm flagging my own last step.\n\n```\ngit clone https://github.com/sumitsuke/ai-silent-defect-scanner && cd ai-silent-defect-scanner\n# Deterministic layer: same 120 files -> same distributions & candidate counts\nPYTHONUTF8=1 python scripts/classify_split.py      # failure-path distributions (the n=50 tables)\nPYTHONUTF8=1 python scripts/scan_and_count.py      # naive Semgrep candidates (the 4 hits)\nPYTHONUTF8=1 python scripts/build_gt.py            # regenerate the adjudication layer, results/gt.csv\nmake scan-mine DIR=/path/to/your/repo              # candidates for YOUR repo (verdicts are on you)\n```\n\n`results/gt.csv` with reasons — by design not machine-rederivable (that's the thesis).`raw/` ships all 150 generations unfiltered; `PROVENANCE` records model, quantization, seeds, temperature, thread count, OS, dates. Regenerating gives you a different distribution — that's LLM sampling for you.`return 0` documented in its docstring both appeared here as If you remember one sentence, please don't make it \"static analysis is useless\" — the detector did its detection job fine. Make it: **syntax can surface the suspects, but conviction requires reading intent — and intent isn't stored in the AST.**\n\n*Detection code, all 150 generations, and the human adjudication layer (`gt.csv`): [github.com/sumitsuke/ai-silent-defect-scanner](https://github.com/sumitsuke/ai-silent-defect-scanner). Every number above is measured; anything not measured is labeled as not measured.*\n\n*This is an English adaptation of [my Japanese article on Qiita](https://qiita.com/sumitsuke/items/dc2a839a8f3618d0da12) (Qiita is a Japanese dev-blogging platform) — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen corpus and correct with a changelog.*\n\n*Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: [https://sumitsuke.jp/lab/ai-code-silent-fallback/](https://sumitsuke.jp/lab/ai-code-silent-fallback/) — code, data and reproduction: [https://github.com/sumitsuke/ai-silent-defect-scanner](https://github.com/sumitsuke/ai-silent-defect-scanner). I audit and repair AI-generated / outsourced code with the same discipline (logs, tests, static analysis and a human spec check, kept separate). Text-only, no calls: [https://sumitsuke.jp/works/repair/](https://sumitsuke.jp/works/repair/)*", "url": "https://wpnews.pro/news/does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-a", "canonical_source": "https://dev.to/tauridev/does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-flagged-case-was-a-241p", "published_at": "2026-09-15 12:32:16+00:00", "updated_at": "2026-09-15 12:43:44.840325+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools", "developer-tools", "ai-safety"], "entities": ["Qwen2.5-Coder", "Ollama", "AIRA", "CWE-703", "CWE-1069", "CWE-390"], "alternates": {"html": "https://wpnews.pro/news/does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-a", "markdown": "https://wpnews.pro/news/does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-a.md", "text": "https://wpnews.pro/news/does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-a.txt", "jsonld": "https://wpnews.pro/news/does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-a.jsonld"}}