If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about — "is this return None a contract or a cover-up?" — is exactly the part the rule cannot decide.
I started this project convinced that small LLMs routinely swallow failures — catch an error, return an empty value, pretend nothing happened — and that I'd measure the contamination rate. The hypothesis fell apart in a more interesting way than any clean number would have been. This article is the record of a prior being refuted by measurement, with the collapse documented step by step.
Setup: Qwen2.5-Coder 1.5B (Apache-2.0) via Ollama, CPU-only, Windows 11. 12 frozen tasks × 10 generations (seeds 0–9, temp 0.7) = 120 samples: 100 failure-path functions (50 Python, 50 TypeScript) + 20 pure-computation controls. A 7B robustness footnote adds 30 more (150 generations total, all frozen unfiltered in the repo). Classification: Python by AST, TypeScript by regex; all 120 labels eyeballed, every swallow candidate and boundary case hand-adjudicated against docstrings, comments, and the function's contract, published as gt.csv.
catch in a try/except and structurally never looked at if-guard default returns, which is where the suspicious shapes actually lived (see finding 3).raise explicitly via if/else (loud), 4 return None behind an return None shows up as a documented, role-appropriate contract (parse_int: None if it cannot be converted) fetch_json: 404, 500 — every non-200 collapses into the same
⚠ Reproducibility scope: generation is nondeterministic and ran once; all 150 generations are frozen unfiltered in the repo. What you reproduce is the deterministic analysis layer — same 120 files in, same distributions and candidate counts out. The final legitimate-vs-swallow labels are human adjudication, published transparently in gt.csv — and the fact that you can't regenerate that layer mechanically is itself the thesis of this article.
AI-generated code quality has a well-cited data point now: AIRA (arXiv:2604.17587, preprint, 2026) ran a deterministic, parser-backed static analyzer over 955 AI-authored and 955 human-authored code samples and reported 1.80× more high-severity findings in AI-authored code (0.435 vs 0.242 per sample), with Broad Exception Suppression (C03) — the "swallowed error" family — as the most frequent check (263 vs 185 in Study 3).
That number gets quoted as "AI code swallows errors." But read the paper closely and it says something more careful, twice:
So the open question isn't "can we detect suppression patterns?" (yes, deterministically). It's the step everyone skips: can the verdict — legitimate fallback or bug-hiding swallow — be automated too? I couldn't find a published measurement of that gap (pointers welcome). This article measures it with 120 locally generated functions, a naive detector, and every candidate opened by hand.
Scope declaration up front: this is neither "AI is dangerous" nor "static analysis is useless." It's a measurement of where the machine's jurisdiction ends.
Three things that look identical in grep output and must not be conflated:
0 / [] / "" instead of raising. A The whole article is about the gap between the first item and the other two. Classification-wise this sits in CWE-703 (Improper Check or Handling of Exceptional Conditions), with CWE-1069 (empty exception block) and CWE-390 nearby; the exception-antipattern literature (de Pádua & Shang, arXiv:1704.00778, Java/C#) maps the same territory.
Checked against current docs, not from memory:
S110 (try-except-pass) and S112 exist but are select/ extend-select. And a except Exception: return [] sails through anyway.no-empty only fires on empty blocks, so catch (e) { return null; } is invisible — it's not empty. So I wrote a Semgrep rule that goes after the meaning-shaped pattern — the naive "before" version, verbatim from the repo:
rules:
- id: py-swallow-return-default
languages: [python]
severity: ERROR
message: "Exception caught and a default value returned without logging/re-raise (silent fallback)."
patterns:
- pattern: |
try:
...
except $E:
return $R
- metavariable-pattern:
metavariable: $R
patterns:
- pattern-either:
- pattern: None
- pattern: "[]"
- pattern: "{}"
- pattern: "0"
- pattern: "False"
- pattern: '""'
- id: py-except-pass
languages: [python]
severity: ERROR
message: "Exception swallowed with pass (silent failure)."
patterns:
- pattern: |
try:
...
except $E:
pass
(The TypeScript rules are the same shape — catch { return null }, empty catch — and ship in the repo.)
My hypothesis at this point: "run this over N generations, get a contamination rate." Here is where the measurement starts disagreeing with me.
tasks/tasks.json before generation. Per language: 5 failure-path I/O tasks (config , HTTP fetch, env vars, numeric parsing, file head) + 1 pure-computation control (mean/sum — no try needed). Prompts are neutral: no "handle errors" nudging.qwen2.5-coder:1.5b (Apache-2.0), temperature 0.7, top_k=40 / top_p=0.9, seeds 0–9 per task, threads fixed at 4. The 7B model appears only in the robustness footnote (N=3 per task).ast (a real parse); TypeScript via regex (not a full parser). Cross-language | Component | Choice |
|---|---|
| Inference | Ollama, CPU (Windows 11) |
| Model (main) | Qwen2.5-Coder 1.5B, default GGUF quantization (Q4_K_M-class) |
| Model (footnote) | Qwen2.5-Coder 7B |
| Detector | Semgrep 1.168.0 (CE) + custom rules above |
| Analysis | Python 3.12, ruff 0.15.12 for the lint baseline |
Measured: 2026-07 (corpus generated 2026-07-01). Full per-run metadata in the repo's PROVENANCE file.
How the 50 failure-path generations per language handled failure:
| Failure-path tasks (n=50 per language) | Python | TypeScript |
|---|---|---|
| try/except + log or re-raise (proper) | 9 | 33 (66%) |
| try/except returning a default (swallow candidate ) | 2 | 0 |
| no try/except at all | 39 (78%) | 17 (34%) |
Reading "no try/except = no handling" would be wrong. AST-splitting Python's 39:
raise ValueError(...)) — that's failing TypeScript's 66% try/catch majority mostly did console.error(...) + throw — textbook handling, no swallowing.
The language difference (Python avoids try/except, TS writes it) is an observation, not a finding: prompt phrasing, language idiom (async/await + try/catch is TS boilerplate), and the AST-vs-regex classifier asymmetry all confound it. The spine of this article is the counterexample below, not this table.
So the original hypothesis — "AI swallows failures at some rate N% I can report" — collapsed in the first table: in this sample, swallowing wasn't the dominant behavior at all.
The naive Semgrep rules flagged 4 of 120 (2 Python, 2 TypeScript). Opening all four by hand:
TypeScript, 2 hits (ts_load_config) = false positives. The function under test was exemplary (code condensed and annotated from the corpus):
async function loadConfig(path: string): Promise<any> {
try {
const data = await fs.promises.readFile(path, 'utf8');
return JSON.parse(data);
} catch (error) {
console.error(`Error reading or parsing the file at ${path}:`, error);
throw error; // logged and re-thrown -- not swallowed (proper)
}
}
// ...but the model appended a usage example after the function:
(async () => {
try { const config = await loadConfig('./config.json'); }
catch (error) { /* Handle any errors... <- comment-only catch */ }
})();
The rule fired on the demo block's catch — (a) outside the function under test, (b) "empty" only because Semgrep's AST ignores comments. Context makes it an obvious false positive; the pattern alone can't know that.
Python, 2 hits (py_parse_int) = documented, legitimate fallbacks. The docstring states the contract — None if it cannot be converted (the second one says the same in a comment).
Bottom line: undisclosed swallowing inside try/except = 0. And now the honest part: my detector's own scope hole. The rules target try/except — the if-guard default returns (the four fetch_json s) were never in scope. So the corpus does contain default returns; the accurate claim is "zero undisclosed swallows within the detector's scope," not "zero problematic fallbacks in the corpus." Whether those four are problems is precisely the question syntax can't answer — next section.
Two functions from the corpus (condensed and annotated). Which one swallows errors?
def parse_int_field(data, key):
"""Returns the int, or None if it cannot be converted."""
try:
return int(data[key])
except ValueError:
return None # role-appropriate fallback
def fetch_json(url):
response = requests.get(url)
if response.status_code == 200:
return response.json()
else:
return None # 404 or 500 -- every non-200 flattened into None
Syntactically, the interesting part is near-identical: on failure, return None. Semantically they're opposites. (A) matches the function's role — "tell me whether this converts" — so None is the answer. (B) permanently destroys the caller's ability to distinguish "no data" from "the fetch failed." And (B) has a comment. Documentation doesn't settle it: a documented return 0 or return None can still silently poison every computation downstream. ("Has a comment = intentional" is exactly the assumption ESLint's no-empty institutionalizes.)
(B) is the strongest specimen this experiment produced: a documented-but-hazardous default return that appeared as a 4-sample cluster (not a one-off), sitting squarely in the blind spot of a try/except-scoped detector — while being exactly the "quietly fails" shape the AIRA numbers gesture at, at population level.
For contrast, non-swallowing code carries its intent inside the syntax:
def get_api_token():
if 'API_TOKEN' in os.environ:
return os.getenv('API_TOKEN')
raise ValueError("The API_TOKEN environment variable is not set.") # fails loudly
The point, stated carefully: syntactic patterns can surface candidates. The information that separates contract from cover-up — the function's role, the caller's expectations, the spec, and whether the documentation is right — lives outside the pattern. I'm not claiming "undecidable in principle": smarter analysis (types, dataflow, call-site analysis) absolutely narrows the candidates. But "what should this function return, in this context?" is a spec question, and the spec comes from outside the analyzer. Sharper tools shrink the pile; the final reconciliation against intent remains.
Honest scope note: this rests on a small corpus and a handful of specimens — read it as a demonstrated boundary, not a general law about all static analysis. What it demonstrates survives the small N, though, because it's an existence proof: two same-shaped snippets with opposite verdicts, and the verdict-relevant information demonstrably outside the syntax.
What I actually do now: machines generate the candidate list (patterns, distributions, CI notifications — semgrep scan --error exits 1 on candidates, which is fine); a human adjudicates candidates against role, caller, and spec. A green scan is read as "zero candidates for a human to look at," never as "pass."
Neatly inside its own fine print, it turns out. AIRA detects suppression patterns deterministically and reports the 1.80× population-level difference — candidate generation is the machine's win, and the paper warns that semantic (LLM) evaluation underperforms there, 44:1. But the same paper marks the boundary: flagged patterns aren't necessarily defects, some fail-softs are intentional, human review precedes remediation, and two checks are permanently human-only.
That division of labor is what this experiment probed at specimen level:
One misquote I need to preempt, because I nearly published it myself (see pitfall 5): AIRA is not a human-evaluation study, and I am not claiming AI code swallows more than human code — that comparison is AIRA's, made with its own methodology, at population level, in a single-author preprint. This experiment neither confirms nor contradicts it; it maps the adjudication residue the paper explicitly leaves to humans.
In the order I got it wrong:
UnicodeDecodeError (exit 2) — on an em dash I'd left in a rule message. Fix: ASCII-only rule files + PYTHONUTF8=1 for every run. gt.csv. A second rater is future work — "the last step is human" cuts both ways, so I'm flagging my own last step.
git clone https://github.com/sumitsuke/ai-silent-defect-scanner && cd ai-silent-defect-scanner
PYTHONUTF8=1 python scripts/classify_split.py # failure-path distributions (the n=50 tables)
PYTHONUTF8=1 python scripts/scan_and_count.py # naive Semgrep candidates (the 4 hits)
PYTHONUTF8=1 python scripts/build_gt.py # regenerate the adjudication layer, results/gt.csv
make scan-mine DIR=/path/to/your/repo # candidates for YOUR repo (verdicts are on you)
results/gt.csv with reasons — by design not machine-rederivable (that's the thesis).raw/ ships all 150 generations unfiltered; PROVENANCE records model, quantization, seeds, temperature, thread count, OS, dates. Regenerating gives you a different distribution — that's LLM sampling for you.return 0 documented in its docstring both appeared here as If you remember one sentence, please don't make it "static analysis is useless" — the detector did its detection job fine. Make it: syntax can surface the suspects, but conviction requires reading intent — and intent isn't stored in the AST.
Detection code, all 150 generations, and the human adjudication layer (gt.csv): github.com/sumitsuke/ai-silent-defect-scanner. Every number above is measured; anything not measured is labeled as not measured.
This is an English adaptation of my Japanese article on Qiita (Qiita is a Japanese dev-blogging platform) — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen corpus and correct with a changelog.
Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: https://sumitsuke.jp/lab/ai-code-silent-fallback/ — code, data and reproduction: https://github.com/sumitsuke/ai-silent-defect-scanner. I audit and repair AI-generated / outsourced code with the same discipline (logs, tests, static analysis and a human spec check, kept separate). Text-only, no calls: https://sumitsuke.jp/works/repair/