# Does AI-generated code silently swallow errors? 120 measured generations: every flagged case was a false positive or a documented fallback

> Source: <https://dev.to/tauridev/does-ai-generated-code-silently-swallow-errors-120-measured-generations-every-flagged-case-was-a-241p>
> Published: 2026-09-15 12:32:16+00:00

If you gate AI-generated code with linters or a CI rule that hunts for swallowed errors, this experiment suggests the part you actually care about — "is this `return None` a contract or a cover-up?" — is exactly the part the rule cannot decide.

I started this project convinced that small LLMs routinely swallow failures — catch an error, return an empty value, pretend nothing happened — and that I'd measure the contamination rate. The hypothesis fell apart in a more interesting way than any clean number would have been. This article is the record of a prior being refuted by measurement, with the collapse documented step by step.

*Setup: Qwen2.5-Coder 1.5B (Apache-2.0) via Ollama, CPU-only, Windows 11. 12 frozen tasks × 10 generations (seeds 0–9, temp 0.7) = **120 samples**: 100 failure-path functions (50 Python, 50 TypeScript) + 20 pure-computation controls. A 7B robustness footnote adds 30 more (150 generations total, all frozen unfiltered in the repo). Classification: Python by AST, TypeScript by regex; **all 120 labels eyeballed, every swallow candidate and boundary case hand-adjudicated** against docstrings, comments, and the function's contract, published as `gt.csv`.*

`catch` in a `try/except` and structurally never looked at `if`-guard default returns, which is where the suspicious shapes actually lived (see finding 3).`raise` explicitly via `if/else` (loud), 4 return `None` behind an `return None` shows up as a documented, role-appropriate contract (`parse_int`: None if it cannot be converted) `fetch_json`: 404, 500 — every non-200 collapses into the same 
⚠ **Reproducibility scope:** generation is nondeterministic and ran once; all 150 generations are frozen unfiltered in the repo. What you reproduce is the deterministic analysis layer — same 120 files in, same distributions and candidate counts out. The final legitimate-vs-swallow labels are *human adjudication*, published transparently in `gt.csv` — and the fact that you can't regenerate that layer mechanically is itself the thesis of this article.

AI-generated code quality has a well-cited data point now: **AIRA** ([arXiv:2604.17587](https://arxiv.org/abs/2604.17587), preprint, 2026) ran a deterministic, parser-backed static analyzer over 955 AI-authored and 955 human-authored code samples and reported **1.80× more high-severity findings in AI-authored code** (0.435 vs 0.242 per sample), with **Broad Exception Suppression (C03)** — the "swallowed error" family — as the most frequent check (263 vs 185 in Study 3).

That number gets quoted as "AI code swallows errors." But read the paper closely and it says something more careful, twice:

So the open question isn't "can we detect suppression patterns?" (yes, deterministically). It's the step everyone skips: **can the *verdict* — legitimate fallback or bug-hiding swallow — be automated too?** I couldn't find a published measurement of that gap (pointers welcome). This article measures it with 120 locally generated functions, a naive detector, and every candidate opened by hand.

Scope declaration up front: this is neither "AI is dangerous" nor "static analysis is useless." It's a measurement of where the machine's jurisdiction ends.

Three things that look identical in grep output and must not be conflated:

`0` / `[]` / `""` instead of raising. A The whole article is about the gap between the first item and the other two. Classification-wise this sits in **CWE-703** (Improper Check or Handling of Exceptional Conditions), with CWE-1069 (empty exception block) and CWE-390 nearby; the exception-antipattern literature ([de Pádua & Shang, arXiv:1704.00778](https://arxiv.org/abs/1704.00778), Java/C#) maps the same territory.

Checked against current docs, not from memory:

`S110` (try-except-pass) and `S112` exist but are `select`/` extend-select`. And a `except Exception: return []` sails through anyway.`no-empty` only fires on empty blocks, so `catch (e) { return null; }` is invisible — it's not empty. So I wrote a Semgrep rule that goes after the *meaning-shaped* pattern — the naive "before" version, verbatim from the repo:

```
rules:
  - id: py-swallow-return-default
    languages: [python]
    severity: ERROR
    message: "Exception caught and a default value returned without logging/re-raise (silent fallback)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            return $R
      - metavariable-pattern:
          metavariable: $R
          patterns:
            - pattern-either:
                - pattern: None
                - pattern: "[]"
                - pattern: "{}"
                - pattern: "0"
                - pattern: "False"
                - pattern: '""'

  - id: py-except-pass
    languages: [python]
    severity: ERROR
    message: "Exception swallowed with pass (silent failure)."
    patterns:
      - pattern: |
          try:
            ...
          except $E:
            pass
```

(The TypeScript rules are the same shape — `catch { return null }`, empty catch — and ship in the repo.)

My hypothesis at this point: "run this over N generations, get a contamination rate." Here is where the measurement starts disagreeing with me.

`tasks/tasks.json` before generation. Per language: 5 failure-path I/O tasks (config loading, HTTP fetch, env vars, numeric parsing, file head) + 1 pure-computation control (mean/sum — no try needed). Prompts are neutral: no "handle errors" nudging.`qwen2.5-coder:1.5b` (Apache-2.0), temperature 0.7, `top_k=40 / top_p=0.9`, seeds 0–9 per task, threads fixed at 4. The 7B model appears only in the robustness footnote (N=3 per task).`ast` (a real parse); TypeScript via regex (not a full parser). Cross-language | Component | Choice | 
|---|---|
| Inference | Ollama, CPU (Windows 11) | 
| Model (main) | Qwen2.5-Coder 1.5B, default GGUF quantization (Q4_K_M-class) | 
| Model (footnote) | Qwen2.5-Coder 7B | 
| Detector | Semgrep 1.168.0 (CE) + custom rules above | 
| Analysis | Python 3.12, ruff 0.15.12 for the lint baseline | 

*Measured: 2026-07 (corpus generated 2026-07-01). Full per-run metadata in the repo's PROVENANCE file.*

How the 50 failure-path generations per language handled failure:

| Failure-path tasks (n=50 per language) | Python | TypeScript | 
|---|---|---|
| try/except + log or re-raise (proper) | 9 | **33 (66%)** | 
| try/except returning a default (swallow *candidate* ) | 2 | 0 | 
| no try/except at all | **39 (78%)** | 17 (34%) | 

Reading "no try/except = no handling" would be wrong. AST-splitting Python's 39:

`raise ValueError(...)`) — that's failing TypeScript's 66% try/catch majority mostly did `console.error(...)` + `throw` — textbook handling, no swallowing.

The language difference (Python avoids try/except, TS writes it) is an **observation, not a finding**: prompt phrasing, language idiom (async/await + try/catch is TS boilerplate), and the AST-vs-regex classifier asymmetry all confound it. The spine of this article is the counterexample below, not this table.

So the original hypothesis — "AI swallows failures at some rate N% I can report" — collapsed in the first table: in this sample, swallowing wasn't the dominant behavior at all.

The naive Semgrep rules flagged **4 of 120** (2 Python, 2 TypeScript). Opening all four by hand:

**TypeScript, 2 hits (`ts_load_config`) = false positives.** The function under test was exemplary (code condensed and annotated from the corpus):

```
async function loadConfig(path: string): Promise<any> {
    try {
        const data = await fs.promises.readFile(path, 'utf8');
        return JSON.parse(data);
    } catch (error) {
        console.error(`Error reading or parsing the file at ${path}:`, error);
        throw error; // logged and re-thrown -- not swallowed (proper)
    }
}
// ...but the model appended a usage example after the function:
(async () => {
    try { const config = await loadConfig('./config.json'); }
    catch (error) { /* Handle any errors... <- comment-only catch */ }
})();
```

The rule fired on the *demo block's* catch — (a) outside the function under test, (b) "empty" only because Semgrep's AST ignores comments. Context makes it an obvious false positive; the pattern alone can't know that.

**Python, 2 hits (`py_parse_int`) = documented, legitimate fallbacks.** The docstring states the contract — None if it cannot be converted (the second one says the same in a comment).

**Bottom line: undisclosed swallowing inside try/except = 0.** And now the honest part: my detector's own scope hole. The rules target `try/except` — the **`if`-guard default returns (the four `fetch_json` s) were never in scope**. So the corpus *does* contain default returns; the accurate claim is "zero undisclosed swallows *within the detector's scope*," not "zero problematic fallbacks in the corpus." Whether those four are problems is precisely the question syntax can't answer — next section.

Two functions from the corpus (condensed and annotated). Which one swallows errors?

```
# (A) numeric parsing: None if unconvertible (contract stated in the docstring)
def parse_int_field(data, key):
    """Returns the int, or None if it cannot be converted."""
    try:
        return int(data[key])
    except ValueError:
        return None            # role-appropriate fallback

# (B) HTTP fetch: None on non-200 (and yes, there's a comment saying so)
def fetch_json(url):
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None            # 404 or 500 -- every non-200 flattened into None
                               # (a network error raises instead: a third behavior)
```

Syntactically, the interesting part is near-identical: *on failure, return `None`*. Semantically they're opposites. (A) matches the function's role — "tell me whether this converts" — so `None` *is* the answer. (B) permanently destroys the caller's ability to distinguish "no data" from "the fetch failed." And (B) **has a comment**. Documentation doesn't settle it: a documented `return 0` or `return None` can still silently poison every computation downstream. ("Has a comment = intentional" is exactly the assumption ESLint's `no-empty` institutionalizes.)

(B) is the strongest specimen this experiment produced: a *documented-but-hazardous* default return that appeared as a **4-sample cluster** (not a one-off), sitting squarely in the blind spot of a try/except-scoped detector — while being exactly the "quietly fails" shape the AIRA numbers gesture at, at population level.

For contrast, non-swallowing code carries its intent *inside* the syntax:

``` python
def get_api_token():
    if 'API_TOKEN' in os.environ:
        return os.getenv('API_TOKEN')
    raise ValueError("The API_TOKEN environment variable is not set.")  # fails loudly
```

The point, stated carefully: **syntactic patterns can surface candidates. The information that separates contract from cover-up — the function's role, the caller's expectations, the spec, and whether the documentation is *right* — lives outside the pattern.** I'm not claiming "undecidable in principle": smarter analysis (types, dataflow, call-site analysis) absolutely narrows the candidates. But "what should this function return, in this context?" is a spec question, and the spec comes from outside the analyzer. Sharper tools shrink the pile; the final reconciliation against intent remains.

**Honest scope note:** this rests on a small corpus and a handful of specimens — read it as a *demonstrated boundary*, not a general law about all static analysis. What it demonstrates survives the small N, though, because it's an existence proof: two same-shaped snippets with opposite verdicts, and the verdict-relevant information demonstrably outside the syntax.

**What I actually do now:** machines generate the candidate list (patterns, distributions, CI notifications — `semgrep scan --error` exits 1 on candidates, which is fine); a human adjudicates candidates against role, caller, and spec. A green scan is read as "**zero candidates for a human to look at**," never as "pass."

Neatly inside its own fine print, it turns out. AIRA detects suppression patterns deterministically and reports the 1.80× population-level difference — **candidate generation is the machine's win**, and the paper warns that semantic (LLM) evaluation *underperforms* there, 44:1. But the same paper marks the boundary: flagged patterns aren't necessarily defects, some fail-softs are intentional, human review precedes remediation, and two checks are permanently human-only.

That division of labor is what this experiment probed at specimen level:

One misquote I need to preempt, because I nearly published it myself (see pitfall 5): AIRA is **not** a human-evaluation study, and I am not claiming AI code swallows more than human code — that comparison is AIRA's, made with its own methodology, at population level, in a single-author preprint. This experiment neither confirms nor contradicts it; it maps the adjudication residue the paper explicitly leaves to humans.

In the order I got it wrong:

`UnicodeDecodeError` (exit 2) — on an em dash I'd left in a rule `message`. Fix: ASCII-only rule files + `PYTHONUTF8=1` for every run. `gt.csv`. A second rater is future work — "the last step is human" cuts both ways, so I'm flagging my own last step.

```
git clone https://github.com/sumitsuke/ai-silent-defect-scanner && cd ai-silent-defect-scanner
# Deterministic layer: same 120 files -> same distributions & candidate counts
PYTHONUTF8=1 python scripts/classify_split.py      # failure-path distributions (the n=50 tables)
PYTHONUTF8=1 python scripts/scan_and_count.py      # naive Semgrep candidates (the 4 hits)
PYTHONUTF8=1 python scripts/build_gt.py            # regenerate the adjudication layer, results/gt.csv
make scan-mine DIR=/path/to/your/repo              # candidates for YOUR repo (verdicts are on you)
```

`results/gt.csv` with reasons — by design not machine-rederivable (that's the thesis).`raw/` ships all 150 generations unfiltered; `PROVENANCE` records model, quantization, seeds, temperature, thread count, OS, dates. Regenerating gives you a different distribution — that's LLM sampling for you.`return 0` documented in its docstring both appeared here as If you remember one sentence, please don't make it "static analysis is useless" — the detector did its detection job fine. Make it: **syntax can surface the suspects, but conviction requires reading intent — and intent isn't stored in the AST.**

*Detection code, all 150 generations, and the human adjudication layer (`gt.csv`): [github.com/sumitsuke/ai-silent-defect-scanner](https://github.com/sumitsuke/ai-silent-defect-scanner). Every number above is measured; anything not measured is labeled as not measured.*

*This is an English adaptation of [my Japanese article on Qiita](https://qiita.com/sumitsuke/items/dc2a839a8f3618d0da12) (Qiita is a Japanese dev-blogging platform) — written by me in Japanese, restructured and translated with AI assistance, human-reviewed. If you spot an error, comments and issues are open; I'll verify against the frozen corpus and correct with a changelog.*

*Verification record (environment, verdict, last verified date, evidence) and the canonical write-up: [https://sumitsuke.jp/lab/ai-code-silent-fallback/](https://sumitsuke.jp/lab/ai-code-silent-fallback/) — code, data and reproduction: [https://github.com/sumitsuke/ai-silent-defect-scanner](https://github.com/sumitsuke/ai-silent-defect-scanner). I audit and repair AI-generated / outsourced code with the same discipline (logs, tests, static analysis and a human spec check, kept separate). Text-only, no calls: [https://sumitsuke.jp/works/repair/](https://sumitsuke.jp/works/repair/)*