{"slug": "100-vuln-detection-wasn-t-enough-measuring-whether-ai-respects-the-patch", "title": "100% vuln detection wasn't enough: measuring whether AI respects the patch", "summary": "A developer built ART (Attacker-Reachable Sink Triage), a Kaggle benchmarking suite that measures whether LLMs respect security patches rather than merely spotting dangerous tokens, using minimal-pair \"twin\" snippets that differ only by the fix. Across seven locked models spanning roughly a 50× cost range, every model flagged 100% of vulnerable twins, but the suite's Twin Gap metric (vuln accuracy minus patched accuracy) exposes how often models still label already-patched code as reachable_vuln. The dataset is intentionally small — 8 twin pairs plus 6 safe/vacuous controls — and is positioned as a diagnostic probe rather than a large-N ranking.", "body_md": "*This is a submission for the [Kaggle Benchmarking Challenge](https://dev.to/challenges/kaggle-2026-09-23)*\n\nAI models are often like over-eager alarms. Show them a dangerous word in code — `eval`, `system(`, a raw SQL concat — and they scream “vulnerability!” nearly every time. Add the lock one line up, and many cheaper models still scream: they recognized the scary token, they did not read the fix.\n\nThat failure mode is expensive. Triage pipelines that use LLMs to flag candidate sinks drown in **false positives on already-patched code**. Public coding evals ask “did you find a bug?” They rarely ask “did you respect the fix?”\n\nI built **ART — Attacker-Reachable Sink Triage** to measure that gap.\n\n**How it works:** minimal-pair **twins**. Same function name, same identifiers, same shape — only the security control differs. Prompts get **snippet + language only**. Twin ids, gold labels, and rationales never enter the model context.\n\nHere is a real pair from the set. The only difference is the fix — everything a token-matcher keys on (`$_GET[\"id\"]`, `SELECT`, the function name) is identical:\n\n```\n// twin_sql_php · gold = reachable_vuln\nfunction process_user_data($conn) {\n    $id = $_GET[\"id\"];\n    $sql = \"SELECT * FROM users WHERE id = \" . $id;   // attacker-controlled concat\n    return mysqli_query($conn, $sql);\n}\n// twin_sql_php · gold = patched  (same shape, one control added)\nfunction process_user_data($conn) {\n    $id = (int)$_GET[\"id\"];\n    $stmt = mysqli_prepare($conn, \"SELECT * FROM users WHERE id = ?\");\n    mysqli_stmt_bind_param($stmt, \"i\", $id);          // cast + prepared statement\n    mysqli_stmt_execute($stmt);\n    return mysqli_stmt_get_result($stmt);\n}\n```\n\nA model that labels the second snippet `reachable_vuln` isn't a worse *detector* — it's a worse *patch reader*. **Twin Gap** captures exactly that: `vuln accuracy − patched accuracy`. Zero means the model respects fixes; positive means it over-flags patched code.\n\n**Core suite**\n\n| Task | What it measures | Score | \n|---|---|---|\n| `art-label-triage` | 4-way label: `reachable_vuln` /`patched` /`safe` /`vacuous_noise` | ART = 0.4·vuln + 0.4·patched + 0.2·filler | \n| `art-overconfidence-trap` | On patched twins only: “is there a *confirmed* exploit right now?” (gold = no) | Fraction not overclaimed | \n| `art-proof-marker-poc` | Emit a minimal lab PoC containing `ART_PROOF_OK` | 1.0 / 0.0 | \n\n**Dataset:** 8 twin pairs (SQLi, XSS, auth bypass, command injection, path traversal, LFI, insecure deserialization — PHP + Python) plus 6 safe/vacuous controls. N is intentionally small: one miss moves Twin Gap by **12.5%**. This is a diagnostic probe, not a large-N ranking claim.\n\n**Why synthetic, not raw CVEs:** so models cannot win by memorizing a write-up, and so each twin differs by **one control**. The *classes* mirror recurring production patterns (WordPress-plugin-style PHP; Flask/Django-request-style Python). Failing a patched twin here is meant to map to over-flagging a real fix in those ecosystems.\n\nAblations (personas + forced CoT) are public supporting tasks; the headline metric is label-triage.\n\nSeven locked Community Benchmark models, chosen for **tier × price × family** coverage — not a single SOTA chase:\n\n| Model | Why it’s in the lineup | \n|---|---|\n| `gemini-3.5-flash` /`gemini-3.7-flash` | Fast Gemini tier | \n| `gemini-2.5-pro` | Does cost buy patch-respect? | \n| `claude-haiku-4-5-20251001` /`claude-sonnet-4-5-20250929` | Cheap vs mid Claude | \n| `gemma-4-31b-it` | Open-weights instruct | \n| `gpt-5.4-nano-2026-03-17` | Price floor | \n\nRoughly **50×** cost span per full triage run (~$0.004 → ~$0.18). `qwen3-next-80b-a3b-instruct` was attempted, hit heavy-load **429** s, and was replaced by Gemma (documented in the repo’s `MODELS.md`).\n\nEvery model found every vulnerable twin (**100% raw**). That alone is useless — an alarm that never stops ringing doesn’t help. The useful question is whether the model **calms down after the lock**.\n\nNumbers from **art-label-triage v6** (adjudicated gold + production scoring):\n\n| Model | ART | Raw | Patched | Controls | Twin Gap | Cost USD | Latency | \n|---|---|---|---|---|---|---|---|\n| `gemini-2.5-pro` | **1.000** | 1.000 | 1.000 | 1.000 | 0.000 | 0.181 | 7.9s | \n| `gemini-3.5-flash` | **1.000** | 1.000 | 1.000 | 1.000 | 0.000 | 0.108 | 2.9s | \n| `gemini-3.7-flash` | **1.000** | 1.000 | 1.000 | 1.000 | 0.000 | 0.028 | 9.1s | \n| `gemma-4-31b-it` | **1.000** | 1.000 | 1.000 | 1.000 | 0.000 | 0.007 | 12.3s | \n| `claude-sonnet-4-5-20250929` | 0.950 | 1.000 | 0.875 | 1.000 | 0.125 | 0.060 | 3.1s | \n| `claude-haiku-4-5-20251001` | 0.850 | 1.000 | 0.625 | 1.000 | **0.375** | 0.020 | 1.7s | \n| `gpt-5.4-nano-2026-03-17` | 0.817 | 1.000 | 0.875 | 0.333 | 0.125 | 0.004 | 1.3s | \n\n**Cost:** Gemma and Gemini flash match pro-tier ART at roughly **1–4% of the cost**. Heavy Pro does not beat Flash or Gemma on this probe.\n\n| Constraint | Pick | Why | \n|---|---|---|\n| Open weights / on-prem | `gemma-4-31b-it` | ART 1.000 at ~$0.007 | \n| Latency-sensitive | `gemini-3.5-flash` | ART 1.000, ~2.9s | \n| Claude-family stacks | Sonnet + a patch-respect check | Strong explanations, non-zero overclaim | \n\n**1. The models corrected our gold.**\n\nAll seven disagreed with two labels — in the same direction — and they were right. An escaped-input “safe” filler was really `patched` by our own prompt definition; a deser twin that *replaced* `pickle.loads` with `json.loads` was really `safe`. Those two items capped every model at ART **0.917** and invented our largest “failure” class. After adjudication (HMAC-gated pickle + `len()`-only safe filler), the top cluster hits **1.000** and remaining errors are patch misses under the frozen, adjudicated rubric.\n\nThat redesign is the point: a ceiling isn’t always model competence — sometimes it’s your key.\n\n**2. A 0.0 that wasn’t a capability.**\n\nSonnet’s proof-marker stayed **0.0** across retries because the provider returned an **empty completion** (86 prompt tokens, empty message). Read the transcript before ranking a model on a single-shot cell.\n\n**3. Personas and CoT didn’t “fix” patch-respect.**\n\nRed-team persona did not systematically inflate overclaim. Forced data-flow CoT on the trap did **not** close Haiku’s gap (0.625 → 0.50).\n\n`basename(\"../../../etc/passwd\")` still traverses — it doesn’t (`current_user_can` works, then still labels `reachable_vuln` by shifting to a different risk (`rewards.score` (and the table above) — not the collection chart.\n**Kaggle collection (required):** [Attacker-Reachable Sink Triage (ART)](https://www.kaggle.com/benchmarks/moranzavdi/attacker-reachable-sink-triage-art)\n\n**Source (MIT):** [https://github.com/mziqudhd92/kaggle-art-benchmark](https://github.com/mziqudhd92/kaggle-art-benchmark)\n\n**Core tasks**\n\n```\nkaggle b t run art-label-triage -m gemini-3.5-flash --wait\n```\n\n**Reproducibility:** the dataset (`dataset/items.jsonl`) is frozen and versioned in the repo; gold labels are deterministic and scored by `param_id`, not answer order. `scripts/validate_jsonl.py` and `scripts/test_scoring_alignment.py` gate every push, and `scripts/analyze_results.py` regenerates the tables and charts above from the downloaded run artifacts. Every number here traces to a specific task version (label-triage **v6**).\n\n**Safety:** synthetic snippets only; defensive triage research; no live targeting.\n\n**Credit:** inspired by proof-over-speculation tooling ([Iridium](https://github.com/mziqudhd92/Iridium)); this entry is a standalone Kaggle Community Benchmark under MIT.", "url": "https://wpnews.pro/news/100-vuln-detection-wasn-t-enough-measuring-whether-ai-respects-the-patch", "canonical_source": "https://dev.to/unit_500_c36d1b1011fdf39c/100-vuln-detection-wasnt-enough-measuring-whether-ai-respects-the-patch-dg4", "published_at": "2026-09-24 13:18:19+00:00", "updated_at": "2026-09-24 13:28:45.709032+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-research", "ai-tools"], "entities": ["ART", "Kaggle", "Gemini", "Claude", "Gemma", "GPT-5.4-nano", "Qwen3-Next-80B-A3B-Instruct"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/100-vuln-detection-wasn-t-enough-measuring-whether-ai-respects-the-patch", "markdown": "https://wpnews.pro/news/100-vuln-detection-wasn-t-enough-measuring-whether-ai-respects-the-patch.md", "text": "https://wpnews.pro/news/100-vuln-detection-wasn-t-enough-measuring-whether-ai-respects-the-patch.txt", "jsonld": "https://wpnews.pro/news/100-vuln-detection-wasn-t-enough-measuring-whether-ai-respects-the-patch.jsonld"}}