{"slug": "i-told-the-ai-a-scanner-flagged-this-and-it-agreed-with-everything", "title": "I Told the AI \"A Scanner Flagged This\" — and It Agreed With Everything", "summary": "A developer found that a commercial AI model from a frontier lab removed only 20% of false alarms in code scanning, while another removed 51%, because the first was told a scanner had flagged the code and agreed with it. The developer built four countermeasures to combat this sycophancy, but found their effectiveness depends more on the model than the prompt.", "body_md": "I gave two AI models the same 200 pieces of code, the same prompt, the same\n\nquestion. One of them removed 51% of the false alarms. The other removed only\n\n20% of the false alarms — and confirmed 90% of everything it was shown.\n\nSame inputs. Same instructions. A 2.5× difference in the only thing I was\n\nmeasuring.\n\nThe model that failed isn't a bad model. It's a well-regarded commercial model\n\nfrom a frontier lab. It failed at this task for a specific, predictable reason:\n\n**I told it that a scanner had already flagged the code, and it believed me.**\n\nThis article is about that failure mode, the four countermeasures I built to\n\nfight it, and the uncomfortable finding that whether those\n\ncountermeasures work is mostly a property of the model, not the prompt.\n\nQuick context if you're new to the series.\n\nMy scanner works in two stages. Fixed rules trace data flows through code and\n\nfind every place where user input reaches something dangerous — a database query,\n\na file open, a system command. That stage is deterministic: same code in, same\n\nsuspects out, every time.\n\nThe problem is that this stage over-reports, badly. It flags code like this:\n\n```\nString id = request.getParameter(\"id\");\nif (!id.matches(\"[0-9]+\")) {\n    throw new IllegalArgumentException(\"id must be numeric\");\n}\nString sql = \"DELETE FROM products WHERE id = \" + id;\nstmt.executeUpdate(sql);\n```\n\nUntrusted input genuinely does reach the SQL string. The path is real. But\n\n`matches(\"[0-9]+\")`\n\nmeans `id`\n\ncan only ever be digits, so there's no attack. The\n\nrules see the *connection*; they can't see the *meaning*.\n\nSo the second stage hands each flagged snippet to a language model and asks one\n\nnarrow question: **is this actually exploitable?**\n\nThat's the whole bet. And it has a flaw sitting right at the centre of it.\n\nThe prompt has to explain the situation. Here's the actual line from my\n\n`llm.py`\n\n:\n\n```\nA static-analysis engine flagged the code below as a possible {vuln_class}\n({cwe}). Decide whether it is a REAL vulnerability or a FALSE ALARM.\n```\n\nRead that as the model reads it. Before it sees a single line of code, it has been\n\ntold that an expert system already concluded something is wrong.\n\nThat's an enormous hint. And language models are trained, deliberately, to be\n\nagreeable — reinforcement learning from human feedback rewards responses people\n\napprove of, and people approve of agreement. The tendency is well documented\n\nenough to have a name: **sycophancy**.\n\nFor most applications, agreeableness is harmless or even desirable. For mine it's\n\nfatal. A judge that agrees with every accusation isn't a judge. It's a rubber\n\nstamp with a token bill.\n\nWorse, it fails *invisibly*. The pipeline runs, verdicts come back, findings get\n\nreported — and the output is identical to having no judge at all. You'd only\n\nnotice by measuring, which is exactly what most people building these systems\n\nnever do.\n\nThe first defence is the bluntest. From the `RULES`\n\nblock of the real prompt:\n\n```\nRULES\n- Judge ONLY the code shown. Never assume code you cannot see.\n- confirmed=true only if attacker-controlled input reaches the dangerous sink\n  with nothing neutralising it on the way.\n- confirmed=false if the input is not attacker-controlled, never reaches the\n  sink, or is neutralised (parameterised query, escaping, encoding, allow-list).\n- Static engines raise false alarms routinely. Rejecting a finding is a correct\n  and expected answer. Do NOT agree just because the engine flagged it.\n```\n\nThat last rule is doing the heavy lifting. It does three things at once:\n\n**It states a fact about the world** — static engines raise false alarms\n\nroutinely. This isn't an instruction, it's context that makes rejection\n\nreasonable rather than contrarian.\n\n**It labels rejection as correct.** Without this, the model may treat \"no\" as a\n\nfailure to be helpful. Models optimise hard for being helpful.\n\n**It names the specific failure** — \"do NOT agree just because the engine flagged\n\nit.\" Naming the exact error is more effective than a general instruction to be\n\ncareful.\n\nNote also what rule 2 and 3 do: they give **concrete, checkable criteria** for\n\nboth answers. Not \"use your judgement\" but \"confirmed=false if... neutralised\n\n(parameterised query, escaping, encoding, allow-list).\" The model is being handed\n\na checklist, and \"allow-list\" is precisely what catches the `matches(\"[0-9]+\")`\n\ncase.\n\nInstructions describe. Examples demonstrate. The prompt includes exactly two\n\nworked examples — and the balance between them is the point.\n\nThe first shows a genuine vulnerability being confirmed:\n\n```\nString id = req.getParameter(\"id\");\nstmt.executeQuery(\"SELECT * FROM t WHERE id = \" + id);\n{\n  \"reasoning\": \"getParameter is attacker-controlled. It is concatenated straight\n                into the SQL string. Nothing escapes or parameterises it.\",\n  \"confirmed\": true,\n  \"severity\": \"Critical\",\n  \"explanation\": \"An attacker can change the SQL statement and read or modify\n                  arbitrary rows.\",\n  \"fix\": \"PreparedStatement ps = conn.prepareStatement(\\\"SELECT * FROM t WHERE id = ?\\\"); ps.setString(1, id);\"\n}\n```\n\nThe second shows a flagged finding being **rejected**:\n\n```\nString id = req.getParameter(\"id\");\nPreparedStatement ps = conn.prepareStatement(\"SELECT * FROM t WHERE id = ?\");\nps.setString(1, id);\nps.executeQuery();\n{\n  \"reasoning\": \"Source is attacker-controlled but bound with setString on a\n                parameterised query. The driver sends it as data, never as SQL.\",\n  \"confirmed\": false,\n  \"severity\": \"Low\",\n  \"explanation\": \"The query is parameterised, so the input cannot change the\n                  SQL structure.\",\n  \"fix\": \"No change needed.\"\n}\n```\n\nWhy this specific pairing matters:\n\n**Both examples start identically.** Both begin with\n\n`req.getParameter(\"id\")`\n\n— attacker-controlled input, no question about it. The\n\ndifference is entirely in what happens next. The model can't shortcut on \"does\n\nthis involve user input?\" because both do.\n\n**The rejection is not a trivial case.** I deliberately didn't use an obviously\n\nsafe example like a hardcoded string. I used a real flagged finding — one a static\n\nengine would genuinely report — being correctly dismissed. That's the behaviour I\n\nneed, so that's what I demonstrate.\n\n**The reasoning models the right thinking.** \"The driver sends it as data, never\n\nas SQL\" is the actual security reasoning, stated compactly. The examples aren't\n\njust teaching output format; they're teaching how to think about the question.\n\nOne-shot prompting with only the confirmation example would have been actively\n\nharmful — it would have taught the model that the expected answer is yes.\n\nMy scanner also retrieves real published vulnerabilities similar to the code under\n\nreview, and pastes them into the prompt as background. It's a genuinely useful\n\nfeature and a genuinely dangerous one.\n\nThink about what's happening: I'm about to ask \"is this code vulnerable?\" and\n\nimmediately before asking, I'm showing the model three real, confirmed\n\nvulnerabilities that look like it. That's textbook priming.\n\nSo the retrieved block carries an explicit disclaimer:\n\n```\nKNOWN RELATED ADVISORIES — real-world {vuln_class} reports, for grounding only.\nThey do NOT prove the code below is vulnerable; judge it on its own merits and\nreject it if it is safe.\n```\n\nAnd one implementation detail that matters more than the wording: **when nothing\nis retrieved, the prompt is byte-for-byte identical to the no-retrieval version.**\n\nThis one isn't in the prompt text at all — it's in the response schema:\n\n```\n_RESPONSE_SCHEMA = {\n    \"type\": \"OBJECT\",\n    \"properties\": {\n        \"reasoning\":   {\"type\": \"STRING\"},\n        \"confirmed\":   {\"type\": \"BOOLEAN\"},\n        \"severity\":    {\"type\": \"STRING\", \"enum\": [\"Critical\", \"High\", \"Medium\", \"Low\"]},\n        \"explanation\": {\"type\": \"STRING\"},\n        \"fix\":         {\"type\": \"STRING\"},\n    },\n    \"required\": [\"reasoning\", \"confirmed\", \"severity\", \"explanation\", \"fix\"],\n}\n```\n\n`reasoning`\n\nis declared **first**. Since the model generates the JSON in order, it\n\nmust produce its analysis before it emits the verdict.\n\nFlip those two fields and the dynamic reverses: the model commits to `confirmed:`\n\nand then generates text justifying a decision it has already made. Same\n\ntrue\n\nmodel, same prompt, worse answers — purely from field order.\n\nThe reasoning is capped at three sentences, incidentally, for a boring reason I'll\n\ncover in a later article: an earlier model spent its entire 8,192-token budget\n\n\"thinking\" and returned nothing at all.\n\nHere's where I have to be careful, because this is exactly the kind of claim\n\npeople make without evidence.\n\n**On my small test project**, the scanner flags 6 candidates, of which 2 are\n\nsanitised-but-flagged cases. The judge rejected exactly those 2. Its verbatim\n\nreasoning on the SQL case:\n\n\"The input 'id' is validated against a numeric-only regex '[0-9]+' before being\n\nconcatenated into the SQL query. This allow-list guard prevents any SQL\n\ninjection characters from reaching the sink.\"\n\nAnd on the XSS case:\n\n\"The source is request.getParameter('comment'). The input is passed through a\n\nstrip() method that uses a regex to allow only alphanumeric characters and\n\nspaces. This effectively neutralizes any XSS payload.\"\n\nThat's correct security reasoning, arriving at \"no\" on a finding an engine\n\nflagged. The countermeasures appear to be working.\n\n**At benchmark scale**, on 200 stratified cases from the OWASP Benchmark:\n\n| result | |\n|---|---|\n| False alarms removed | 50 of 98 (51%) |\n| Real bugs lost | 2 of 98 (2%) |\n| Precision | 0.50 → 0.67\n|\n| 95% CI | [0.43–0.57] → [0.59–0.74], non-overlapping |\n\nHalf the false alarms gone, at a cost of 2% of the real vulnerabilities, and the\n\nconfidence intervals don't overlap — so the improvement isn't sampling noise.\n\nThat's the result I wanted.\n\nI ran the identical experiment with a different model. Same 200 candidates. Same\n\ncode slices. Same prompt, character for character.\n\n| Judge | Confirms | False alarms removed | Precision |\n|---|---|---|---|\n`gemma-4-31b-it` (31B, open weights) |\n73% | 51% |\n0.50 → 0.67\n|\n`gpt-4o-mini` (commercial) |\n90% |\n20% | 0.50 → 0.55 |\n\n`gpt-4o-mini`\n\nconfirmed 90% of everything put in front of it. Its precision\n\nimprovement — 0.50 to 0.55 — has confidence intervals that *overlap* with doing\n\nnothing at all. Statistically, I cannot distinguish it from having no judge.\n\nAll four countermeasures were present in both runs. One model followed them. The\n\nother largely didn't.\n\n*\"You compared against the cheap mini model. Of course it lost.\"*\n\nFair. So I ran the identical 200 candidates — same slices, same prompt,\n\ncharacter for character — through `gpt-4o`\n\n, the frontier sibling.\n\n| Judge | Confirms | False alarms removed | Real bugs lost | Precision |\n|---|---|---|---|---|\n`gemma-4-31b-it` (31B, open weights) |\n73% | 51% |\n2% | 0.50 → 0.67\n|\n`gpt-4o` (frontier) |\n80% | 40% | 0% |\n0.50 → 0.62 |\n`gpt-4o-mini` (small commercial) |\n90% |\n20% | 1% | 0.50 → 0.55 |\n\nCapability does matter within a family: `gpt-4o`\n\nremoved twice the false alarms\n\nof its little sibling, and it was the only judge that kept every single real\n\nbug. But it still finished behind a free, mid-size open model, and its\n\nconfidence interval ([0.55–0.70]) still touches the no-judge interval\n\n([0.43–0.57]). Gemma remains the only judge whose improvement is statistically\n\nseparated at this sample size. The leaderboards predicted the mini → 4o step.\n\nNothing predicted the open model on top.\n\nGoing deeper(skip if you just want the lesson)First: does the headline survive the scanner's own later fixes? After this\n\nexperiment ran, the discovery stage gained its final recall improvements, so I\n\nre-ran the same protocol on the finished pipeline. The result reproduces —\n\n50 of 97 false alarms removed (52%) at the same 2% real-bug cost, precision\n\n0.51 → 0.67, interval still separated. I cite the original run throughout\n\nbecause it's the one all three judges shared.I also ran\n\n`gpt-4o-mini`\n\nover theentirecandidate set — 4,356 of 4,357\n\njudged, at 148 judgments/minute for about $1.60 — to check whether the sample\n\nwas misleading me. It wasn't: at full census it removed 111 of 611 false alarms\n\n(18%) and lost 3 of 743 true positives (candidate-level counts from an earlier\n\nrun, before the final recall fixes landed). The intervals still overlap.It's worth being precise about what this does and doesn't show. It does\n\nnot\n\nshow that`gpt-4o-mini`\n\nis a worse model in general — it's faster, cheaper per\n\ntoken, and better at plenty of things. It shows that onthistask, withthis\n\nprompt, it is much more likely to agree with a premise handed to it.I'd also caution against over-generalising from n=3 models. What I can defend is\n\nnarrow: the spread between three reasonable choices was large enough to\n\ndominate every other engineering decision I made, and the public benchmark\n\nscores predicted only part of the ordering — the step up inside the OpenAI\n\nfamily, not the open model finishing first.\n\n**1. If your prompt asserts something, measure whether the model just agrees.**\n\nThis applies far beyond security. Any time you write \"the system detected X, is\n\nthis correct?\" or \"the user reported Y, is that plausible?\", you've handed the\n\nmodel a conclusion. The polite output you get back may be pure echo. The only way\n\nto know is to feed it cases where the right answer is \"no\" and count.\n\n**2. Prompt engineering has a ceiling set by the model.** I spent real effort on\n\nthose four countermeasures and I'd write them the same way again — the model that\n\nfollows them produces a statistically solid result. But identical instructions\n\nproduced a 2.5× difference in outcome. The prompt is necessary; it isn't\n\nsufficient.\n\n**3. For judgement tasks, skepticism beats capability.** Moving up a capability\n\ntier helped — gpt-4o doubled its sibling's false-alarm removal at zero recall\n\ncost. But the property that made Gemma the best judge here isn't reasoning power\n\nor knowledge. It's willingness to contradict a premise supplied in the prompt.\n\nThat trait doesn't appear on any leaderboard I know of, which means **you cannot\npick a judge by reputation — you have to measure it on your own task.**\n\n**4. Design the schema, not just the prose.** Putting `reasoning`\n\nbefore\n\n`confirmed`\n\ncost me nothing and changes the model's process. Field order is\n\nprompt engineering.\n\nNext in this series: the OWASP Benchmark — 1,478 test cases, 701 of them built\n\nspecifically to trick tools like mine.\n\n*I'm Ali Afana — AI builder and security researcher, writing from Gaza. I\nbuild systems in public, measure them against ground truth, and keep the\nreceipts. This scanner is one project on a longer road — follow for what\ncomes next.*", "url": "https://wpnews.pro/news/i-told-the-ai-a-scanner-flagged-this-and-it-agreed-with-everything", "canonical_source": "https://dev.to/alimafana/i-told-the-ai-a-scanner-flagged-this-and-it-agreed-with-everything-4jn6", "published_at": "2026-08-27 11:26:31+00:00", "updated_at": "2026-08-27 11:48:22.349496+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/i-told-the-ai-a-scanner-flagged-this-and-it-agreed-with-everything", "markdown": "https://wpnews.pro/news/i-told-the-ai-a-scanner-flagged-this-and-it-agreed-with-everything.md", "text": "https://wpnews.pro/news/i-told-the-ai-a-scanner-flagged-this-and-it-agreed-with-everything.txt", "jsonld": "https://wpnews.pro/news/i-told-the-ai-a-scanner-flagged-this-and-it-agreed-with-everything.jsonld"}}