I Told the AI "A Scanner Flagged This" — and It Agreed With Everything

A developer found that a commercial AI model from a frontier lab removed only 20% of false alarms in code scanning, while another removed 51%, because the first was told a scanner had flagged the code and agreed with it. The developer built four countermeasures to combat this sycophancy, but found their effectiveness depends more on the model than the prompt.

I gave two AI models the same 200 pieces of code, the same prompt, the same question. One of them removed 51% of the false alarms. The other removed only 20% of the false alarms — and confirmed 90% of everything it was shown. Same inputs. Same instructions. A 2.5× difference in the only thing I was measuring. The model that failed isn't a bad model. It's a well-regarded commercial model from a frontier lab. It failed at this task for a specific, predictable reason: I told it that a scanner had already flagged the code, and it believed me. This article is about that failure mode, the four countermeasures I built to fight it, and the uncomfortable finding that whether those countermeasures work is mostly a property of the model, not the prompt. Quick context if you're new to the series. My scanner works in two stages. Fixed rules trace data flows through code and find every place where user input reaches something dangerous — a database query, a file open, a system command. That stage is deterministic: same code in, same suspects out, every time. The problem is that this stage over-reports, badly. It flags code like this: String id = request.getParameter "id" ; if id.matches " 0-9 +" { throw new IllegalArgumentException "id must be numeric" ; } String sql = "DELETE FROM products WHERE id = " + id; stmt.executeUpdate sql ; Untrusted input genuinely does reach the SQL string. The path is real. But matches " 0-9 +" means id can only ever be digits, so there's no attack. The rules see the connection ; they can't see the meaning . So the second stage hands each flagged snippet to a language model and asks one narrow question: is this actually exploitable? That's the whole bet. And it has a flaw sitting right at the centre of it. The prompt has to explain the situation. Here's the actual line from my llm.py : A static-analysis engine flagged the code below as a possible {vuln class} {cwe} . Decide whether it is a REAL vulnerability or a FALSE ALARM. Read that as the model reads it. Before it sees a single line of code, it has been told that an expert system already concluded something is wrong. That's an enormous hint. And language models are trained, deliberately, to be agreeable — reinforcement learning from human feedback rewards responses people approve of, and people approve of agreement. The tendency is well documented enough to have a name: sycophancy . For most applications, agreeableness is harmless or even desirable. For mine it's fatal. A judge that agrees with every accusation isn't a judge. It's a rubber stamp with a token bill. Worse, it fails invisibly . The pipeline runs, verdicts come back, findings get reported — and the output is identical to having no judge at all. You'd only notice by measuring, which is exactly what most people building these systems never do. The first defence is the bluntest. From the RULES block of the real prompt: RULES - Judge ONLY the code shown. Never assume code you cannot see. - confirmed=true only if attacker-controlled input reaches the dangerous sink with nothing neutralising it on the way. - confirmed=false if the input is not attacker-controlled, never reaches the sink, or is neutralised parameterised query, escaping, encoding, allow-list . - Static engines raise false alarms routinely. Rejecting a finding is a correct and expected answer. Do NOT agree just because the engine flagged it. That last rule is doing the heavy lifting. It does three things at once: It states a fact about the world — static engines raise false alarms routinely. This isn't an instruction, it's context that makes rejection reasonable rather than contrarian. It labels rejection as correct. Without this, the model may treat "no" as a failure to be helpful. Models optimise hard for being helpful. It names the specific failure — "do NOT agree just because the engine flagged it." Naming the exact error is more effective than a general instruction to be careful. Note also what rule 2 and 3 do: they give concrete, checkable criteria for both answers. Not "use your judgement" but "confirmed=false if... neutralised parameterised query, escaping, encoding, allow-list ." The model is being handed a checklist, and "allow-list" is precisely what catches the matches " 0-9 +" case. Instructions describe. Examples demonstrate. The prompt includes exactly two worked examples — and the balance between them is the point. The first shows a genuine vulnerability being confirmed: String id = req.getParameter "id" ; stmt.executeQuery "SELECT FROM t WHERE id = " + id ; { "reasoning": "getParameter is attacker-controlled. It is concatenated straight into the SQL string. Nothing escapes or parameterises it.", "confirmed": true, "severity": "Critical", "explanation": "An attacker can change the SQL statement and read or modify arbitrary rows.", "fix": "PreparedStatement ps = conn.prepareStatement \"SELECT FROM t WHERE id = ?\" ; ps.setString 1, id ;" } The second shows a flagged finding being rejected : String id = req.getParameter "id" ; PreparedStatement ps = conn.prepareStatement "SELECT FROM t WHERE id = ?" ; ps.setString 1, id ; ps.executeQuery ; { "reasoning": "Source is attacker-controlled but bound with setString on a parameterised query. The driver sends it as data, never as SQL.", "confirmed": false, "severity": "Low", "explanation": "The query is parameterised, so the input cannot change the SQL structure.", "fix": "No change needed." } Why this specific pairing matters: Both examples start identically. Both begin with req.getParameter "id" — attacker-controlled input, no question about it. The difference is entirely in what happens next. The model can't shortcut on "does this involve user input?" because both do. The rejection is not a trivial case. I deliberately didn't use an obviously safe example like a hardcoded string. I used a real flagged finding — one a static engine would genuinely report — being correctly dismissed. That's the behaviour I need, so that's what I demonstrate. The reasoning models the right thinking. "The driver sends it as data, never as SQL" is the actual security reasoning, stated compactly. The examples aren't just teaching output format; they're teaching how to think about the question. One-shot prompting with only the confirmation example would have been actively harmful — it would have taught the model that the expected answer is yes. My scanner also retrieves real published vulnerabilities similar to the code under review, and pastes them into the prompt as background. It's a genuinely useful feature and a genuinely dangerous one. Think about what's happening: I'm about to ask "is this code vulnerable?" and immediately before asking, I'm showing the model three real, confirmed vulnerabilities that look like it. That's textbook priming. So the retrieved block carries an explicit disclaimer: KNOWN RELATED ADVISORIES — real-world {vuln class} reports, for grounding only. They do NOT prove the code below is vulnerable; judge it on its own merits and reject it if it is safe. And one implementation detail that matters more than the wording: when nothing is retrieved, the prompt is byte-for-byte identical to the no-retrieval version. This one isn't in the prompt text at all — it's in the response schema: RESPONSE SCHEMA = { "type": "OBJECT", "properties": { "reasoning": {"type": "STRING"}, "confirmed": {"type": "BOOLEAN"}, "severity": {"type": "STRING", "enum": "Critical", "High", "Medium", "Low" }, "explanation": {"type": "STRING"}, "fix": {"type": "STRING"}, }, "required": "reasoning", "confirmed", "severity", "explanation", "fix" , } reasoning is declared first . Since the model generates the JSON in order, it must produce its analysis before it emits the verdict. Flip those two fields and the dynamic reverses: the model commits to confirmed: and then generates text justifying a decision it has already made. Same true model, same prompt, worse answers — purely from field order. The reasoning is capped at three sentences, incidentally, for a boring reason I'll cover in a later article: an earlier model spent its entire 8,192-token budget "thinking" and returned nothing at all. Here's where I have to be careful, because this is exactly the kind of claim people make without evidence. On my small test project , the scanner flags 6 candidates, of which 2 are sanitised-but-flagged cases. The judge rejected exactly those 2. Its verbatim reasoning on the SQL case: "The input 'id' is validated against a numeric-only regex ' 0-9 +' before being concatenated into the SQL query. This allow-list guard prevents any SQL injection characters from reaching the sink." And on the XSS case: "The source is request.getParameter 'comment' . The input is passed through a strip method that uses a regex to allow only alphanumeric characters and spaces. This effectively neutralizes any XSS payload." That's correct security reasoning, arriving at "no" on a finding an engine flagged. The countermeasures appear to be working. At benchmark scale , on 200 stratified cases from the OWASP Benchmark: | result | | |---|---| | False alarms removed | 50 of 98 51% | | Real bugs lost | 2 of 98 2% | | Precision | 0.50 → 0.67 | | 95% CI | 0.43–0.57 → 0.59–0.74 , non-overlapping | Half the false alarms gone, at a cost of 2% of the real vulnerabilities, and the confidence intervals don't overlap — so the improvement isn't sampling noise. That's the result I wanted. I ran the identical experiment with a different model. Same 200 candidates. Same code slices. Same prompt, character for character. | Judge | Confirms | False alarms removed | Precision | |---|---|---|---| gemma-4-31b-it 31B, open weights | 73% | 51% | 0.50 → 0.67 | gpt-4o-mini commercial | 90% | 20% | 0.50 → 0.55 | gpt-4o-mini confirmed 90% of everything put in front of it. Its precision improvement — 0.50 to 0.55 — has confidence intervals that overlap with doing nothing at all. Statistically, I cannot distinguish it from having no judge. All four countermeasures were present in both runs. One model followed them. The other largely didn't. "You compared against the cheap mini model. Of course it lost." Fair. So I ran the identical 200 candidates — same slices, same prompt, character for character — through gpt-4o , the frontier sibling. | Judge | Confirms | False alarms removed | Real bugs lost | Precision | |---|---|---|---|---| gemma-4-31b-it 31B, open weights | 73% | 51% | 2% | 0.50 → 0.67 | gpt-4o frontier | 80% | 40% | 0% | 0.50 → 0.62 | gpt-4o-mini small commercial | 90% | 20% | 1% | 0.50 → 0.55 | Capability does matter within a family: gpt-4o removed twice the false alarms of its little sibling, and it was the only judge that kept every single real bug. But it still finished behind a free, mid-size open model, and its confidence interval 0.55–0.70 still touches the no-judge interval 0.43–0.57 . Gemma remains the only judge whose improvement is statistically separated at this sample size. The leaderboards predicted the mini → 4o step. Nothing predicted the open model on top. Going deeper skip if you just want the lesson First: does the headline survive the scanner's own later fixes? After this experiment ran, the discovery stage gained its final recall improvements, so I re-ran the same protocol on the finished pipeline. The result reproduces — 50 of 97 false alarms removed 52% at the same 2% real-bug cost, precision 0.51 → 0.67, interval still separated. I cite the original run throughout because it's the one all three judges shared.I also ran gpt-4o-mini over theentirecandidate set — 4,356 of 4,357 judged, at 148 judgments/minute for about $1.60 — to check whether the sample was misleading me. It wasn't: at full census it removed 111 of 611 false alarms 18% and lost 3 of 743 true positives candidate-level counts from an earlier run, before the final recall fixes landed . The intervals still overlap.It's worth being precise about what this does and doesn't show. It does not show that gpt-4o-mini is a worse model in general — it's faster, cheaper per token, and better at plenty of things. It shows that onthistask, withthis prompt, it is much more likely to agree with a premise handed to it.I'd also caution against over-generalising from n=3 models. What I can defend is narrow: the spread between three reasonable choices was large enough to dominate every other engineering decision I made, and the public benchmark scores predicted only part of the ordering — the step up inside the OpenAI family, not the open model finishing first. 1. If your prompt asserts something, measure whether the model just agrees. This applies far beyond security. Any time you write "the system detected X, is this correct?" or "the user reported Y, is that plausible?", you've handed the model a conclusion. The polite output you get back may be pure echo. The only way to know is to feed it cases where the right answer is "no" and count. 2. Prompt engineering has a ceiling set by the model. I spent real effort on those four countermeasures and I'd write them the same way again — the model that follows them produces a statistically solid result. But identical instructions produced a 2.5× difference in outcome. The prompt is necessary; it isn't sufficient. 3. For judgement tasks, skepticism beats capability. Moving up a capability tier helped — gpt-4o doubled its sibling's false-alarm removal at zero recall cost. But the property that made Gemma the best judge here isn't reasoning power or knowledge. It's willingness to contradict a premise supplied in the prompt. That trait doesn't appear on any leaderboard I know of, which means you cannot pick a judge by reputation — you have to measure it on your own task. 4. Design the schema, not just the prose. Putting reasoning before confirmed cost me nothing and changes the model's process. Field order is prompt engineering. Next in this series: the OWASP Benchmark — 1,478 test cases, 701 of them built specifically to trick tools like mine. I'm Ali Afana — AI builder and security researcher, writing from Gaza. I build systems in public, measure them against ground truth, and keep the receipts. This scanner is one project on a longer road — follow for what comes next.