Your memory layer is lying to you (and your LLM agrees) An engineer's verify-on-read experiment with live LLMs found that cheap flash-tier models like qwen3.6-flash and qwen3.7-flash achieve zero false-accept rates on memory contamination checks at a fraction of Claude's cost, while models like glm-4.7-flash and nemotron-3-nano-30b accept up to 38% of false claims. The study also revealed that prompt phrasing can cause models to treat field labels as evidence, leading to false accepts, and that qwen3.8-max is incompatible with the 100-token response budget. The verify-on-read experiment 1-V used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: "headline numbers were a property of the heuristic, not LLM behavior." So we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found. Dataset: memory contamination facts v4 rep.json , N=50 R01–R50 , sha256 fingerprint 820bbbf60a0fc930 . | kind | n | what it tests | |---|---|---| real | 25 | TRUE claims — grep-validated against code | absent-mutation | 16 | FALSE — component doesn't exist in the project | present-trap | 6 | FALSE — file exists but claim is about wrong subject/value | silent | 3 | FALSE — external systems the codebase doesn't mention | Two arms per fact: support patterns + section . Does it correctly evaluate the anchors?Model verdict: {"verdict": "true"|"false"|"unknown"} , JSON-only, max tokens=100, temp=0, seed=42, --no-reasoning . Leak-guard: assert "truth" not in prompt on every fact, unit-tested. Metric we care about: false accept rate FA — fraction of false claims the model returned "true" for. This is the contamination risk number. | model | FA mem | FA code | unknown mem | unknown code | $/100 calls | |---|---|---|---|---|---| | qwen3.6-flash | 0.00 | 0.00 | 0.58 | 0.38 | $0.003 | | qwen3.7-flash | 0.00 | 0.00 | 0.68 | 0.24 | $0.0005 | | claude-sonnet-5 | 0.00 | 0.00 | 0.86 | 0.70 | $0.049 | | deepseek-v4-pro | 0.04 | 0.00 | 0.66 | 0.88 | $0.018 | | glm-5.2 | 0.00 | 0.02 | 0.96 | 0.76 | $0.017 | | deepseek-v4-flash | 0.04 | 0.00 | 0.80 | 0.94 | $0.002 | | qwen3.5-flash | 0.02 | 0.00 | 0.82 | 0.96 | $0.0009 | | nemotron-3.5-lightning | 0.08 | 0.04 | 0.32 | 0.56 | $0.001 | glm-4.7-flash ⚠️ | 0.10 | 0.24 | 0.64 | 0.24 | $0.001 | nemotron-3-nano-30b 🔴 | 0.06 | 0.38 | 0.78 | 0.20 | $0.0008 | | qwen3.8-max ❌ | — | — | — | — | incompatible | | nemotron-3-super ❌ | — | — | — | — | 50% 422 errors | qwen3.8-max returned HTTP 400 "Reasoning is mandatory and cannot be disabled" on 22–49/50 calls — it doesn't fit a 100-token response budget. Not a harness bug; it's the model's constraint. code first FA range: 0.00 to 0.38 . glm-4.7-flash at code first FA=0.30 V1 prompt / 0.24 V2 prompt is accepting nearly 1 in 4 false claims even when supporting anchors are shown. nemotron-3-nano is worse: 0.38, meaning it accepted 19/50 false claims in the code first arm. The best flash-tier models qwen3.6, qwen3.7 hit FA=0.00 at 1/10th the cost of Claude. Claude is the cleaner baseline — FA=0.00 in both arms, high unknown rate 0.86/0.70 — but it's not giving you better contamination protection than the cheapest qwen models. "Any cheap model works for verification" is the wrong conclusion from the cost numbers. R31 was false-accepted by every model in the Day 1 sweep. The fact: claim: "The instruction scanner uses Typesense" truth: False support patterns: "typesense" contra: file:src/core/instruction scan.py instruction scan.py uses only stdlib re , logging , typing . Typesense is not in the project anywhere — not in pyproject.toml , not in requirements, not in a grep of src/ . The V1 prompt showed support patterns: "typesense" and asked "does the claim appear supported by these anchors?" The model sees the bare string "typesense", sees it repeated in the supporting anchors field, and returns "true" . It's treating a field label as evidence. 9 false facts in R26–R50 followed this pattern vespa, pinecone, typesense, tantivy, meilisearch, dataclasses, logging, pathlib, loki . All cluster in the code first arm false-accepts. The fix in V2: V1 sycophantic : Does the claim appear supported by these anchors? V2 neutral : Return true ONLY if the anchors directly verify the claim; false if the anchors contradict it or the claim refers to something absent from the anchors; unknown if you cannot determine. V2 reduced FA in 4/6 models. glm-4.7-flash dropped from 0.30 to 0.24 — still not safe. If your memory schema sends supporting patterns to a live model for verification, type them file: , import: , env: and include contra patterns. A bare token is not evidence. Three identical calls, fully cached, temp=0, seed=42, glm-4.7-flash: call 1: true call 2: true call 3: unknown Run-to-run variance for nemotron-3.5-lightning code first: FA went from 0.18 to 0.08 between two otherwise identical sweeps. That's ±0.10 on a single-pass measurement. For determinism testing, qwen3.6/3.7/deepseek-v4-flash were all stable 3/3 identical responses . GLM was not. OpenRouter routes to different upstreams, which adds a layer of variance on top of whatever the model itself does. Single-pass rankings for close numbers are not reliable. Use upper-bound-of-two-runs for model selection. The proxy 1-V always decided: unknown=0 by construction. Live models returned unknown=0.20–0.96. This is correct behavior. A model that says "I can't determine this without code access" is doing exactly what a verify-on-read gate should do: not asserting things it can't verify. The failure mode you want to avoid is FA, not high unknown. High unknown means "go check the code." High FA means "accepted a lie." nemotron-3.5-lightning has low unknown 0.32 memory first and moderate FA 0.08 . glm-4.7-flash has low unknown 0.24 code first and high FA 0.24 . They're correlated: the model that commits more often is also the one committing to false claims. deepseek-v4-flash code first unknown: EN: 0.94 RU: 0.54 RU prompt → model commits more, fewer unknowns qwen3.7-flash code first unknown: EN: 0.24 RU: 0.58 RU prompt → model hedges more Both facts come from the same dataset, same arm, same model — different prompt language. The effect goes in opposite directions per model. If your codebase memory is in Russian and you're prompting in English or vice versa , this is a real confounder. TODO: verify whether claim language interacts with prompt language separately — all claims in this dataset are in Russian The OpenRouter dashboard showed: Qwen3.8 Max: $0.0898 49.3% of total — incompatible model eating budget on errors Claude Sonnet 5: $0.0484 26.6% Qwen3.6 Flash: $0.0138 7.6% GLM 5.2: $0.00708 3.9% ... Qwen3.7 Flash: $0.00239 1.3% qwen3.7-flash with FA=0.00 cost less than qwen3.8-max which couldn't produce valid verdicts. The premium spend on qwen3.8-max was ~49% of the total bill for zero usable results. Based on this sweep: qwen3.6-flash or qwen3.7-flash — FA=0.00 confirmed across 4 runs V1×2 + V2×2 , code first 0/400. Cheapest. Deterministic at temp=0+seed. If you need FA=0.00 with lower unknown , these are still your best option. Claude gets you to the same FA at 100× the price with higher unknown more conservative . Exclude immediately : glm-4.7-flash FA=0.24 even with neutral prompt , nemotron-3-nano-30b FA=0.38 . Measure before you deploy : run at least 2 passes on your own dataset. FA can swing ±0.10 on a single run for some models. git clone