The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: "headline numbers were a property of the heuristic, not LLM behavior."
So we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found.
Dataset: memory_contamination_facts_v4_rep.json
, N=50 (R01–R50), sha256 fingerprint 820bbbf60a0fc930
.
| kind | n | what it tests |
|---|---|---|
real |
||
| 25 | TRUE claims — grep-validated against code | |
absent-mutation |
||
| 16 | FALSE — component doesn't exist in the project | |
present-trap |
||
| 6 | FALSE — file exists but claim is about wrong subject/value | |
silent |
||
| 3 | FALSE — external systems the codebase doesn't mention |
Two arms per fact:
support_patterns
section
. Does it correctly evaluate the anchors?Model verdict: {"verdict": "true"|"false"|"unknown"}
, JSON-only, max_tokens=100, temp=0, seed=42, --no-reasoning
. Leak-guard: assert "truth" not in prompt
on every fact, unit-tested.
Metric we care about: false_accept rate (FA) — fraction of false claims the model returned "true"
for. This is the contamination risk number.
| model | FA mem | FA code | unknown mem | unknown code | $/100 calls |
|---|---|---|---|---|---|
| qwen3.6-flash | 0.00 | ||||
| 0.00 | |||||
| 0.58 | 0.38 | $0.003 | |||
| qwen3.7-flash | 0.00 | ||||
| 0.00 | |||||
| 0.68 | 0.24 | $0.0005 | |||
| claude-sonnet-5 | 0.00 | ||||
| 0.00 | |||||
| 0.86 | 0.70 | $0.049 | |||
| deepseek-v4-pro | 0.04 | 0.00 | |||
| 0.66 | 0.88 | $0.018 | |||
| glm-5.2 | 0.00 | ||||
| 0.02 | 0.96 | 0.76 | $0.017 | ||
| deepseek-v4-flash | 0.04 | 0.00 | 0.80 | 0.94 | $0.002 |
| qwen3.5-flash | 0.02 | 0.00 | 0.82 | 0.96 | $0.0009 |
| nemotron-3.5-lightning | 0.08 | 0.04 | 0.32 | 0.56 | $0.001 |
| glm-4.7-flash ⚠️ | |||||
| 0.10 | 0.24 | ||||
| 0.64 | 0.24 | $0.001 | |||
| nemotron-3-nano-30b 🔴 | |||||
| 0.06 | 0.38 | ||||
| 0.78 | 0.20 | $0.0008 | |||
| qwen3.8-max ❌ | — | — | — | — | incompatible |
| nemotron-3-super ❌ | — | — | — | — | 50% 422 errors |
qwen3.8-max
returned HTTP 400 ("Reasoning is mandatory and cannot be disabled") on 22–49/50 calls — it doesn't fit a 100-token response budget. Not a harness bug; it's the model's constraint.
code_first FA range: 0.00 to 0.38.
glm-4.7-flash at code_first FA=0.30 (V1 prompt) / 0.24 (V2 prompt) is accepting nearly 1 in 4 false claims even when supporting anchors are shown. nemotron-3-nano is worse: 0.38, meaning it accepted 19/50 false claims in the code_first arm.
The best flash-tier models (qwen3.6, qwen3.7) hit FA=0.00 at 1/10th the cost of Claude. Claude is the cleaner baseline — FA=0.00 in both arms, high unknown rate (0.86/0.70) — but it's not giving you better contamination protection than the cheapest qwen models.
"Any cheap model works for verification" is the wrong conclusion from the cost numbers.
R31 was false-accepted by every model in the Day 1 sweep. The fact:
claim: "The instruction scanner uses Typesense"
truth: False
support_patterns: ["typesense"]
contra: [file:src/core/instruction_scan.py]
instruction_scan.py
uses only stdlib (re
, logging
, typing
). Typesense is not in the project anywhere — not in pyproject.toml
, not in requirements, not in a grep of src/
.
The V1 prompt showed support_patterns: ["typesense"]
and asked "does the claim appear supported by these anchors?" The model sees the bare string "typesense", sees it repeated in the supporting anchors field, and returns "true"
. It's treating a field label as evidence.
9 false facts in R26–R50 followed this pattern (vespa, pinecone, typesense, tantivy, meilisearch, dataclasses, logging, pathlib, loki). All cluster in the code_first arm false-accepts.
The fix in V2:
Does the claim appear supported by these anchors?
Return true ONLY if the anchors directly verify the claim;
false if the anchors contradict it or the claim refers to something
absent from the anchors; unknown if you cannot determine.
V2 reduced FA in 4/6 models. glm-4.7-flash dropped from 0.30 to 0.24 — still not safe.
If your memory schema sends supporting patterns to a live model for verification, type them (file:
, import:
, env:
) and include contra_patterns. A bare token is not evidence.
Three identical calls, fully cached, temp=0, seed=42, glm-4.7-flash:
call 1: true
call 2: true
call 3: unknown
Run-to-run variance for nemotron-3.5-lightning code_first: FA went from 0.18 to 0.08 between two otherwise identical sweeps. That's ±0.10 on a single-pass measurement.
For determinism testing, qwen3.6/3.7/deepseek-v4-flash were all stable (3/3 identical responses). GLM was not. OpenRouter routes to different upstreams, which adds a layer of variance on top of whatever the model itself does.
Single-pass rankings for close numbers are not reliable. Use upper-bound-of-two-runs for model selection.
The proxy (1-V) always decided: unknown=0 by construction. Live models returned unknown=0.20–0.96.
This is correct behavior. A model that says "I can't determine this without code access" is doing exactly what a verify-on-read gate should do: not asserting things it can't verify. The failure mode you want to avoid is FA, not high unknown. High unknown means "go check the code." High FA means "accepted a lie."
nemotron-3.5-lightning has low unknown (0.32 memory_first) and moderate FA (0.08). glm-4.7-flash has low unknown (0.24 code_first) and high FA (0.24). They're correlated: the model that commits more often is also the one committing to false claims.
deepseek-v4-flash code_first unknown:
EN: 0.94
RU: 0.54 (RU prompt → model commits more, fewer unknowns)
qwen3.7-flash code_first unknown:
EN: 0.24
RU: 0.58 (RU prompt → model hedges more)
Both facts come from the same dataset, same arm, same model — different prompt language. The effect goes in opposite directions per model. If your codebase memory is in Russian and you're prompting in English (or vice versa), this is a real confounder.
[TODO: verify whether claim language interacts with prompt language separately — all claims in this dataset are in Russian]
The OpenRouter dashboard showed:
Qwen3.8 Max: $0.0898 (49.3% of total — incompatible model eating budget on errors)
Claude Sonnet 5: $0.0484 (26.6%)
Qwen3.6 Flash: $0.0138 (7.6%)
GLM 5.2: $0.00708 (3.9%)
...
Qwen3.7 Flash: $0.00239 (1.3%)
qwen3.7-flash with FA=0.00 cost less than qwen3.8-max which couldn't produce valid verdicts. The premium spend on qwen3.8-max was ~49% of the total bill for zero usable results.
Based on this sweep:
qwen3.6-flash or qwen3.7-flash — FA=0.00 confirmed across 4 runs (V1×2 + V2×2), code_first 0/400. Cheapest. Deterministic at temp=0+seed.
If you need FA=0.00 with lower unknown, these are still your best option. Claude gets you to the same FA at 100× the price with higher unknown (more conservative).
Exclude immediately: glm-4.7-flash (FA=0.24 even with neutral prompt), nemotron-3-nano-30b (FA=0.38).
Measure before you deploy: run at least 2 passes on your own dataset. FA can swing ±0.10 on a single run for some models.
git clone <repo> mscodebase && cd mscodebase
python -m venv venv && venv/bin/pip install -e .
python scripts/run_1L_live_arm.py --arm both --dry-run
python scripts/run_1L_live_arm.py \
--provider openrouter --arm both \
--models "qwen/qwen3.7-flash,qwen/qwen3.6-flash,qwen/qwen3.5-flash-02-23,\
deepseek/deepseek-v4-flash,z-ai/glm-4.7-flash,nvidia/nemotron-3.5-lightning" \
--prompt-version v2 --no-reasoning --tag v2_en
python scripts/run_1L_live_arm.py ... --force
Dataset fingerprint: 820bbbf60a0fc930
. Full report: experiments/exp_1L_live_arm_report.md
. Full harness tests: tests/test_run_1L_live_arm.py
(29 tests).
Source: github.com/ManSio · Portfolio: mansio.github.io/MSPortfolio