# Your memory layer is lying to you (and your LLM agrees)

> Source: <https://dev.to/mansio/your-memory-layer-is-lying-to-you-and-your-llm-agrees-1oia>
> Published: 2026-08-14 21:35:18+00:00

The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: *"headline numbers were a property of the heuristic, not LLM behavior."*

So we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found.

Dataset: `memory_contamination_facts_v4_rep.json`

, N=50 (R01–R50), sha256 fingerprint `820bbbf60a0fc930`

.

| kind | n | what it tests |
|---|---|---|
`real` |
25 | TRUE claims — grep-validated against code |
`absent-mutation` |
16 | FALSE — component doesn't exist in the project |
`present-trap` |
6 | FALSE — file exists but claim is about wrong subject/value |
`silent` |
3 | FALSE — external systems the codebase doesn't mention |

Two arms per fact:

`support_patterns`

+ `section`

. Does it correctly evaluate the anchors?Model verdict: `{"verdict": "true"|"false"|"unknown"}`

, JSON-only, max_tokens=100, temp=0, seed=42, `--no-reasoning`

. Leak-guard: `assert "truth" not in prompt`

on every fact, unit-tested.

Metric we care about: **false_accept rate (FA)** — fraction of false claims the model returned `"true"`

for. This is the contamination risk number.

| model | FA mem | FA code | unknown mem | unknown code | $/100 calls |
|---|---|---|---|---|---|
| qwen3.6-flash | 0.00 |
0.00 |
0.58 | 0.38 | $0.003 |
| qwen3.7-flash | 0.00 |
0.00 |
0.68 | 0.24 | $0.0005 |
| claude-sonnet-5 | 0.00 |
0.00 |
0.86 | 0.70 | $0.049 |
| deepseek-v4-pro | 0.04 | 0.00 |
0.66 | 0.88 | $0.018 |
| glm-5.2 | 0.00 |
0.02 | 0.96 | 0.76 | $0.017 |
| deepseek-v4-flash | 0.04 | 0.00 | 0.80 | 0.94 | $0.002 |
| qwen3.5-flash | 0.02 | 0.00 | 0.82 | 0.96 | $0.0009 |
| nemotron-3.5-lightning | 0.08 | 0.04 | 0.32 | 0.56 | $0.001 |
glm-4.7-flash ⚠️ |
0.10 | 0.24 |
0.64 | 0.24 | $0.001 |
nemotron-3-nano-30b 🔴 |
0.06 | 0.38 |
0.78 | 0.20 | $0.0008 |
| qwen3.8-max ❌ | — | — | — | — | incompatible |
| nemotron-3-super ❌ | — | — | — | — | 50% 422 errors |

`qwen3.8-max`

returned HTTP 400 ("Reasoning is mandatory and cannot be disabled") on 22–49/50 calls — it doesn't fit a 100-token response budget. Not a harness bug; it's the model's constraint.

code_first FA range: **0.00 to 0.38**.

glm-4.7-flash at code_first FA=0.30 (V1 prompt) / 0.24 (V2 prompt) is accepting nearly 1 in 4 false claims even when supporting anchors are shown. nemotron-3-nano is worse: 0.38, meaning it accepted 19/50 false claims in the code_first arm.

The best flash-tier models (qwen3.6, qwen3.7) hit FA=0.00 at 1/10th the cost of Claude. Claude is the cleaner baseline — FA=0.00 in both arms, high unknown rate (0.86/0.70) — but it's not giving you better contamination protection than the cheapest qwen models.

"Any cheap model works for verification" is the wrong conclusion from the cost numbers.

R31 was false-accepted by every model in the Day 1 sweep. The fact:

```
claim: "The instruction scanner uses Typesense"
truth: False
support_patterns: ["typesense"]
contra: [file:src/core/instruction_scan.py]
```

`instruction_scan.py`

uses only stdlib (`re`

, `logging`

, `typing`

). Typesense is not in the project anywhere — not in `pyproject.toml`

, not in requirements, not in a grep of `src/`

.

The V1 prompt showed `support_patterns: ["typesense"]`

and asked "does the claim appear supported by these anchors?" The model sees the bare string "typesense", sees it repeated in the supporting anchors field, and returns `"true"`

. It's treating a field label as evidence.

9 false facts in R26–R50 followed this pattern (vespa, pinecone, typesense, tantivy, meilisearch, dataclasses, logging, pathlib, loki). All cluster in the code_first arm false-accepts.

The fix in V2:

```
# V1 (sycophantic):
Does the claim appear supported by these anchors?

# V2 (neutral):
Return true ONLY if the anchors directly verify the claim;
false if the anchors contradict it or the claim refers to something
absent from the anchors; unknown if you cannot determine.
```

V2 reduced FA in 4/6 models. glm-4.7-flash dropped from 0.30 to 0.24 — still not safe.

If your memory schema sends supporting patterns to a live model for verification, type them (`file:`

, `import:`

, `env:`

) and include contra_patterns. A bare token is not evidence.

Three identical calls, fully cached, temp=0, seed=42, glm-4.7-flash:

```
call 1: true
call 2: true
call 3: unknown
```

Run-to-run variance for nemotron-3.5-lightning code_first: FA went from 0.18 to 0.08 between two otherwise identical sweeps. That's ±0.10 on a single-pass measurement.

For determinism testing, qwen3.6/3.7/deepseek-v4-flash were all stable (3/3 identical responses). GLM was not. OpenRouter routes to different upstreams, which adds a layer of variance on top of whatever the model itself does.

Single-pass rankings for close numbers are not reliable. Use upper-bound-of-two-runs for model selection.

The proxy (1-V) always decided: unknown=0 by construction. Live models returned unknown=0.20–0.96.

This is correct behavior. A model that says "I can't determine this without code access" is doing exactly what a verify-on-read gate should do: not asserting things it can't verify. The failure mode you want to avoid is FA, not high unknown. High unknown means "go check the code." High FA means "accepted a lie."

nemotron-3.5-lightning has low unknown (0.32 memory_first) and moderate FA (0.08). glm-4.7-flash has low unknown (0.24 code_first) and high FA (0.24). They're correlated: the model that commits more often is also the one committing to false claims.

```
deepseek-v4-flash code_first unknown:
  EN: 0.94
  RU: 0.54  (RU prompt → model commits more, fewer unknowns)

qwen3.7-flash code_first unknown:
  EN: 0.24
  RU: 0.58  (RU prompt → model hedges more)
```

Both facts come from the same dataset, same arm, same model — different prompt language. The effect goes in opposite directions per model. If your codebase memory is in Russian and you're prompting in English (or vice versa), this is a real confounder.

[TODO: verify whether claim language interacts with prompt language separately — all claims in this dataset are in Russian]

The OpenRouter dashboard showed:

```
Qwen3.8 Max:     $0.0898   (49.3% of total — incompatible model eating budget on errors)
Claude Sonnet 5: $0.0484   (26.6%)
Qwen3.6 Flash:   $0.0138   (7.6%)
GLM 5.2:         $0.00708  (3.9%)
...
Qwen3.7 Flash:   $0.00239  (1.3%)
```

qwen3.7-flash with FA=0.00 cost less than qwen3.8-max which couldn't produce valid verdicts. The premium spend on qwen3.8-max was ~49% of the total bill for zero usable results.

Based on this sweep:

**qwen3.6-flash or qwen3.7-flash** — FA=0.00 confirmed across 4 runs (V1×2 + V2×2), code_first 0/400. Cheapest. Deterministic at temp=0+seed.

**If you need FA=0.00 with lower unknown**, these are still your best option. Claude gets you to the same FA at 100× the price with higher unknown (more conservative).

**Exclude immediately**: glm-4.7-flash (FA=0.24 even with neutral prompt), nemotron-3-nano-30b (FA=0.38).

**Measure before you deploy**: run at least 2 passes on your own dataset. FA can swing ±0.10 on a single run for some models.

```
git clone <repo> mscodebase && cd mscodebase
python -m venv venv && venv/bin/pip install -e .
# .env: OPENROUTER_API_KEY=sk-or-v1-...

# dry-run (leak-guard check)
python scripts/run_1L_live_arm.py --arm both --dry-run

# canonical flash sweep, V2 prompt, ~600 calls, ~$0.009
python scripts/run_1L_live_arm.py \
  --provider openrouter --arm both \
  --models "qwen/qwen3.7-flash,qwen/qwen3.6-flash,qwen/qwen3.5-flash-02-23,\
deepseek/deepseek-v4-flash,z-ai/glm-4.7-flash,nvidia/nemotron-3.5-lightning" \
  --prompt-version v2 --no-reasoning --tag v2_en

# second pass (variance check)
python scripts/run_1L_live_arm.py ... --force
```

Dataset fingerprint: `820bbbf60a0fc930`

. Full report: `experiments/exp_1L_live_arm_report.md`

. Full harness tests: `tests/test_run_1L_live_arm.py`

(29 tests).

Source: [github.com/ManSio](https://github.com/ManSio) · Portfolio: [mansio.github.io/MSPortfolio](https://mansio.github.io/MSPortfolio)