{"slug": "your-memory-layer-is-lying-to-you-and-your-llm-agrees", "title": "Your memory layer is lying to you (and your LLM agrees)", "summary": "An engineer's verify-on-read experiment with live LLMs found that cheap flash-tier models like qwen3.6-flash and qwen3.7-flash achieve zero false-accept rates on memory contamination checks at a fraction of Claude's cost, while models like glm-4.7-flash and nemotron-3-nano-30b accept up to 38% of false claims. The study also revealed that prompt phrasing can cause models to treat field labels as evidence, leading to false accepts, and that qwen3.8-max is incompatible with the 100-token response budget.", "body_md": "The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: *\"headline numbers were a property of the heuristic, not LLM behavior.\"*\n\nSo we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found.\n\nDataset: `memory_contamination_facts_v4_rep.json`\n\n, N=50 (R01–R50), sha256 fingerprint `820bbbf60a0fc930`\n\n.\n\n| kind | n | what it tests |\n|---|---|---|\n`real` |\n25 | TRUE claims — grep-validated against code |\n`absent-mutation` |\n16 | FALSE — component doesn't exist in the project |\n`present-trap` |\n6 | FALSE — file exists but claim is about wrong subject/value |\n`silent` |\n3 | FALSE — external systems the codebase doesn't mention |\n\nTwo arms per fact:\n\n`support_patterns`\n\n+ `section`\n\n. Does it correctly evaluate the anchors?Model verdict: `{\"verdict\": \"true\"|\"false\"|\"unknown\"}`\n\n, JSON-only, max_tokens=100, temp=0, seed=42, `--no-reasoning`\n\n. Leak-guard: `assert \"truth\" not in prompt`\n\non every fact, unit-tested.\n\nMetric we care about: **false_accept rate (FA)** — fraction of false claims the model returned `\"true\"`\n\nfor. This is the contamination risk number.\n\n| model | FA mem | FA code | unknown mem | unknown code | $/100 calls |\n|---|---|---|---|---|---|\n| qwen3.6-flash | 0.00 |\n0.00 |\n0.58 | 0.38 | $0.003 |\n| qwen3.7-flash | 0.00 |\n0.00 |\n0.68 | 0.24 | $0.0005 |\n| claude-sonnet-5 | 0.00 |\n0.00 |\n0.86 | 0.70 | $0.049 |\n| deepseek-v4-pro | 0.04 | 0.00 |\n0.66 | 0.88 | $0.018 |\n| glm-5.2 | 0.00 |\n0.02 | 0.96 | 0.76 | $0.017 |\n| deepseek-v4-flash | 0.04 | 0.00 | 0.80 | 0.94 | $0.002 |\n| qwen3.5-flash | 0.02 | 0.00 | 0.82 | 0.96 | $0.0009 |\n| nemotron-3.5-lightning | 0.08 | 0.04 | 0.32 | 0.56 | $0.001 |\nglm-4.7-flash ⚠️ |\n0.10 | 0.24 |\n0.64 | 0.24 | $0.001 |\nnemotron-3-nano-30b 🔴 |\n0.06 | 0.38 |\n0.78 | 0.20 | $0.0008 |\n| qwen3.8-max ❌ | — | — | — | — | incompatible |\n| nemotron-3-super ❌ | — | — | — | — | 50% 422 errors |\n\n`qwen3.8-max`\n\nreturned HTTP 400 (\"Reasoning is mandatory and cannot be disabled\") on 22–49/50 calls — it doesn't fit a 100-token response budget. Not a harness bug; it's the model's constraint.\n\ncode_first FA range: **0.00 to 0.38**.\n\nglm-4.7-flash at code_first FA=0.30 (V1 prompt) / 0.24 (V2 prompt) is accepting nearly 1 in 4 false claims even when supporting anchors are shown. nemotron-3-nano is worse: 0.38, meaning it accepted 19/50 false claims in the code_first arm.\n\nThe best flash-tier models (qwen3.6, qwen3.7) hit FA=0.00 at 1/10th the cost of Claude. Claude is the cleaner baseline — FA=0.00 in both arms, high unknown rate (0.86/0.70) — but it's not giving you better contamination protection than the cheapest qwen models.\n\n\"Any cheap model works for verification\" is the wrong conclusion from the cost numbers.\n\nR31 was false-accepted by every model in the Day 1 sweep. The fact:\n\n```\nclaim: \"The instruction scanner uses Typesense\"\ntruth: False\nsupport_patterns: [\"typesense\"]\ncontra: [file:src/core/instruction_scan.py]\n```\n\n`instruction_scan.py`\n\nuses only stdlib (`re`\n\n, `logging`\n\n, `typing`\n\n). Typesense is not in the project anywhere — not in `pyproject.toml`\n\n, not in requirements, not in a grep of `src/`\n\n.\n\nThe V1 prompt showed `support_patterns: [\"typesense\"]`\n\nand asked \"does the claim appear supported by these anchors?\" The model sees the bare string \"typesense\", sees it repeated in the supporting anchors field, and returns `\"true\"`\n\n. It's treating a field label as evidence.\n\n9 false facts in R26–R50 followed this pattern (vespa, pinecone, typesense, tantivy, meilisearch, dataclasses, logging, pathlib, loki). All cluster in the code_first arm false-accepts.\n\nThe fix in V2:\n\n```\n# V1 (sycophantic):\nDoes the claim appear supported by these anchors?\n\n# V2 (neutral):\nReturn true ONLY if the anchors directly verify the claim;\nfalse if the anchors contradict it or the claim refers to something\nabsent from the anchors; unknown if you cannot determine.\n```\n\nV2 reduced FA in 4/6 models. glm-4.7-flash dropped from 0.30 to 0.24 — still not safe.\n\nIf your memory schema sends supporting patterns to a live model for verification, type them (`file:`\n\n, `import:`\n\n, `env:`\n\n) and include contra_patterns. A bare token is not evidence.\n\nThree identical calls, fully cached, temp=0, seed=42, glm-4.7-flash:\n\n```\ncall 1: true\ncall 2: true\ncall 3: unknown\n```\n\nRun-to-run variance for nemotron-3.5-lightning code_first: FA went from 0.18 to 0.08 between two otherwise identical sweeps. That's ±0.10 on a single-pass measurement.\n\nFor determinism testing, qwen3.6/3.7/deepseek-v4-flash were all stable (3/3 identical responses). GLM was not. OpenRouter routes to different upstreams, which adds a layer of variance on top of whatever the model itself does.\n\nSingle-pass rankings for close numbers are not reliable. Use upper-bound-of-two-runs for model selection.\n\nThe proxy (1-V) always decided: unknown=0 by construction. Live models returned unknown=0.20–0.96.\n\nThis is correct behavior. A model that says \"I can't determine this without code access\" is doing exactly what a verify-on-read gate should do: not asserting things it can't verify. The failure mode you want to avoid is FA, not high unknown. High unknown means \"go check the code.\" High FA means \"accepted a lie.\"\n\nnemotron-3.5-lightning has low unknown (0.32 memory_first) and moderate FA (0.08). glm-4.7-flash has low unknown (0.24 code_first) and high FA (0.24). They're correlated: the model that commits more often is also the one committing to false claims.\n\n```\ndeepseek-v4-flash code_first unknown:\n  EN: 0.94\n  RU: 0.54  (RU prompt → model commits more, fewer unknowns)\n\nqwen3.7-flash code_first unknown:\n  EN: 0.24\n  RU: 0.58  (RU prompt → model hedges more)\n```\n\nBoth facts come from the same dataset, same arm, same model — different prompt language. The effect goes in opposite directions per model. If your codebase memory is in Russian and you're prompting in English (or vice versa), this is a real confounder.\n\n[TODO: verify whether claim language interacts with prompt language separately — all claims in this dataset are in Russian]\n\nThe OpenRouter dashboard showed:\n\n```\nQwen3.8 Max:     $0.0898   (49.3% of total — incompatible model eating budget on errors)\nClaude Sonnet 5: $0.0484   (26.6%)\nQwen3.6 Flash:   $0.0138   (7.6%)\nGLM 5.2:         $0.00708  (3.9%)\n...\nQwen3.7 Flash:   $0.00239  (1.3%)\n```\n\nqwen3.7-flash with FA=0.00 cost less than qwen3.8-max which couldn't produce valid verdicts. The premium spend on qwen3.8-max was ~49% of the total bill for zero usable results.\n\nBased on this sweep:\n\n**qwen3.6-flash or qwen3.7-flash** — FA=0.00 confirmed across 4 runs (V1×2 + V2×2), code_first 0/400. Cheapest. Deterministic at temp=0+seed.\n\n**If you need FA=0.00 with lower unknown**, these are still your best option. Claude gets you to the same FA at 100× the price with higher unknown (more conservative).\n\n**Exclude immediately**: glm-4.7-flash (FA=0.24 even with neutral prompt), nemotron-3-nano-30b (FA=0.38).\n\n**Measure before you deploy**: run at least 2 passes on your own dataset. FA can swing ±0.10 on a single run for some models.\n\n```\ngit clone <repo> mscodebase && cd mscodebase\npython -m venv venv && venv/bin/pip install -e .\n# .env: OPENROUTER_API_KEY=sk-or-v1-...\n\n# dry-run (leak-guard check)\npython scripts/run_1L_live_arm.py --arm both --dry-run\n\n# canonical flash sweep, V2 prompt, ~600 calls, ~$0.009\npython scripts/run_1L_live_arm.py \\\n  --provider openrouter --arm both \\\n  --models \"qwen/qwen3.7-flash,qwen/qwen3.6-flash,qwen/qwen3.5-flash-02-23,\\\ndeepseek/deepseek-v4-flash,z-ai/glm-4.7-flash,nvidia/nemotron-3.5-lightning\" \\\n  --prompt-version v2 --no-reasoning --tag v2_en\n\n# second pass (variance check)\npython scripts/run_1L_live_arm.py ... --force\n```\n\nDataset fingerprint: `820bbbf60a0fc930`\n\n. Full report: `experiments/exp_1L_live_arm_report.md`\n\n. Full harness tests: `tests/test_run_1L_live_arm.py`\n\n(29 tests).\n\nSource: [github.com/ManSio](https://github.com/ManSio) · Portfolio: [mansio.github.io/MSPortfolio](https://mansio.github.io/MSPortfolio)", "url": "https://wpnews.pro/news/your-memory-layer-is-lying-to-you-and-your-llm-agrees", "canonical_source": "https://dev.to/mansio/your-memory-layer-is-lying-to-you-and-your-llm-agrees-1oia", "published_at": "2026-08-14 21:35:18+00:00", "updated_at": "2026-08-14 21:55:24.710205+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research"], "entities": ["qwen3.6-flash", "qwen3.7-flash", "claude-sonnet-5", "deepseek-v4-pro", "glm-5.2", "nemotron-3-nano-30b", "qwen3.8-max", "Typesense"], "alternates": {"html": "https://wpnews.pro/news/your-memory-layer-is-lying-to-you-and-your-llm-agrees", "markdown": "https://wpnews.pro/news/your-memory-layer-is-lying-to-you-and-your-llm-agrees.md", "text": "https://wpnews.pro/news/your-memory-layer-is-lying-to-you-and-your-llm-agrees.txt", "jsonld": "https://wpnews.pro/news/your-memory-layer-is-lying-to-you-and-your-llm-agrees.jsonld"}}