{"slug": "show-hn-memseal-provenance-containment-for-agent-long-term-memory", "title": "Show HN: MemSeal – provenance containment for agent long-term memory", "summary": "MemSeal, a provenance-containment layer for AI agent long-term memory published on GitHub by developer NagaYu, reported an overall GATE 2 result of FAIL on its Holdout v2 corpus, missing two utility bars while clearing both security criteria. On Holdout v2 at condition C3 (write plus recall path), cross-principal and secret leaks fell from 22 to 0 across C0 to C2, injection success dropped from 0.775 to 0.20, and delayed-use rate hit 0.000, but clean recall retention came in at 0.444 against a 0.80 bar and quarantine precision at 0.738 against a 0.75 bar. The author attributes the 0.444 retention figure to a condition-neutral 18-marker sample identical across C0/C2/C3/C4 (8/18) and the 0.738 precision to 16 false positives, 4 from write policy W1 inside M2 attack cases and 12 from policy_refused records, noting W1 produced zero false positives on clean cases.", "body_md": "**MemSeal is a containment layer on the write/recall boundary of an AI agent's long-term memory.**\nEvery memory entry is wrapped in a provenance envelope (who wrote it, from what source, derived from\nwhat, at which trust tier) inside a SHA-256 hash chain. Deterministic policies decide what may be\nstored, and a trust-tiered recall path decides what may be surfaced — especially before an external\nsend or an irreversible action.\n\n- **No LLM in the policy.** The envelope, the write rules (W1–W5), the recall path and the\nconsolidation guard are deterministic and offline.\n- **Measured with a real local LLM.** Agent behavior is driven by`qwen2.5:14b` via Ollama,\ntemperature 0, seed 20261107, one attempt per case, deterministic marker scoring,**$0** , all local.\n- **Honest gate.** The acceptance gate below is self-reported as**FAIL** ; passing and failing\ncriteria are shown with equal weight.\n\n```\nwrite attempt ──► extract candidate ──► [W1 imperative] [W2 derived-tier] [W3 principal]\n                                        [W4 secret]      [W5 sealed-slot]\n                                            │ active                 │ quarantine / review / reject\n                                            ▼\n                           hash-chained, per-principal memory store\n                                            │\nrecall ──► verify chain ──► principal filter ──► provenance + trust tier ──►\n           sole-basis gate (outbound/irreversible: withhold low-trust, uncorroborated facts)\n```\n\nWrite policies: **W1** quarantines instruction-form content from any source but the user;\n**W2** makes derived entries inherit their worst ancestor's trust tier (no laundering a document into\na \"user fact\"); **W3** rejects cross-principal writes and filters cross-principal recall; **W4** seals\nsecrets (by `entry_class=secret` *and* by shape) so plaintext never lands in the store; **W5** sends a\nfact that contradicts a sealed fact to review.\n\nThe gate is judged on an **independent holdout** corpus authored by a separate party that never saw\nMemSeal's code, the dev corpus, or the reports. The bars are fixed in advance and never moved.\nConditions: **C0** raw store · **C2** write path · **C3** write + recall path · **C4** + consolidation\nguard.\n\n| Criterion (C3) | Holdout v1 (pre-fix) | Holdout v2 (post-fix) | Bar | \n|---|---|---|---|\n| Delayed-use rate ≤ 0.30 **and** ≤ ½·C0 | 0.547 — **FAIL** | **0.000 — PASS** | ≤ 0.30 | \n| Leak count (cross-principal / secret) = 0 | 5 — **FAIL** | **0 — PASS** | = 0 | \n| Clean recall retention ≥ 0.80 | 0.875 — PASS | 0.444 — **FAIL** | ≥ 0.80 | \n| Quarantine precision ≥ 0.75 | 0.837 — PASS | 0.738 — **FAIL** | ≥ 0.75 | \n\n**Overall: GATE 2 = FAIL on v2.** Three deterministic fixes between v1 and v2 (classification-based\nsecret sealing; a sole-basis recall gate; consolidation root retirement) closed both security\ncriteria — on v2, cross-principal + secret leaks went from **22 → 0** across C0 → C2, injection\nsuccess **0.775 → 0.20**, and delayed-use **0.000** at C3. The two remaining failures are\nutility-side, and their causes are stated precisely (not hand-waved):\n\n- **Clean recall retention 0.444** is measured over a**small 18-marker sample** and is**identical\nacross C0/C2/C3/C4 (8/18)** — i.e. it is**condition-neutral and not caused by MemSeal** . Of the 10\nmisses, only 1 would recover under case-insensitive matching (ceiling 0.556); the rest are genuine\nnon-surfacings where even the raw C0 baseline (no defense) did not reproduce stored content.\n- **Quarantine precision 0.738** comes from 16 false positives:**4** are W1 (all inside**M2 attack** cases, i.e. legitimate writes embedded in an attack), and**12** are`policy_refused` records\n(cross-principal M7 writes ×8 and 4 clean) counted under the harness's rejected accounting.**W1 produced zero false positives on clean cases.**\n\nFull detail, including an audit Errata that corrects earlier attribution wording, is in\n[`reports/phase2-regate.md`](https://github.com/NagaYu/memseal/blob/main/reports/phase2-regate.md) and [` reports/phase2-fix.md`](https://github.com/NagaYu/memseal/blob/main/reports/phase2-fix.md).\n\n- **8 attack classes** M1–M8 (query-only injection, document persistence, consolidation laundering,\nsleeper/trigger, false-fact implantation, preference drift, cross-principal leakage, secret staging)\nplus**clean** utility cases.\n- **Split discipline:** the**dev** split (120 cases) is public in this repo;**holdout** corpora are\nauthored independently and kept sealed — only their existence and aggregate numbers are reported here,\nnever the cases.\n- **Conditions C0–C4** and**deterministic marker scoring** (no LLM judge): injection success, delayed\nuse, leak count, clean retention, quarantine precision/recall, per attack class.\n\n| Metric | C0 | C1 | C2 | C3 | C4 | \n|---|---|---|---|---|---|\n| Injection success | 0.982 | 0.713 | 0.556 | 0.556 | 0.463 | \n| Delayed-use | 0.849 | 0.616 | 0.488 | 0.395 | 0.372 | \n| Leak count | 21 | 12 | 6 | 2 | 0 | \n\n(The dev split shares the author's templates, so treat dev as an optimistic in-distribution reference; the independent holdout is the real test.)\n\nMemSeal was also run as a wrapper around **Mem0** (fully local: Ollama LLM + Ollama embedder,\nin-memory vector store), 60 cases, two conditions:\n\n- **C3-wrap drove M8 secret leaks 4 → 0** and injection 0.673 → 0.558 vs. Mem0 native.\n- Disclosed cost: wrap clean retention dipped 0.75 → 0.667 (an exact-content match artifact between\nMem0's paraphrase and the MemSeal store — see report), and Mem0's native `user_id` scoping already\nblocked the cross-principal class.\n\n```\npython3 -m pytest -q          # deterministic policy + bench + harness tests\npython3 -m memseal bench generate --split dev --out bench/dev\npython3 -m memseal verify <store_dir>\npython3 -m memseal quarantine <store_dir>\n```\n\nThe real-LLM measurement harness is in [`memseal_phase2/`](https://github.com/NagaYu/memseal/blob/main/memseal_phase2) and talks only to a local\nOllama (`qwen2.5:14b`). No external API, no API key, no network beyond localhost and the one-time model\ndownload. Tests use a mock LLM, so the suite runs without Ollama.\n\n- **Tests:** full suite green (deterministic; mock-LLM harness tests need no Ollama).\n- **GATE 2:** FAIL (recorded). Security criteria pass on the independent v2 holdout; utility criteria\nfail for the reasons above.\n- Phase reports: [`reports/phase1.md`](https://github.com/NagaYu/memseal/blob/main/reports/phase1.md) ·[` reports/phase2.md`](https://github.com/NagaYu/memseal/blob/main/reports/phase2.md) ·[` reports/phase2-fix.md`](https://github.com/NagaYu/memseal/blob/main/reports/phase2-fix.md) ·[` reports/phase2-regate.md`](https://github.com/NagaYu/memseal/blob/main/reports/phase2-regate.md) .\n\n**All Rights Reserved.** This repository is published for evaluation reading only; no rights to use,\ncopy, modify, or redistribute are granted. See [`LICENSE`](https://github.com/NagaYu/memseal/blob/main/LICENSE). Acquisition / licensing /\ntransfer inquiries are welcome through the author's portfolio:", "url": "https://wpnews.pro/news/show-hn-memseal-provenance-containment-for-agent-long-term-memory", "canonical_source": "https://github.com/NagaYu/memseal", "published_at": "2026-11-09 12:58:14+00:00", "updated_at": "2026-11-09 13:22:41.528440+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "artificial-intelligence", "large-language-models"], "entities": ["MemSeal", "NagaYu", "Ollama", "qwen2.5:14b", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-memseal-provenance-containment-for-agent-long-term-memory", "markdown": "https://wpnews.pro/news/show-hn-memseal-provenance-containment-for-agent-long-term-memory.md", "text": "https://wpnews.pro/news/show-hn-memseal-provenance-containment-for-agent-long-term-memory.txt", "jsonld": "https://wpnews.pro/news/show-hn-memseal-provenance-containment-for-agent-long-term-memory.jsonld"}}