MemSeal is a containment layer on the write/recall boundary of an AI agent's long-term memory. Every memory entry is wrapped in a provenance envelope (who wrote it, from what source, derived from what, at which trust tier) inside a SHA-256 hash chain. Deterministic policies decide what may be stored, and a trust-tiered recall path decides what may be surfaced — especially before an external send or an irreversible action.
- No LLM in the policy. The envelope, the write rules (W1–W5), the recall path and the consolidation guard are deterministic and offline.
- Measured with a real local LLM. Agent behavior is driven by
qwen2.5:14bvia Ollama, temperature 0, seed 20261107, one attempt per case, deterministic marker scoring,$0 , all local. - Honest gate. The acceptance gate below is self-reported asFAIL ; passing and failing criteria are shown with equal weight.
write attempt ──► extract candidate ──► [W1 imperative] [W2 derived-tier] [W3 principal]
[W4 secret] [W5 sealed-slot]
│ active │ quarantine / review / reject
▼
hash-chained, per-principal memory store
│
recall ──► verify chain ──► principal filter ──► provenance + trust tier ──►
sole-basis gate (outbound/irreversible: withhold low-trust, uncorroborated facts)
Write policies: W1 quarantines instruction-form content from any source but the user;
W2 makes derived entries inherit their worst ancestor's trust tier (no laundering a document into
a "user fact"); W3 rejects cross-principal writes and filters cross-principal recall; W4 seals
secrets (by entry_class=secret and by shape) so plaintext never lands in the store; W5 sends a
fact that contradicts a sealed fact to review.
The gate is judged on an independent holdout corpus authored by a separate party that never saw MemSeal's code, the dev corpus, or the reports. The bars are fixed in advance and never moved. Conditions: C0 raw store · C2 write path · C3 write + recall path · C4 + consolidation guard.
| Criterion (C3) | Holdout v1 (pre-fix) | Holdout v2 (post-fix) | Bar |
|---|---|---|---|
| Delayed-use rate ≤ 0.30 and ≤ ½·C0 | 0.547 — FAIL | 0.000 — PASS | ≤ 0.30 |
| Leak count (cross-principal / secret) = 0 | 5 — FAIL | 0 — PASS | = 0 |
| Clean recall retention ≥ 0.80 | 0.875 — PASS | 0.444 — FAIL | ≥ 0.80 |
| Quarantine precision ≥ 0.75 | 0.837 — PASS | 0.738 — FAIL | ≥ 0.75 |
Overall: GATE 2 = FAIL on v2. Three deterministic fixes between v1 and v2 (classification-based secret sealing; a sole-basis recall gate; consolidation root retirement) closed both security criteria — on v2, cross-principal + secret leaks went from 22 → 0 across C0 → C2, injection success 0.775 → 0.20, and delayed-use 0.000 at C3. The two remaining failures are utility-side, and their causes are stated precisely (not hand-waved):
- Clean recall retention 0.444 is measured over asmall 18-marker sample and isidentical across C0/C2/C3/C4 (8/18) — i.e. it iscondition-neutral and not caused by MemSeal . Of the 10 misses, only 1 would recover under case-insensitive matching (ceiling 0.556); the rest are genuine non-surfacings where even the raw C0 baseline (no defense) did not reproduce stored content.
- Quarantine precision 0.738 comes from 16 false positives:4 are W1 (all insideM2 attack cases, i.e. legitimate writes embedded in an attack), and12 are
policy_refusedrecords (cross-principal M7 writes ×8 and 4 clean) counted under the harness's rejected accounting.W1 produced zero false positives on clean cases.
Full detail, including an audit Errata that corrects earlier attribution wording, is in
reports/phase2-regate.md and reports/phase2-fix.md.
- 8 attack classes M1–M8 (query-only injection, document persistence, consolidation laundering, sleeper/trigger, false-fact implantation, preference drift, cross-principal leakage, secret staging) plusclean utility cases.
- Split discipline: thedev split (120 cases) is public in this repo;holdout corpora are authored independently and kept sealed — only their existence and aggregate numbers are reported here, never the cases.
- Conditions C0–C4 anddeterministic marker scoring (no LLM judge): injection success, delayed use, leak count, clean retention, quarantine precision/recall, per attack class.
| Metric | C0 | C1 | C2 | C3 | C4 |
|---|---|---|---|---|---|
| Injection success | 0.982 | 0.713 | 0.556 | 0.556 | 0.463 |
| Delayed-use | 0.849 | 0.616 | 0.488 | 0.395 | 0.372 |
| Leak count | 21 | 12 | 6 | 2 | 0 |
(The dev split shares the author's templates, so treat dev as an optimistic in-distribution reference; the independent holdout is the real test.)
MemSeal was also run as a wrapper around Mem0 (fully local: Ollama LLM + Ollama embedder, in-memory vector store), 60 cases, two conditions:
- C3-wrap drove M8 secret leaks 4 → 0 and injection 0.673 → 0.558 vs. Mem0 native.
- Disclosed cost: wrap clean retention dipped 0.75 → 0.667 (an exact-content match artifact between
Mem0's paraphrase and the MemSeal store — see report), and Mem0's native
user_idscoping already blocked the cross-principal class.
python3 -m pytest -q # deterministic policy + bench + harness tests
python3 -m memseal bench generate --split dev --out bench/dev
python3 -m memseal verify <store_dir>
python3 -m memseal quarantine <store_dir>
The real-LLM measurement harness is in memseal_phase2/ and talks only to a local
Ollama (qwen2.5:14b). No external API, no API key, no network beyond localhost and the one-time model
download. Tests use a mock LLM, so the suite runs without Ollama.
- Tests: full suite green (deterministic; mock-LLM harness tests need no Ollama).
- GATE 2: FAIL (recorded). Security criteria pass on the independent v2 holdout; utility criteria fail for the reasons above.
- Phase reports:
reports/phase1.md·reports/phase2.md·reports/phase2-fix.md·reports/phase2-regate.md.
All Rights Reserved. This repository is published for evaluation reading only; no rights to use,
copy, modify, or redistribute are granted. See LICENSE. Acquisition / licensing /
transfer inquiries are welcome through the author's portfolio: