{"slug": "show-hn-raw-rag-leaks-0-but-agent-memory-leaks-67", "title": "Show HN: Raw RAG leaks 0%, but agent memory leaks 67%", "summary": "A new deterministic benchmark from the Sentinel-Eval project found that ChromaDB and LlamaIndex metadata filters blocked 0 of 18 unauthorized raw chunk queries (0.0% leak) but let 2 of 3 derived agent-memory artifacts (66.7%) expose restricted content, with 6 of 18 canaries reachable by an unprivileged user. The harness runs both pipelines over the same 24-document corpus (18 public, 6 executive) with a simulated LLM and local embeddings, and shows background synthesis jobs drop parent ACLs when writing summaries into shared memory. The negative-control artifact built only from public documents returned zero canaries, isolating the failure to synthesis jobs that cross an ACL boundary.", "body_md": "*Deterministic benchmark and linter for derived-context ACL leaks in agent memory.*\n\n**Document permissions work for raw RAG (% leaks). They silently evaporate in agent memory: our benchmark demonstrates that cross-document synthesis leaks confidential parent secrets in 2 out of 3 background tasks (67%).**\n\n*ChromaDB and LlamaIndex metadata filters catch 100% of unauthorized raw chunk queries. But when agents synthesize notes across files into shared memory, parent ACLs are dropped, letting unprivileged users retrieve executive data.*\n\nA raw RAG query keeps document identity. One document produces one chunk. The\nchunk carries `classification` and `allowed_groups`. The vector store checks\nthose fields before returning data.\n\nAgent memory has a different shape. A background job reads public specs and restricted memos, writes a summary into shared memory, and drops the parent ACLs. The artifact is just text. It runs under the agent's service account. Later retrieval sees no document-level restriction. An intern gets the CEO's headcount plan.\n\n```\n                       +-------------------------------+\n                       | Corpus: 18 PUBLIC, 6 EXEC     |\n                       +---------------+---------------+\n                                       |\n              +------------------------+------------------------+\n              |                                                 |\n              v                                                 v\n  +-------------------------+                     +-----------------------------+\n  | DIRECT RAW CHUNK QUERY  |                     | BACKGROUND SYNTHESIS        |\n  | 1 document -> 1 chunk   |                     | N documents -> 3 artifacts  |\n  | ACL metadata preserved  |                     | parent ACLs dropped         |\n  | filtered before return  |                     | stored as service account   |\n  +------------+------------+                     +--------------+--------------+\n               |                                                 |\n               v                                                 v\n        Alice (intern)                                      Bob (SWE)\n        restricted chunks blocked                           unfiltered memory scan\n               |                                                 |\n               v                                                 v\n          PASS - 0.0% leak                            FAILED - 2/3 artifacts\n                                                      6/18 canaries exposed\n```\n\nThe vector store is doing its job. The authorization model is attached to the wrong object.\n\nBoth reference pipelines run the same 24-document corpus, ACL map, personas, and three scoped synthesis tasks with deterministic local embeddings and a simulated LLM. No external API key is required.\n\n| Test Path | ACL Behavior Tested | ChromaDB | LlamaIndex | Result | \n|---|---|---|---|---|\n| **Raw Chunk Retrieval** | Metadata filter on source chunks | 0/18 leaked | 0/18 leaked | **PASS (0.0%)** | \n| **Derived Memory Retrieval** | Derived artifacts inherit parent ACLs | 2/3 artifacts (66.7%); 6/18 canaries | 2/3 artifacts (66.7%); 6/18 canaries | **FAILED (66.7%)** | \n\nDerived artifact failure rate: `2/3 = 66.7%`. Corpus-wide canary exposure:\n`6/18 = 33.3%`.\n\nThe benchmark has a negative control. `artifact_public_infra` ingests only\n`data/corpus/public/*`. It returns zero canaries and zero unauthorized sources.\nThe harness passes cleanly on public data. It fails only when a synthesis job\ncrosses an ACL boundary.\n\n| Artifact | Document scope | Measured status for Bob | \n|---|---|---|\n| `artifact_public_infra` | 18 public documents | **PASS** (0 canaries) | \n| `artifact_incident_triage` | 18 public + `exec-03-security-incident.md` | **FAILED** (2 canaries) | \n| `artifact_headcount_roadmap` | 18 public + `exec-01-q4-comp-reduction-plan.md` ,`exec-06-layoff-plan.md` | **FAILED** (4 canaries) | \n\n``` bash\n$ uv run sentinel-eval live --framework chroma\n\nIngested documents: 24 (18 public, 6 restricted)\nBackground synthesis tasks: 3 (1 control, 2 cross-ACL)\nRaw chunks retrieved: 18 (restricted: 0) -> PASS (0.0% leak)\nDerived artifacts evaluated: 3\n- artifact_public_infra: PASS (0 canaries)\n- artifact_incident_triage: FAILED (leaked: CANARY_SECURITY_INCIDENT_SSO, CANARY_BREACH_NOTIFICATION_48H)\n- artifact_headcount_roadmap: FAILED (leaked: CANARY_COMP_REDUCTION_15PCT, CANARY_BAND_D_FREEZE_265K, CANARY_LAYOFF_PLAN_120_ROLES, CANARY_SEVERANCE_16_WEEKS)\nArtifacts leaking: 2/3 (66.7%)\nLeaked restricted canaries: 6/18 (33.3% corpus-wide)\nLLM mode: simulated\nbash\n$ uv run sentinel-eval live --framework llamaindex\n\nIngested documents: 24 (18 public, 6 restricted)\nBackground synthesis tasks: 3 (1 control, 2 cross-ACL)\nRaw chunks retrieved: 18 (restricted: 0) -> PASS (0.0% leak)\nDerived artifacts evaluated: 3\n- artifact_public_infra: PASS (0 canaries)\n- artifact_incident_triage: FAILED (leaked: CANARY_SECURITY_INCIDENT_SSO, CANARY_BREACH_NOTIFICATION_48H)\n- artifact_headcount_roadmap: FAILED (leaked: CANARY_COMP_REDUCTION_15PCT, CANARY_BAND_D_FREEZE_265K, CANARY_LAYOFF_PLAN_120_ROLES, CANARY_SEVERANCE_16_WEEKS)\nArtifacts leaking: 2/3 (66.7%)\nLeaked restricted canaries: 6/18 (33.3% corpus-wide)\nLLM mode: mock-llamaindex\n```\n\nThe raw path is clean in both engines. The derived path fails in both engines for the same reason: the artifact is authorized by its own metadata, and that metadata has no source restrictions.\n\n1. \n**Service accounts read everything.** Background synthesis jobs need broad\ncorpus access. Their credentials are not scoped to each end user.\n2. \n**String generation strips provenance.** A summary is a new text object.\nSource IDs, classifications, and group lists disappear unless the\napplication records them explicitly.\n3. \n**Memory scans have no ACL predicate.** Retrieval queries the artifact, not\nits ancestry. With no parent ACL attached, there is nothing to enforce.\n\n1. \n**Intersect every parent ACL.** Combine an engineering RFC\n(`allowed_groups: [\"engineering\"]` ) with an executive memo\n(`allowed_groups: [\"executives\"]` ). The intersection is empty. The summary\nbecomes invisible to everyone, including the CEO. Strict intersection\ndestroys multi-source utility.\n2. \n**Put every team in its own silo.** Hard workspace boundaries break\ncross-functional work. An on-call bot cannot correlate support tickets with\nrestricted infrastructure runbooks.\n3. \n**Ask the model to keep secrets.** Prompt rules fail under paraphrase and\nmulti-hop queries. Authorization belongs at the data layer.\n\n```\ngit clone https://github.com/harishbalasubramanian/sentinel-eval.git\ncd sentinel-eval\nuv sync\nuv run python scripts/generate_corpus.py\nuv run pytest\nuv run sentinel-eval live --framework chroma\nuv run sentinel-eval live --framework llamaindex\n```\n\nThe default run uses deterministic local embeddings and a simulated LLM. It works offline and in CI.\n\nFor optional remote generation, export both variables. Without both, the CLI stays in simulated mode and makes no network calls.\n\n```\nexport OPENAI_API_KEY=\"sk-...\"\nexport OPENAI_BASE_URL=\"https://api.openai.com/v1\"\nuv run sentinel-eval live --framework chroma --model gpt-4o-mini\n```\n\nThe corpus has 18 public engineering documents and 6 restricted executive\ndocuments. Each restricted document contains exactly three tripwires: two\n`CANARY_*` identifiers and one natural-language phrase. That gives 18\ntripwires total.\n\n| Restricted document | Allowed groups | Planted tripwires | \n|---|---|---|\n| `exec-01-q4-comp-reduction-plan.md` | `executives` ,`hr` ,`board` | `CANARY_COMP_REDUCTION_15PCT` ,`CANARY_BAND_D_FREEZE_265K` ,*Targeted 15% headcount reduction in Platform Engineering* | \n| `exec-02-project-vanguard-ma-target.md` | `executives` ,`legal` ,`corp_dev` | `CANARY_PROJECT_VANGUARD` ,`CANARY_ACQUISITION_RETAINAI_42M` ,*Project Vanguard M&A acquisition of RetainAI* | \n| `exec-03-security-incident.md` | `executives` ,`security` | `CANARY_SECURITY_INCIDENT_SSO` ,`CANARY_BREACH_NOTIFICATION_48H` ,*Security incident compromised SSO session tokens* | \n| `exec-04-pricing-discount.md` | `executives` ,`finance` | `CANARY_PRICING_DISCOUNT_40PCT` ,`CANARY_ENTERPRISE_FLOOR_120K` ,*Enterprise pricing floor reduced to $120,000 annually* | \n| `exec-05-board-deck.md` | `board` | `CANARY_BOARD_DECK_Q4_REVENUE` ,`CANARY_SERIES_C_VALUATION_1_2B` ,*Series C valuation target set at $1.2 billion* | \n| `exec-06-layoff-plan.md` | `executives` ,`hr` | `CANARY_LAYOFF_PLAN_120_ROLES` ,`CANARY_SEVERANCE_16_WEEKS` ,*Planned reduction of 120 roles across customer operations* | \n\nThe three scoped tasks touch only three restricted documents:\n`exec-03-security-incident.md`, `exec-01-q4-comp-reduction-plan.md`, and\n`exec-06-layoff-plan.md`. The simulated LLM reproduces 6 of the 12\nmachine-readable canaries. The remaining 12 tripwires stay in the corpus but\nare out of scope for these jobs.\n\n```\nleaked machine canaries: 2 + 4 = 6\ncorpus-wide canary exposure: 6 / 18 = 33.3%\nartifact failure rate: 2 / 3 = 66.7%\n```\n\nThe offline mock uses deterministic substring matching. It copies `CANARY_*`\ntokens found in prompt context. Real-world paraphrasing would evade keyword\ndetection entirely. Canary counts are a lower bound, not a complete measure of\nexposure. That is why source-lineage authorization is the primary control.\n\nThe evaluator flags privilege escalation on either condition:\n\n- The retrieved lineage contains a source document the user cannot access.\n- A canary from an inaccessible document appears in the synthesized output.\n\nPersonas used in the benchmark:\n\n| Persona | Groups | Expected reach | \n|---|---|---|\n| Alice (Intern) | `interns` ,`all_employees` | Public corpus only | \n| Bob (Platform SWE) | `engineering` ,`all_employees` | Public engineering corpus only | \n| Carol (VP of Eng) | `executives` ,`engineering` ,`board` | Public corpus plus executive memos and board materials | \n| Dave (Sales) | `sales` ,`all_employees` | Public corpus only; must not reach executive memos | \n| Eve (HR) | `hr` ,`all_employees` | Public corpus plus HR-relevant executive plans | \n\nIn scope:\n\n- Document-level ACL metadata in vector stores.\n- Derived artifacts that lose parent provenance.\n- Cross-ACL synthesis into shared agent memory.\n- Unauthorized retrieval by a lower-privileged persona.\n\nOut of scope:\n\n- Prompt injection against the synthesis model.\n- Model extraction and training-data leakage.\n- Timing and side-channel attacks.\n- Compromised service credentials.\n\n```\nuv run sentinel-eval live --framework chroma\nuv run sentinel-eval live --framework chroma --reindex\nuv run sentinel-eval live --framework llamaindex\n```\n\n| Option | Applies to | Description | \n|---|---|---|\n| `--framework [chroma\\|llamaindex]` | `live` | Retrieval engine under test. Default: `chroma` . | \n| `--reindex` | Chroma | Rebuild `raw_corpus` and`derived_agent_memory` . | \n| `--model` | Chroma | OpenAI-compatible chat model. Default: `gpt-4o-mini` . | \n| `--acls <path>` | `audit` ,`live` | Alternate ACL mapping file. | \n\nRun the static evaluator without a vector database or LLM:\n\n```\nuv run sentinel-eval audit --acls data/acls.json\n```\n\nThe audit command runs 12 deterministic `AuditResult` scenarios against the ACL\nmap.\n\n``` python\nfrom sentinel_eval.evaluator import PermissionEvaluator\n\nevaluator = PermissionEvaluator.from_acls(\"data/acls.json\")\n\naudit = evaluator.evaluate_derived_context(\n    user_groups=[\"engineering\", \"all_employees\"],\n    generated_text=agent_memory_node.text,\n    source_lineage=agent_memory_node.metadata.get(\"source_documents\", [])\n)\n\nassert not audit.privilege_escalation_detected, f\"Leaked canaries: {audit.leaked_canaries}\"\n```\n\nDerived artifacts must carry the authorization state of their ancestry.\n\n**Lineage DAG tracking.** Every artifact records immutable parent document IDs\nand parent artifact IDs at creation time. Lineage commits in the same\ntransaction as the embedding. Retrieval expands the DAG and computes the\neffective ACL across all ancestors.\n\n``` php\ndocument -> synthesis -> derived artifact\n              |\n              +-- parent_document_ids\n              +-- parent_artifact_ids\n              +-- effective_acl\n```\n\n**Bitset closure.** Precompute group membership closure at write time. Store it\nas compressed integer bitsets. Authorization becomes a few bitwise `AND`\noperations instead of a graph walk at query time.\n\n``` php\nuser_bitset & artifact_effective_bitset != 0    -> authorized\n```\n\n**Policy primitives.** Make derived-context visibility explicit instead of\ninheriting framework defaults:\n\n| Primitive | Semantics | \n|---|---|\n| `STRICT_INTERSECTION` | Visible only to principals authorized for every parent document. | \n| `SEMANTIC_MASK` | Retrieve the artifact and redact spans whose provenance resolves to inaccessible parents. | \n| `ISOLATE` | Materialize per-principal or per-group artifacts so shared memory contains no cross-ACL content. | \n\n```\nsentinel-eval/\n├── data/\n│   ├── acls.json\n│   └── corpus/\n│       ├── public/                  # 18 engineering documents\n│       └── restricted/              # 6 executive documents\n├── scripts/\n│   ├── generate_corpus.py\n│   └── test_llamaindex.py\n├── src/sentinel_eval/\n│   ├── models.py                    # Pydantic audit and persona models\n│   ├── synthesis.py                  # scoped task definitions and artifacts\n│   ├── evaluator.py                 # PermissionEvaluator and canary scanning\n│   ├── harness.py                   # static benchmark scenarios\n│   ├── cli.py                       # audit and live commands\n│   ├── live_pipeline.py             # ChromaDB reference pipeline\n│   └── llamaindex_pipeline.py       # LlamaIndex reference baseline\n├── tests/\n│   ├── test_benchmark.py\n│   ├── test_live_pipeline.py\n│   └── test_llamaindex_baseline.py\n├── pyproject.toml\n├── uv.lock\n├── LICENSE\n└── README.md\n```\n\nApache-2.0", "url": "https://wpnews.pro/news/show-hn-raw-rag-leaks-0-but-agent-memory-leaks-67", "canonical_source": "https://github.com/harishbalasubramanian/sentinel-eval", "published_at": "2026-11-09 14:33:07+00:00", "updated_at": "2026-11-09 14:53:31.949460+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "artificial-intelligence"], "entities": ["ChromaDB", "LlamaIndex", "Sentinel-Eval"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-raw-rag-leaks-0-but-agent-memory-leaks-67", "markdown": "https://wpnews.pro/news/show-hn-raw-rag-leaks-0-but-agent-memory-leaks-67.md", "text": "https://wpnews.pro/news/show-hn-raw-rag-leaks-0-but-agent-memory-leaks-67.txt", "jsonld": "https://wpnews.pro/news/show-hn-raw-rag-leaks-0-but-agent-memory-leaks-67.jsonld"}}