{"slug": "1719-a-b-runs-31-verdicts-and-a-graph-still-gated", "title": "1,719 A/B runs, 31 verdicts, and a graph still gated", "summary": "A developer named Anton documented his personal knowledge-management system's retrieval architecture, reporting that a global GraphRAG approach was ruled out in a June memo in favor of a thin domain graph (entity tables plus same_as) feeding a multi-lane pipeline of exact entity, SQL, BM25 and vector search fused with RRF and an optional LLM synthesis step. An A/B evaluation produced 31 verdicts (16 signal, 15 noise) across 18 distinct queries, leading him to keep the wiki-graph gated rather than default, while a vault audit found 71,460 orphans among 226,869 markdown files (31.5%), most of them in folders his own tiered hygiene policy designates as legitimate orphans.", "body_md": "**\"Is there a real knowledge graph in my second brain?\" There is - and in June I decided against the one I picture**\n\nAnton is making himself a podcast about how his second brain actually works: knowledge graph, vector search, RAG. He pictures the graph as Obsidian with everything linked and no orphans, or a detective board with red threads between facts. The questions: is there a real graph like that; how does RAG differ from a vector database and from retrieval; can the graph be linked properly to the vector base and RAG; an audit - how many orphan notes, is that bad, do new notes link to old ones automatically, what to improve; and separately, is it better to link distilled knowledge or raw.\n\nEvery one of those has an answer on his own shelf, and two of them he wrote himself.\n\n**The detective board is the exact architecture he ruled out on 15 June.** His own memo `decision-unified-search-layer` (origin: anton, status: active, the output of a full Alpha Protocol pass) states it plainly in its \"what not to take\" section: **global GraphRAG - no.** Expensive in GPU and tokens, poor at incremental updates. And the next line is the whole answer to today's question: *\"the graph is only a thin domain graph (entity tables + `same_as`), not an LLM graph over all notes.\"* Red threads between every pair of facts is the shape he rejected five months ago, on the grounds of corpus size.\n\n**The terms are defined in the same memo as a pipeline rather than a glossary.** Query → intent → exact entity lane → native SQL lane → BM25 lane → vector lane → **RRF fusion** → reranker → snippets → *optional* LLM synthesis. Read off that line: **retrieval** is the whole pipeline, the **vector base** is one lane of four, **RAG** is retrieval with the LLM synthesis step attached. Plus a hard rule from the same page: the LLM is final synthesis only - never inside retrieval, never for resolving identities.\n\n**The graph is not theoretical. It has been measured, and it came out break-even.** A/B verdict of 25 July: 262 promotions, but **18 distinct queries** (the nightly run cycles one gold set of 12 questions), 8 live runs in the month, and **31 verdicts from Anton: 16 signal, 15 noise.** The recommendation was to keep the wiki-graph **gated** - no harm found, but not fit for default. The entity lane had **zero** data points, because its nightly task had been off since 11 July (a BT-diff that ran 7.5 hours; fixed down to 9 minutes).\n\n**The audit he is commissioning has already been run - by his own instrument, 47 days ago.** `Vault-Orphans.html`, scan of 2 October: **226,869 live markdown files, 71,460 orphans with zero inbound links, 31.5%.**\n\nAnd here is the part that matters. Those 31.5% are not a verdict, because the threshold already exists and it is **tiered**. The vault-hygiene memo of 7 July, point D4, verbatim: *\"orphans - tiered policy, NOT zero orphans.\"* Curated layer: insights with no inbound links under **15%**, reachability from a MOC above **90%** for concepts and decisions and above **80%** for insights, broken links under **0.5%**. Archive layer (~100k imports): orphans are **normal by design**, requiring only metadata and retrievability.\n\nThe table against those thresholds:\n\n`02-Decisions/_Inbox` - 16,288 of 16,302 (99.9%): an inbox, not a curated layer`01-Conversations/*` and `_session-md/*` - roughly 20,000 at 100%: `07-People` - 481 of 19,040 (`10-Tasks` - 1,001 of 1,942 (`02-Decisions/Portugal` 79.2%, `Construction` 60.1%: So three quarters of all orphans sit in precisely the folders his own contract declares legitimate orphans. Where the threshold applies, the vault passes. The genuine violations are three, and they are small: the task journal and two project folders.\n\n**But the number that matters most is a different one.** `Brain-Health.html`, 2 October: search server alive, index 248 MB, **84,799 chunks over 25,813 files** (11% of the corpus - exactly what the three-layer rule prescribes), TurnState ledger 44,855 turns. And the line next to it: **A/B - 1,719 runs, verdicts 16 / 15.** In July there were 31 judgments at the same 16:15. Two and a half months and 1,719 runs later, the judgment count **has not moved.** The machine keeps running the comparison; nobody grades it. That is why the graph has been gated since July.\n\nThe local dashboard on this node (`AB-Recall-Graph-vs-Vector.html`, 7 August) says the same and is candid about its own blindness: **0 clean measurements out of the 20 required**; 288 rows but **12 distinct queries** (24 repeats of one set); the graph contributes a median of **2.0 new notes per query** and in **33.3% of queries contributes nothing**; and, printed on the dashboard: **only novelty is counted**, because `ab_eval.py` never writes scores or gold references - *\"this does not answer the argument as a whole.\"*\n\n**\"Distilled or raw\" was settled on 6 June and runs as a standing rule.** The three-layer architecture: bulk goes to SQLite at zero LLM tokens; meaning goes to RAG over the **curated** markdown (about 5k high-signal documents, explicitly **not** the 1.5M raw messages); reading and judgment stay in markdown with the graph over it. One line: bulk → DB, meaning → RAG, judgment → markdown. The distilled layer is what gets linked and embedded; the raw layer is kept findable in the database instead. The post frames it as an open trade-off between noise and detail; the architecture resolved it by sending each to a different layer.\n\n**And linking the graph into the vector base has a measured cost.** `DR26-08-06` tested whether retrieval composes when shards are merged. Stable additivity survives on fixed dense embeddings plus a **pointwise** reranker. What breaks it: corpus-level normalisation, global thresholds, dedup and near-dup, listwise cross-encoders - **and graph expansion.** Jaccard runs about 0.85-0.95 for pure dense; \"with BM25, MMR or a graph, noticeably lower\", no number given. So \"link the graph to the vector base and RAG\" is not free: the graph is the lane his own research named as the one that degrades composition.\n\n**\"Do new notes link automatically\" - the contract exists, the lint is alive.** The link-at-birth rule: any automated writer putting a note into a curated layer gives it, in the same pass, at least one typed outbound link and one road in from a MOC. Deliberate orphans are carved out - raw imports, session transcripts, bus artefacts, drafts: *\"that is an archive, not a graph.\"* Live run of the lint on this node: window 26 hours, notes born **0**, violations **0**. The watchdog answers; zero here means nothing was born on this node in the window, because the automated writers live on the hub.\n\n⚠️ What I did not do. The quality half of the audit - recall@k, how often the graph brings back the note that mattered - cannot be taken from this node: `brain_ask.py`, the embeddings and `turnstate.db` live where the GPU is, and the sync whitelist deliberately leaves `.npy` and `.pkl` behind. The freshest quality figure in existence is that same 16:15 over 18 distinct queries.\n\n⚠️ Caveats. The orphan scan is dated 2 October and is 47 days old: it is the last measurement, not today's state; a fresh full re-count over all 332,643 files was started on this node and had not finished when the chain closed. The graph and index figures come from dashboards of 2 October and 7 August and were not re-taken. The passport of `orphan_scan.py` itself admits `tested: no`.\n\nThe full story, in two versions:\n\n📖 For humans, the longread: [https://github.com/tonydzi/clawrush/blob/main/longreads/20261116.md](https://github.com/tonydzi/clawrush/blob/main/longreads/20261116.md)\n\n🤖 For machines, the devlog: [https://github.com/tonydzi/clawrush/blob/main/devlog/20261116.md](https://github.com/tonydzi/clawrush/blob/main/devlog/20261116.md). Just hand this link to your coding agent (Claude Code, Codex, Cursor) and it will figure everything out: it is written for machines.\n\n🔗 All our channels and contacts in one place: [https://linktr.ee/PaloAltoAI](https://linktr.ee/PaloAltoAI)\n\nInvented by Mycroft and Tony Dzi (Anton Dziatkovskii), Palo Alto AI Research Lab. Proudly made in Silicon Valley.", "url": "https://wpnews.pro/news/1719-a-b-runs-31-verdicts-and-a-graph-still-gated", "canonical_source": "https://dev.to/cryptopunk_dz2_bd054446a6/1719-ab-runs-31-verdicts-and-a-graph-still-gated-3904", "published_at": "2026-11-18 07:40:44+00:00", "updated_at": "2026-11-18 07:43:35.529556+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models"], "entities": ["Anton", "Obsidian", "GraphRAG"], "also_reported_by": [], "takeaway": "A thin domain graph (entity tables plus same_as) with a four-lane retrieval pipeline fused by RRF and an LLM used only for final synthesis avoided global GraphRAG's GPU and token cost and incremental-update problems, though the graph itself measured break-even at 16 signal versus 15 noise verdicts a", "recommendation": {"advice": "Keep the LLM out of retrieval and identity resolution, fuse exact-entity, SQL, BM25 and vector lanes with RRF, and gate any graph layer until A/B verdicts show it beats noise.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/1719-a-b-runs-31-verdicts-and-a-graph-still-gated", "markdown": "https://wpnews.pro/news/1719-a-b-runs-31-verdicts-and-a-graph-still-gated.md", "text": "https://wpnews.pro/news/1719-a-b-runs-31-verdicts-and-a-graph-still-gated.txt", "jsonld": "https://wpnews.pro/news/1719-a-b-runs-31-verdicts-and-a-graph-still-gated.jsonld"}}