{"slug": "the-graph-is-not-the-trust-layer", "title": "The graph is not the trust layer", "summary": "A review of Blake McCarn's Paperless Knowledge Graph finds that while it enables chat with scanned documents and includes evidence-backed retrieval with strict refusal modes, the project lacks public fixtures to substantiate its reported accuracy for medical, tax, financial, and legal queries. The repository also has unpinned dependencies and no LICENSE file despite a README stating MIT, limiting its reusability. Related projects like Portable Brain and Herdr were also assessed, with Portable Brain's test suite passing locally.", "body_md": "Disclosure: Software Sausage is our product. Blake McCarn did not sponsor, review, or endorse this article. AI tools helped draft and edit it; the evidence boundaries are stated below.\n\nBlake McCarn's [Paperless Knowledge Graph](https://blakemccarn.dev/work/paperless-knowledge-graph) is not interesting merely because it lets someone chat with scanned documents. The stronger idea is that retrieval leaves evidence behind and can refuse to answer when that evidence is weak.\n\nThis is a source review, not a field test. We reviewed McCarn's case study, inspected the current [public repository](https://github.com/bmccarn/paperless-knowledge-graph), and checked related projects. We did not connect the stack to his private archive or reproduce its reported accuracy, scale, speed, or cost.\n\n```\nPaperless source documents\n  → baseline OCR → selective enhanced OCR\n  → classify → extract → verify → relate\n  → graph + vector + keyword indexes\n  → quick | deep | timeline | strict query\n  → cited answer + claim ledger + trace\n```\n\nMcCarn reports more than 800 documents, roughly 7,000 graph nodes, 25,000 relationships, and 6,000 searchable chunks. Those are author-reported results, not Software Sausage measurements.\n\nPaperless remains the document authority. Enhanced OCR runs as a companion layer, so the archive is not held hostage by the AI pipeline.\n\nThe freshness check compares exact document ID sets and hashes across Paperless, Neo4j, and vector chunks. Equal counts can still conceal one missing document and one stale replacement.\n\nEntity lookups use the graph. Deeper questions combine vector, keyword, and graph retrieval. Timeline mode keeps dates explicit. Strict mode can refuse weak evidence instead of filling the gap with confident prose.\n\nThe repository includes evidence-pack, claim-ledger, trust-dimension, trace, and answer-repair paths. A separate model pass may catch inconsistencies, but it is not independent ground truth. The verifier can share the drafter's blind spots.\n\nLiteLLM centralizes model aliases, credentials, limits, and cost visibility. That reduces provider coupling; it does not eliminate the need to test authentication and response behavior end to end.\n\nThe source contains the advertised evidence helpers, exact-drift audit, API smoke checks, and an evaluation harness for fixed questions. Its published container workflow is green, but it builds images rather than gating publication on those checks.\n\nThe Python dependency file is mostly unpinned. The README says MIT, while the repository had no LICENSE file and GitHub detected no license when we reviewed it. Until that is corrected, treat the code as publicly readable rather than reusable MIT material.\n\nMost importantly, no public fixture establishes the accuracy implied by its medical, tax, financial, and legal query modes. Strict refusal and visible citations are useful controls. They do not replace opening the source.\n\n[Portable Brain](https://github.com/bmccarn/portable-brain) applies the portability boundary to knowledge work: plain Markdown, YAML, and relative links remain useful without a runtime, while source evidence stays separate from synthesized notes. We ran its current test suite locally: 75 tests passed. A full-repository lint also found one executable-bit issue in an optional Paperless integration outside the project's narrower CI lint scope.\n\nMcCarn's [Herdr write-up](https://blakemccarn.dev/blog/managing-ai-coding-agents-with-herdr) separates terminal topology from provider logic: one worktree per implementation, distinct review and test surfaces, and status labels used for routing rather than proof. His [LiteLLM operating note](https://blakemccarn.dev/blog/self-hosting-litellm-proxy) makes a similar boundary explicit for model traffic, budgets, and credentials.\n\nBuild a synthetic, non-sensitive Paperless archive with ordinary text, tables, handwriting, duplicate entities, contradictory dates, and one deliberately stale index. Freeze a dozen questions and score source coverage, exact-value accuracy, unsupported claims, refusal behavior, elapsed time, and model cost.\n\nThen remove one document, add another so the total count stays equal, and require the exact freshness check to catch the swap.\n\nThat would turn a strong architecture story into a reproducible recipe. Until then, the honest label is community source review.\n\nRead the complete review and follow the proposed verification run at [Software Sausage](https://softwaresausage.com/blog/paperless-knowledge-graph-evidence-pipeline?source=community&utm_campaign=dev_paperless_kg).", "url": "https://wpnews.pro/news/the-graph-is-not-the-trust-layer", "canonical_source": "https://dev.to/softwaresausage/the-graph-is-not-the-trust-layer-4hd4", "published_at": "2026-09-03 23:08:33+00:00", "updated_at": "2026-09-03 23:53:50.563935+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools"], "entities": ["Blake McCarn", "Paperless Knowledge Graph", "Portable Brain", "Herdr", "LiteLLM", "Neo4j", "Software Sausage"], "alternates": {"html": "https://wpnews.pro/news/the-graph-is-not-the-trust-layer", "markdown": "https://wpnews.pro/news/the-graph-is-not-the-trust-layer.md", "text": "https://wpnews.pro/news/the-graph-is-not-the-trust-layer.txt", "jsonld": "https://wpnews.pro/news/the-graph-is-not-the-trust-layer.jsonld"}}