{"slug": "a-folder-of-docs-is-not-a-knowledge-base", "title": "A Folder of Docs Is Not a Knowledge Base", "summary": "A developer building agentproto corpus tooling argues that a folder of documents with RAG is not a knowledge base because it retrieves prose instead of facts and lacks provenance. They propose a pipeline that turns sources into atomic, sourced claims before querying, enabling agents to cite and users to audit. The developer warns that RAG over a docs folder creates a confident-but-unverifiable setup that is dangerous when stale facts are stated.", "body_md": "Disclosure up front: I build\n\n[agentproto]and its corpus\n\ntooling, which is what the walkthrough uses. The commands are real and\n\ncheckable; the problem in the first half is one every RAG setup hits.\n\nCorrections welcome — file an issue.\n\nThe advice is everywhere and it sounds complete: *give your agent your knowledge —\njust do RAG over your docs folder.* So you point a retriever at\n\n`docs/`\n\n, embed theThen you ask it something that matters, and it gives you a fluent, confident\n\nparagraph — and you have no idea whether it's true, because you have no idea where\n\nit came from. Was that from the current runbook or a design doc you abandoned in\n\n2024? Is that retry number the real one, or a chunk from a stale README that\n\nhappened to score well on cosine similarity? You can't tell. Neither can the\n\nagent.\n\nThat's the tell that you don't have a knowledge base. You have a search index over\n\na pile of documents, which is a different and much weaker thing.\n\nThe one idea, if you remember nothing else:\n\nA folder of docs is not a knowledge base. What makes it one isprovenance—\n\nevery fact knowing which source it came from, so the agent can cite it and you\n\ncan check it.\n\nA folder of markdown fails as a knowledge base for two structural reasons, and no\n\namount of better embeddings fixes either.\n\n**It retrieves prose, not facts.** A document is a wall of sentences about many\n\nthings at once. Chunk it and you get arbitrary slices — half a paragraph about\n\nretries glued to the start of a paragraph about logging. The agent gets *near* the\n\nanswer and reconstructs the rest, which is exactly where confident-but-wrong comes\n\nfrom. The unit of knowledge should be a *claim*, not a 512-token window.\n\n**It forgets where everything came from.** Once a chunk is embedded, it's just a\n\nfloating string. There's no thread back to \"this is from `runbooks/payments.md`\n\n,\n\nlast touched in the April incident.\" So the agent can't cite, and — worse — *you*\n\ncan't audit. A knowledge base you can't audit is a knowledge base you can't trust,\n\nwhich means it's a liability the first time it's confidently wrong.\n\nBoth problems have the same fix, and it's not a fancier retriever. It's a\n\n**pipeline**: turn sources into atomic, sourced claims *before* anything queries\n\nthem. That pipeline is what separates a corpus from a folder.\n\nPlace yourself before you build. Four honest answers.\n\n**Nothing.** The agent runs on pre-training and whatever you paste. Generic by\n\ndefault. *Ceiling: it never knows one thing that isn't already public.*\n\n**A docs folder + RAG.** You retrieve chunks by similarity. Better than nothing,\n\nand it's the trap this piece is about: no provenance, no dedup, no notion of a\n\nclaim — it answers from sources it can't name. *Ceiling: confident, unverifiable,\nand stale the day a doc rots.*\n\n**A hand-curated wiki.** Real, human-maintained, often good — but it's written for\n\n*people* to read, not for an agent to query claim-by-claim, and it goes stale\n\nbetween the edits nobody makes. *Ceiling: high quality, low queryability, manual\nupkeep.*\n\n**A distilled, sourced corpus.** Sources imported with their origin attached,\n\ndistilled to claim-level entries, each carrying its provenance, queryable by topic.\n\n*This is the one an agent can cite and you can audit — and it's four commands.*\n\nWhere are you?If your honest answer is\"RAG over a docs folder,\"you have\n\nthe confident-but-unverifiable setup — the most dangerous kind, because itfeels\n\nlike it works right up until it states a stale fact with a straight face. The\n\nrest of this turns the pile into a corpus.\n\nStart by pulling your sources into a corpus workspace — and the non-negotiable is\n\nthat each source keeps a thread back to where it came from.\n\n```\nnpm i -g @agentproto/corpus-cli\n\n# scaffold an AIP-10 workspace (generic research preset, no seed noise)\ncorpus init house-knowledge knowledge/house --preset research\n\n# import your sources; --tags is what makes the corpus queryable by topic later\ncorpus import-web knowledge/house --urls-file runbooks.txt --tags runbooks\ncorpus import-web knowledge/house --urls-file incidents.txt --tags incidents\n```\n\nEvery imported source lands as an AIP-10 record carrying its `originalUrl`\n\n— the\n\nprovenance thread. An article comes in via readability; a walled page routes\n\nthrough a browser; **a recorded incident review or an architecture talk on video\ncomes in via transcription** — same workspace, same provenance, so a decision your\n\nThis is the step a folder-and-RAG setup skips entirely, and it's the one that\n\nmatters. Distillation reads each source and emits **atomic, claim-level entries**,\n\nroughly seven per source, each one carrying the source it came from.\n\n```\n# cheap + strong: distill on an open model via OpenRouter (saves frontier credits)\ncorpus distill knowledge/house --engine opencode -m openrouter/deepseek/deepseek-v4-pro\n```\n\nNow the unit of knowledge is a claim, not a chunk — *\"payment retries are capped at\n3 with an idempotency key, per incident 2026-04\"* is one entry with one source, not\n\nWhy claims beat chunks, grounded.Anthropic's context-engineering guidance is\n\nto keep the working context small and retrieve high-signal factsjust in time\n\nrather than pre-loading walls of text. You can't do just-in-time retrieval on a\n\ndocument — it's too big and says too much. You can do it on a claim. Distillation\n\nis what makes your knowledgeretrievable at the right grain.\n\nA pile of docs is fill-once-and-forget. A corpus is an artifact you *keep*, so it\n\ngets a quality gate before anything trusts it.\n\n```\ncorpus validate knowledge/house    # schema-valid sources, 0 errors\ncorpus lint knowledge/house        # structural checks\ncorpus knowledge knowledge/house --tags incidents --max 20   # preview what an agent would pull\n```\n\nTwo failure modes to catch here, because they're the ones that poison a corpus\n\nquietly. A **bad scrape** — a retriever grabbing a cookie-wall or a nav menu\n\ninstead of the article — still gets distilled into confident-sounding entries;\n\ngrep for self-flags (*\"content does not match\"*, off-topic nouns) and quarantine\n\nthem into a `demoted/`\n\nfolder so they never reach an agent. And **coverage**: for\n\neach topic you care about, does the corpus actually have sourced claims, or is that\n\ntag empty? A gap you find now is a gap the agent won't paper over with a guess\n\nlater.\n\nThis is the discipline the doc-dump never had: **your knowledge base is only as\ngood as its worst source, so you check the sources.**\n\nNow the corpus becomes a tool. Back it with a `knowledge.search`\n\ndriver pointed at\n\nthe workspace you just built, and serve it over MCP — the full contract-and-driver\n\npattern is [its own walkthrough](https://dev.to/agentiknet/your-coding-agent-knows-the-internets-average-heres-how-to-make-it-know-yours-1aeg), but the payoff is one\n\nenvironment variable and one command:\n\n```\nCORPUS_WS=knowledge/house agentproto serve   # serves knowledge.search to every agent\n```\n\nEvery agent that mounts the daemon — Claude Code, Codex, a cheap local model — now\n\nqueries the *same* distilled, sourced corpus, and because the driver reads local\n\nfiles, **your knowledge never leaves the machine.** Ask any of them your retry\n\npolicy and the answer comes back with `incident 2026-04`\n\nattached. That citation\n\nisn't decoration — it's the thing that lets a skeptical reviewer (human or agent)\n\ncheck the answer against the source instead of trusting it.\n\nA corpus is a real artifact, so it has real upkeep and real failure modes.\n\n**A wrong corpus is worse than no corpus.** A stale claim gets retrieved with the\n\nsame confidence as a fresh one — now *with a source attached*, which makes it more\n\npersuasive and more dangerous. Provenance is what saves you: because every claim\n\nnames its origin, a wrong answer is *traceable* to the stale source and fixable,\n\ninstead of an anonymous hallucination you can't hunt down. You maintain a corpus\n\nlike you maintain docs, because that's what it is.\n\n**Don't let the agent write your knowledge base.** A corpus distilled from real\n\nrunbooks, incidents, and decisions is yours; one an agent generates about itself is\n\nthe internet's average with your repo's name on it. Distill from sources you'd\n\nstand behind — the whole point is that the knowledge is *earned*, not synthesized.\n\n**And the fair comparison.** Managed RAG products and vendor retrieval are less\n\nsetup — if you want \"upload your docs, get a chatbot,\" they'll get you there faster,\n\nand for some teams that's the right trade. What you give up is the two things this\n\nwhole piece is about: claim-level provenance you can audit, and a knowledge base\n\nthat lives in your repo, on your machine, not in someone's index. If your knowledge\n\nis your edge, those aren't details.\n\nEveryone rents the same models now, so the one part of the stack that's genuinely\n\nyours is what your agent *knows* — and that's only true if what it knows is real.\n\nA folder of docs behind a retriever gives you fluent answers from sources you can't\n\nname and can't check, which is the exact opposite of what a knowledge base is for.\n\nThe fix isn't a better embedding model. It's a pipeline: import your sources with\n\ntheir provenance intact, distill them into claims small enough to retrieve and\n\nhonest enough to cite, gate the result like the artifact it is, and serve it to\n\nevery agent you run. Four commands, and your agent stops guessing from a pile and\n\nstarts answering from a corpus — one where every fact can tell you exactly where it\n\ncame from.\n\nGive your agent knowledge. Just make sure it's the kind that can show its work.\n\nIf you build your knowledge base a way that keeps provenance without the distill\n\nstep — or you've made folder-RAG genuinely auditable — tell me where. I'll fix the\n\npiece.\n\n*Written by the maintainer of agentproto (Apache-2.0,\nsource) — a local, cross-vendor daemon for\nrunning and governing coding agents. Same contract as our\n/compare page: dated facts, named strengths,\ncorrections by issue. Got something wrong?\nFile an issue.*\n\n*Building agentproto in the open — follow @theagentproto and @agentik_ai on X.*", "url": "https://wpnews.pro/news/a-folder-of-docs-is-not-a-knowledge-base", "canonical_source": "https://dev.to/agentiknet/a-folder-of-docs-is-not-a-knowledge-base-55b9", "published_at": "2026-07-14 01:36:49+00:00", "updated_at": "2026-07-14 01:56:52.812954+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools", "developer-tools"], "entities": ["agentproto"], "alternates": {"html": "https://wpnews.pro/news/a-folder-of-docs-is-not-a-knowledge-base", "markdown": "https://wpnews.pro/news/a-folder-of-docs-is-not-a-knowledge-base.md", "text": "https://wpnews.pro/news/a-folder-of-docs-is-not-a-knowledge-base.txt", "jsonld": "https://wpnews.pro/news/a-folder-of-docs-is-not-a-knowledge-base.jsonld"}}