Disclosure up front: I build
[agentproto]and its corpus
tooling, which is what the walkthrough uses. The commands are real and
checkable; the problem in the first half is one every RAG setup hits.
Corrections welcome — file an issue.
The advice is everywhere and it sounds complete: give your agent your knowledge — just do RAG over your docs folder. So you point a retriever at
docs/
, embed theThen you ask it something that matters, and it gives you a fluent, confident
paragraph — and you have no idea whether it's true, because you have no idea where
it came from. Was that from the current runbook or a design doc you abandoned in
2024? Is that retry number the real one, or a chunk from a stale README that
happened to score well on cosine similarity? You can't tell. Neither can the
agent.
That's the tell that you don't have a knowledge base. You have a search index over
a pile of documents, which is a different and much weaker thing.
The one idea, if you remember nothing else:
A folder of docs is not a knowledge base. What makes it one isprovenance—
every fact knowing which source it came from, so the agent can cite it and you
can check it.
A folder of markdown fails as a knowledge base for two structural reasons, and no
amount of better embeddings fixes either.
It retrieves prose, not facts. A document is a wall of sentences about many
things at once. Chunk it and you get arbitrary slices — half a paragraph about
retries glued to the start of a paragraph about logging. The agent gets near the
answer and reconstructs the rest, which is exactly where confident-but-wrong comes
from. The unit of knowledge should be a claim, not a 512-token window.
It forgets where everything came from. Once a chunk is embedded, it's just a
floating string. There's no thread back to "this is from runbooks/payments.md
,
last touched in the April incident." So the agent can't cite, and — worse — you
can't audit. A knowledge base you can't audit is a knowledge base you can't trust,
which means it's a liability the first time it's confidently wrong.
Both problems have the same fix, and it's not a fancier retriever. It's a
pipeline: turn sources into atomic, sourced claims before anything queries
them. That pipeline is what separates a corpus from a folder.
Place yourself before you build. Four honest answers.
Nothing. The agent runs on pre-training and whatever you paste. Generic by
default. Ceiling: it never knows one thing that isn't already public.
A docs folder + RAG. You retrieve chunks by similarity. Better than nothing,
and it's the trap this piece is about: no provenance, no dedup, no notion of a
claim — it answers from sources it can't name. Ceiling: confident, unverifiable, and stale the day a doc rots.
A hand-curated wiki. Real, human-maintained, often good — but it's written for
people to read, not for an agent to query claim-by-claim, and it goes stale
between the edits nobody makes. Ceiling: high quality, low queryability, manual upkeep.
A distilled, sourced corpus. Sources imported with their origin attached,
distilled to claim-level entries, each carrying its provenance, queryable by topic.
This is the one an agent can cite and you can audit — and it's four commands.
Where are you?If your honest answer is"RAG over a docs folder,"you have
the confident-but-unverifiable setup — the most dangerous kind, because itfeels
like it works right up until it states a stale fact with a straight face. The
rest of this turns the pile into a corpus.
Start by pulling your sources into a corpus workspace — and the non-negotiable is
that each source keeps a thread back to where it came from.
npm i -g @agentproto/corpus-cli
corpus init house-knowledge knowledge/house --preset research
corpus import-web knowledge/house --urls-file runbooks.txt --tags runbooks
corpus import-web knowledge/house --urls-file incidents.txt --tags incidents
Every imported source lands as an AIP-10 record carrying its originalUrl
— the
provenance thread. An article comes in via readability; a walled page routes
through a browser; a recorded incident review or an architecture talk on video comes in via transcription — same workspace, same provenance, so a decision your
This is the step a folder-and-RAG setup skips entirely, and it's the one that
matters. Distillation reads each source and emits atomic, claim-level entries,
roughly seven per source, each one carrying the source it came from.
corpus distill knowledge/house --engine opencode -m openrouter/deepseek/deepseek-v4-pro
Now the unit of knowledge is a claim, not a chunk — "payment retries are capped at 3 with an idempotency key, per incident 2026-04" is one entry with one source, not
Why claims beat chunks, grounded.Anthropic's context-engineering guidance is
to keep the working context small and retrieve high-signal factsjust in time
rather than pre- walls of text. You can't do just-in-time retrieval on a
document — it's too big and says too much. You can do it on a claim. Distillation
is what makes your knowledgeretrievable at the right grain.
A pile of docs is fill-once-and-forget. A corpus is an artifact you keep, so it
gets a quality gate before anything trusts it.
corpus validate knowledge/house # schema-valid sources, 0 errors
corpus lint knowledge/house # structural checks
corpus knowledge knowledge/house --tags incidents --max 20 # preview what an agent would pull
Two failure modes to catch here, because they're the ones that poison a corpus
quietly. A bad scrape — a retriever grabbing a cookie-wall or a nav menu
instead of the article — still gets distilled into confident-sounding entries;
grep for self-flags ("content does not match", off-topic nouns) and quarantine
them into a demoted/
folder so they never reach an agent. And coverage: for
each topic you care about, does the corpus actually have sourced claims, or is that
tag empty? A gap you find now is a gap the agent won't paper over with a guess
later.
This is the discipline the doc-dump never had: your knowledge base is only as good as its worst source, so you check the sources.
Now the corpus becomes a tool. Back it with a knowledge.search
driver pointed at
the workspace you just built, and serve it over MCP — the full contract-and-driver
pattern is its own walkthrough, but the payoff is one
environment variable and one command:
CORPUS_WS=knowledge/house agentproto serve # serves knowledge.search to every agent
Every agent that mounts the daemon — Claude Code, Codex, a cheap local model — now
queries the same distilled, sourced corpus, and because the driver reads local
files, your knowledge never leaves the machine. Ask any of them your retry
policy and the answer comes back with incident 2026-04
attached. That citation
isn't decoration — it's the thing that lets a skeptical reviewer (human or agent)
check the answer against the source instead of trusting it.
A corpus is a real artifact, so it has real upkeep and real failure modes.
A wrong corpus is worse than no corpus. A stale claim gets retrieved with the
same confidence as a fresh one — now with a source attached, which makes it more
persuasive and more dangerous. Provenance is what saves you: because every claim
names its origin, a wrong answer is traceable to the stale source and fixable,
instead of an anonymous hallucination you can't hunt down. You maintain a corpus
like you maintain docs, because that's what it is.
Don't let the agent write your knowledge base. A corpus distilled from real
runbooks, incidents, and decisions is yours; one an agent generates about itself is
the internet's average with your repo's name on it. Distill from sources you'd
stand behind — the whole point is that the knowledge is earned, not synthesized.
And the fair comparison. Managed RAG products and vendor retrieval are less
setup — if you want "upload your docs, get a chatbot," they'll get you there faster,
and for some teams that's the right trade. What you give up is the two things this
whole piece is about: claim-level provenance you can audit, and a knowledge base
that lives in your repo, on your machine, not in someone's index. If your knowledge
is your edge, those aren't details.
Everyone rents the same models now, so the one part of the stack that's genuinely
yours is what your agent knows — and that's only true if what it knows is real.
A folder of docs behind a retriever gives you fluent answers from sources you can't
name and can't check, which is the exact opposite of what a knowledge base is for.
The fix isn't a better embedding model. It's a pipeline: import your sources with
their provenance intact, distill them into claims small enough to retrieve and
honest enough to cite, gate the result like the artifact it is, and serve it to
every agent you run. Four commands, and your agent stops guessing from a pile and
starts answering from a corpus — one where every fact can tell you exactly where it
came from.
Give your agent knowledge. Just make sure it's the kind that can show its work.
If you build your knowledge base a way that keeps provenance without the distill
step — or you've made folder-RAG genuinely auditable — tell me where. I'll fix the
piece.
Written by the maintainer of agentproto (Apache-2.0, source) — a local, cross-vendor daemon for running and governing coding agents. Same contract as our /compare page: dated facts, named strengths, corrections by issue. Got something wrong? File an issue.
Building agentproto in the open — follow @theagentproto and @agentik_ai on X.