There's a comfortable assumption behind a lot of "agent + knowledge base" work: garbage in, garbage out. Feed an agent a messy, stale, duplicate-ridden wiki and it'll confidently tell users the wrong thing. So we invest in dedup, freshness, clean ingestion — to stop the agent from hallucinating.
I built a small testbed to measure that assumption, and it's wrong. Or rather: it's wrong about how bad ingest hurts, and the real answer is more interesting — and harder to catch.
I built a tiny agent that navigates a wiki the way a person does: it has a
wiki_search
tool and a wiki_read
tool, it decides what to look up, reads a page, and answers. No vector database, no RAG injection — the agent navigates. (This is increasingly how capable models prefer to work: they know what they need better than a query-time embedding match.)
The wiki is ten markdown pages of deliberately synthetic facts — invented services, made-up numbers the model cannot possibly know from training ("the Orion canary ring holds for 45 minutes before auto-promoting"). That last part matters: because the facts are synthetic, an agent with no wiki genuinely can't answer, so any success is attributable to the wiki, not to the model reciting what it already knew.
First, does the wiki help at all? Baseline (no wiki tools) scored 0/4.
Augmented (wiki tools) scored 4/4. A clean +100-point lift. Good — the testbed works, and the tasks are honest. (the runner)
Then I started degrading the ingest quality and measuring what happened.
I built three versions of the wiki:
I ran each condition several times per question (agents are stochastic; a single run tells you almost nothing) and measured the rate of confident, correct answers. The result:
clean-answer rate
clean 100%
stale-present 8%
stale-outranks 0%
The wiki's usefulness collapses as ingest quality drops. Ingest quality clearly matters enormously. But here's the part that changed how I think about it.
Not once. I went and read the traces expecting to find the agent confidently reciting "30 minutes." Instead, in the degraded runs it did this:
The wiki has
conflicting information: [deploy-orion] says the canary ring holds for45 minutes, while [orion-canary-ring-guide] says30 minutes.
It read both pages, noticed they disagreed, and refused to pick. It hedged. A capable model is remarkably good at not being confidently wrong — it detects the contradiction and hands it back to you.
So "bad ingest makes the agent hallucinate" is the wrong model. The real cost is subtler, and worse in a way that's easy to miss:
And a caveat that makes it worse, not better: I tested a strong model. A weaker one — or a subtler contradiction than "45 vs 30" — is exactly where the hedge turns into a confident wrong answer. The robustness I measured is a property of this model, not a guarantee.
Notice the gap between the two degraded conditions: 8% when the stale page ranked below the real one, 0% when it outranked it. That's the actionable part. A capable navigation agent reads the top results and tends to stop — so what decides your outcome is whether the authoritative page wins retrieval. Bad ingest hurts most when it lets a stale or duplicate page rank above the real one.
Which reframes the ingest-quality job. It isn't "clean everything." It's the specific things that keep the authoritative page on top: dedup (so there's no stale copy to compete), staleness detection and removal, and recency or authority signals in ranking. Prettifying pages the agent never reads is wasted effort; keeping the wrong page out of the #1 slot is the whole game.
My first grader checked whether the answer contained the right value — did "45" appear anywhere? By that measure, every condition scored 100%. No degradation at all. I nearly concluded the agent was invincible.
It scored 100% because a hedge contains the right value too ("the wiki says 45 or 30"). The lenient grader counted "45 or 30, unclear" as a win. Only when I changed the metric to a confident clean answer — the right value present and the stale value absent — did the collapse from 100% to 0% appear.
This is the quiet lesson under the loud one: the metric you pick decides whether you can see the problem at all. "Does the right token appear?" is the easy thing to measure and it would have told me, confidently, that ingest quality doesn't matter. For agent knowledge systems, measure the thing users actually care about — a clean, trustworthy answer — not the presence of a substring.
If you're building a knowledge base or memory system for agents, don't reach for the "it'll hallucinate" argument to justify ingest quality — capable agents largely won't, and a skeptic will call your bluff. Reach for the real one:
Bad ingest doesn't make your agent lie. It turns clean, authoritative answers into hedges, makes every query cost more, and hides a wrong-answer risk that surfaces the moment your model is a little weaker or the conflict a little subtler. And you won't see any of it unless you measure clean-answer rate against ground truth.
Bad ingest doesn't cause wrong answers. It destroys clean ones — quietly.
*Built as a learning project exploring agent memory / knowledge systems. The navigation-wiki testbed, the three wiki conditions, and the full results are in zachzwy/agentloop — eval/wiki-eval.js runs it, and
eval/wiki-findings.md