Imagine a filing cabinet with half a million documents about home renovation. Somewhere inside sits one file about a football club.
Now someone asks your AI assistant: “Do we have anything about the club’s sponsorship?”
The assistant has a search tool pointed at that cabinet. Should it use it? It knows the tool exists. It has no idea what’s behind it. So it guesses. Sometimes it searches a renovation archive for football news; sometimes it answers confidently from memory while the one document that matters sits unread in the cabinet.
Remember that football file. It comes back near the end, where it almost breaks one of the techniques.
The term sounds like database furniture, so let’s pin it down.
A data store — OpenAI says vector store, Google says data store, others say knowledge base — is a box of files an AI can search. You upload PDFs, wikis, contracts, chat logs. Each file is split into chunks, every chunk gets an embedding (a numerical fingerprint of its meaning), and a tool like file_search retrieves the chunks whose fingerprints resemble the question.
Two properties of the box matter here. It’s opaque: the agent sees a name like customer-docs and maybe a one-line description. And it's alive: users add and delete files whenever they want, so last month's truth about the contents probably isn't this month's.
Every message your agent receives triggers a quiet decision: call the search tool, or answer from what the model already knows. Teams pour energy into what happens after that decision — chunking, rerankers, hybrid search. The decision itself usually rests on one hand-written sentence, drafted months ago, describing content that has changed daily ever since.
Guessing wrong costs both ways. Needless retrieval actively hurts: irrelevant retrieved text can flip otherwise-correct answers, and the worst passages are exactly what a pointless search returns — topically related text with no answer in it. Skipping fails too, because model memory is weakest on the long-tail private facts data stores exist to hold.
The research on this decision — Self-RAG, Adaptive-RAG, FLARE — almost all works the query side: is the question hard, is the model uncertain. Almost nothing asks the data side: what is in the store right now?
That signal is worth real accuracy. Elastic measured it: an agent picking which index to search scored 77% from index names alone, and over 90% once each index carried a stored description. Same model, same data. The only change was telling the agent what’s inside.
Survey what platforms actually give you for this and a pattern appears:
Meanwhile the same design keeps getting reinvented wherever routing works well. llms.txt is a precomputed site summary for LLMs. Aider’s repo map is a token-budgeted summary of a codebase. Anthropic’s Agent Skills preload a 30-to-100-token description and pull the payload on demand. It’s the same pattern five times: a precomputed, bounded summary used as a routing signal.
Every version shares the same missing half: who writes the summary, and who keeps it true when the data changes? For a store fed by user uploads, “a developer writes it” is not an answer. The summary has to be derived from the data, and re-derived when the data moves.
Everything below was priced against a real production deployment: 80 non-empty data stores holding 512,287 files, 1.4 million chunks and 722 million tokens. One store dominates, with 511,962 of those files and 94% of the tokens. The largest single file is a 6.7-million-token book.
The naive fix — dump the store’s file list into the router’s prompt — dies on those numbers: half a million slow rows per decision, file names like scan_0042_final_v3.pdf carrying zero signal, and bloated context degrading the model's own reasoning.
So the summary gets hard constraints. Under ~200 tokens, because a router LLM reads it on every turn. Free to read — a plain column read, zero LLM calls. Updated on file events from the ingestion pipeline, never on user traffic (a stable summary also keeps the router’s prompt cacheable, and Manus reports a 10x cost gap between cached and uncached input). And nothing whose LLM cost grows with file count, because a store this size would bankrupt it.
Five strategies, priced with a Haiku-class model ($0.80/M input tokens). Backfill means summarizing all 80 stores once.
Two die immediately. Strategy A — abstract every file with an LLM — costs about $1,000 in backfill plus an LLM call for every future upload, forever. Strategy C — extract topics from every chunk — lands near $864. Both scale with corpus size; Microsoft’s LazyGraphRAG post prices this style at roughly 1000x vector-RAG indexing cost. The lesson from both corpses: never let the LLM read the corpus. Something cheaper has to read it first and hand the LLM a compressed view.
The ingestion pipeline already embedded every chunk into a vector database. Those 1.4 million vectors are a sunk cost, and they already encode topical structure. So: pull the store’s vectors (sample down to 50k), run KMeans into ~30 clusters, take the 3 chunks nearest each centroid, and make one small LLM call per cluster: “label this in 2–5 words.” Write the labels to a topics column. Done.
Backfilling all 80 stores: $2.96, or $0.037 per store. LLM cost grows with topics, not files — a 512k-file store costs the same to summarize as a 12-file one, because both reduce to ~30 clusters before any LLM sees a word.
If this looks familiar, it’s BERTopic’s embed-cluster-represent recipe, and a single level of RAPTOR, which clusters chunk embeddings and LLM-summarizes each cluster recursively into a tree. RAPTOR builds that hierarchy to retrieve from; here, one level of it becomes a routing signal. The contrarian option: no embeddings, no semantics at all. Tokenize chunks in the ingestion pipeline (lowercase, stopwords, stemming), keep per-file term counts plus a store-wide document-frequency table, and let one LLM call turn the top terms into labels. Backfill for all 722 million tokens: $0.10. If RAPTOR is the maximalist pole of corpus summarization — a whole tree of LLM calls — this is the opposite one: pure counting, a single call at the very end.
Now the football file. Raw counts bury it:
Top terms by count: renovation (4,012,118), wall (1,802,394), permit, contractor, drywall… football is nowhere. The most distinctive file in the store is invisible.
The fix is ranking twice. One list by raw count, for the dominant themes. One by count × IDF, where IDF asks: how rare is this term across the store’s own files? Football’s 200 mentions of a term confined to a single file among half a million score near the top of the novelty list. The sponsorship question from the opening is now answerable.
The honest downsides: bag-of-words can’t see synonyms (renovation and remodeling are strangers to it), and the novelty list promotes noise too — employee first names float up, dense in one file and absent across the rest. Cheap’s price is paid in label quality.
The two signals fail differently, which is why they combine. Embeddings answer “what is this text about”; keyword statistics answer “what is dense and distinctive here.” The hybrid labels each cluster from its representative chunks plus its top count×IDF keywords — same number of LLM calls as B, richer prompt, $3.45 total.
The gain shows on fuzzy clusters. A real one, full of Java/Spring/Hibernate/Maven content:
Generating a summary once is a weekend project. The moving data is the real problem — and where published techniques quietly give up. GraphRAG re-summarizes changed communities, but document removal is explicitly out of scope. RAPTOR’s summary tree requires full recomputation after changes; follow-up work patches additions, not deletions. BERTopic’s online mode doesn’t support deletion either.
So freshness has to be designed. The skeleton: a file event marks the store dirty, a ~60-second debounce absorbs bursts (50 uploads trigger one regeneration, not fifty), a cheap incremental update runs, and a drift gate decides whether an LLM gets involved at all.
If a labeling call fails, the old topics stay in place and a retry fires with backoff. Stale-but-present beats fresh-but-flaky for a signal read on every turn. Steady state: single-digit dollars per week for the store, scaling with file churn — not query traffic, not corpus size.
Two findings from running it for real. HDBSCAN behaves well on small, clean, mixed content — clear clusters, an honest refusal to classify the ambiguous quarter of chunks. But on a 20k-vector sample of the store it collapsed 70.5% of everything into one giant blob: two usable topics for half a million files. KMeans with k=10 forced out ten usable topics (~120 tokens) by assigning every point, noise included. Decisive beat honest; KMeans shipped, with silhouette-picked K.
That giant blob is also where the hybrid earns its keep. An embedding summary of a homogeneous store says “enterprise support docs” — but users ask about clients by name, and names are exactly the dense-and-distinctive terms the IDF list promotes. Embeddings see themes; keywords see names.
Ship Strategy B first: $2.96 to backfill all 80 stores, at most ~$0.04 per store refresh, no new dependency beyond a clustering library. B+D is the pre-planned upgrade the moment label-quality evals complain — 16% more cost, an addition rather than a rewrite. D is the fallback where embeddings don’t exist.
Three things worth keeping:
This pattern — a precomputed, token-budgeted, machine-maintained summary as a routing signal — keeps getting reinvented as llms.txt, repo maps, skill descriptions, index descriptions. It will keep being reinvented until platforms treat it as infrastructure. Until then, you can build it yourself for the price of a coffee.
And your agent will finally know about the football file.
Numbers come from pricing five prototype strategies against a live production deployment of 80 data stores, using a Haiku-class labeling model at late-2025 rates. Your data will differ; the scaling arguments won’t.
Your AI Agent Has No Idea What It Knows was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.