New evidence shows agentic keyword search hits 94.5% of RAG faithfulness with zero vector store — here's how to decide whether your team still needs one.
Table of Contents #
Every RAG kickoff meeting still starts the same way: pick a vector database, decide on chunking strategy, choose an embedding model, then argue about hybrid search weights for two sprints before anyone ships a working retrieval path. That sequence has become so ritualized that most teams never ask whether it’s the right starting point at all. They inherit it from a tutorial, a vendor pitch, or the last RAG project they shipped, and the vector database becomes a load-bearing piece of infrastructure before anyone has measured whether it’s earning its keep.
The uncomfortable question underneath that ritual is simple: for a large share of RAG workloads, is the vector database actually where the retrieval quality comes from — or is it the LLM’s reasoning over whatever gets retrieved, however crude the retrieval mechanism? That question just got a real, empirical answer, and it should change how architects scope the next RAG or agentic-search project rather than defaulting to the standard stack.
The Evidence: Keyword Search Closing the Gap #
A paper out of Amazon Science, presented at AAAI 2026 and titled “Keyword Search Is All You Need: Achieving RAG-Level Performance Without Vector Databases Using Agentic Tool Use,” ran a systematic comparison between classic RAG pipelines (embedding-based semantic retrieval into an LLM context) and tool-augmented agents that only had access to basic keyword search over the same document corpus. No embeddings, no ANN index, no vector store — just an LLM agent deciding what to search for, issuing keyword queries as a tool call, reading results, and deciding whether to search again.
The result: agentic keyword search reached 94.5% of RAG’s faithfulness score with zero vector store in the loop. Separately, teams using Search-R1 — a reinforcement-learning framework that trains the retrieval policy itself, letting the model learn when to query, what to ask, and when it has enough information to stop — beat standard RAG baselines by roughly 24% relative on multi-hop QA benchmarks. The mechanism generating retrieval quality, in other words, is shifting from “how good is your embedding space” to “how good is your agent’s search policy.”
This isn’t an argument that vector databases are obsolete. It’s evidence that the field has been bundling two separable things — retrieval mechanism and reasoning-over-results — and attributing most of the quality gain to the wrong one. For architects, that’s a scoping problem, not a trivia fact.
Why This Changes the Default Architecture Decision #
The standard RAG reference architecture — embed at ingest, store vectors, do ANN similarity search at query time, stuff top-k chunks into context — was built for a world where the LLM was a relatively passive consumer of retrieved text. It couldn’t decide to search again, couldn’t reformulate a query, couldn’t recognize that its first retrieval was insufficient. Given that constraint, investing heavily in retrieval precision up front made sense: you got one shot, so the vector index had to be good.
Agentic models remove that constraint. A model that can call a search tool multiple times, inspect intermediate results, and decide to refine its query compensates for a cruder retrieval mechanism through iteration. That’s the architectural insight underneath the 94.5% number — it’s not that keyword search became smarter, it’s that the agent loop absorbed work that used to be the vector index’s job alone. This is directly analogous to what happened with tool-use agents replacing single-shot function calling: give the model the ability to retry and inspect, and a lot of upfront precision engineering becomes redundant.
For teams currently scoping a new RAG system, that reframes the build order. The vector database stops being the foundational decision made in week one and becomes one candidate retrieval backend to evaluate against an agentic keyword-search baseline, chosen based on measured faithfulness and cost — not chosen by default because every RAG tutorial starts there.
Where Vector Search Still Wins #
None of this means ripping out Pinecone, Weaviate, or Qdrant from a working production system. Semantic retrieval still meaningfully outperforms keyword search in specific, identifiable conditions: when queries use different vocabulary than the source documents (paraphrase-heavy retrieval, cross-lingual search), when the corpus lacks strong lexical structure (short, jargon-light text, or content where meaning doesn’t correlate with shared terms), and when latency budgets don’t allow multi-turn agentic search loops — a single ANN lookup is still faster than three or four sequential tool calls.
The Amazon Science result is strongest on QA-style benchmarks over structured or moderately structured document corpora — the kind of internal knowledge base, policy document set, or ticket history that a lot of enterprise RAG systems are actually built on. It’s weaker evidence for use cases like semantic product search, recommendation-adjacent retrieval, or multilingual support corpora, where the vocabulary mismatch problem vector embeddings solve is the dominant failure mode, not an edge case.
Architecture Impact #
What changes in system design? Retrieval backend becomes a configurable, swappable component evaluated against faithfulness and cost metrics rather than a fixed upfront commitment. Systems built agent-first can start with a keyword/BM25 index behind a search tool, defer vector infrastructure until measured evidence justifies it, and add semantic search as an additional tool the agent can choose to call rather than the sole retrieval path.
What new failure mode appears? Uncontrolled agentic search loops introduce a new latency and cost failure mode: an agent that keeps re-querying because it can’t tell when it has “enough” context will burn tokens and wall-clock time unpredictably, unlike a fixed top-k RAG call with bounded latency. Teams that swap to agentic search without a hard cap on search iterations or a trained stopping policy (as in Search-R1) will see p99 latency and per-query cost variance spike even as average quality improves.
What enterprise teams should evaluate:
ML/platform engineering: Run a faithfulness and cost comparison — agentic keyword search vs. current vector pipeline — on a held-out sample of production queries before committing infrastructure spend either direction.Data engineering: Assess whether the document corpus has strong lexical overlap with expected queries (favors keyword) or heavy vocabulary mismatch (favors embeddings), since that single factor predicts most of the outcome.SRE/observability: Instrument search-iteration count and per-query cost as first-class metrics if adopting agentic retrieval — this is the new tail-latency risk that didn’t exist in single-shot RAG.
Cost / latency / governance / reliability implications: Dropping a vector database removes embedding compute, index storage, and ANN query infrastructure — often a meaningful line item at scale (embedding refresh jobs alone can run into five figures monthly for large, frequently-updated corpora). Against that, agentic search trades infrastructure cost for inference cost: multiple sequential tool calls per query can push per-request latency from ~200-400ms (single ANN lookup) to 2-5 seconds (three to five search-and-reason turns), which matters directly for any user-facing SLA.
Decision Framework #
Start by measuring, not by defaulting. Take a representative sample of production or expected queries — a few hundred is enough for a first pass — and run them through both a plain BM25/keyword index wrapped as an agent tool, and your existing (or planned) vector pipeline. Score both on faithfulness (does the answer match ground truth in the source docs) and hallucination rate, not just retrieval hit rate, since hit rate can look fine while the LLM still fabricates details the retrieved chunk didn’t support.
If keyword search lands within 5-10 percentage points of vector search on that faithfulness metric, the infrastructure savings and reduced operational surface area (no embedding pipeline to maintain, no index to keep in sync with source updates) usually justify shipping keyword-first and adding vector search later as an opt-in tool for query types that need it. If the gap is large — which will show up clearly on paraphrase-heavy or cross-lingual query sets — the vector index is earning its cost and the standard architecture remains correct.
Implementation Guide #
Start with the cheapest possible baseline: stand up a keyword/BM25 search tool (Elasticsearch, OpenSearch, or even SQLite FTS5 for small corpora) behind your existing agent framework, and give the model a single search tool with the ability to call it up to some fixed number of times per query — three to five is a reasonable starting cap. Don’t build a custom query-reformulation layer yet; let the model’s own reasoning handle reformulation for the first few weeks of testing. This gets you a working comparison point in days, not the multi-sprint timeline a vector pipeline usually requires.
The mistake to avoid is treating this as an all-or-nothing migration. Teams that rip out a working vector pipeline in one move lose the ability to A/B test, and teams that bolt keyword search on as a token gesture without measuring faithfulness end up back where they started, just with more moving parts. Run both paths in parallel against logged production queries for at least two to three weeks before making an infrastructure commitment either way — retrieval quality on a small eval set does not reliably predict behavior at production query volume and diversity.
The signal that this is working is a stable or improving faithfulness score alongside a search-iteration count that plateaus rather than climbs — if your agent is calling search five, six, seven times on an increasing share of queries, that’s a sign the retrieval mechanism (keyword or vector) isn’t surfacing the right content, not that the agent needs a higher iteration cap. Track iteration count as a leading indicator of retrieval quality, not just an infrastructure cost metric.
Over six to twelve months, the teams getting this right converge on a hybrid architecture that stops treating “vector vs. keyword” as a binary choice: a keyword-first agentic search tool as the default path, with a vector index as a second tool the agent invokes specifically for paraphrase-heavy or semantically ambiguous queries, selected by the model itself based on query characteristics or by a lightweight classifier upstream. That’s a smaller, cheaper vector index serving a narrower job — not eliminated, just no longer the default answer to every retrieval problem.
Sources #
Keyword Search Is All You Need: Achieving RAG-Level Performance Without Vector Databases Using Agentic Tool Use — Amazon ScienceSearch-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (arXiv 2503.09516)Best Vector Databases in 2026: Pricing, Scale Limits, and Architecture Tradeoffs — MarkTechPost
Enterprise AI Architecture
Want more enterprise AI architecture breakdowns? #
Subscribe to SuperML.