The Heavy Lifting of the Inverted Index #
The core of traditional search is the inverted index. Think of it as the index at the back of a massive textbook. Instead of scanning every page for the word "transformer," the engine looks up the term and immediately gets a list of every document containing it. This is the only reason we can query petabytes of data in under 50ms.
But it's not just about matching strings. A production-grade search engine performs complex preprocessing during indexing:
Tokenization & Normalization: Ensuring "running" and "run" are treated as the same concept.Stop-word Filtering: Ignoring "the" or "and" to prioritize high-signal terms.Weighting (BM25/TF-IDF): Calculating the rarity of a term. If a document contains the word "ergonomic" five times, it's significantly more relevant than a document that contains the word "computer" five times.
Databases are built for transactional correctness (ACID compliance). Search engines are built for relevance. Trying to "just add search" to a relational database using
LIKE %query%
is a recipe for a system crash once your dataset hits a certain scale.## Benchmarking Lexical vs. Semantic Search
To understand why we need both, look at how they handle different query types. In my experience benchmarking these workflows, the failure points are predictable.
Exact Match (e.g., "SKU-99281-X"):
Lexical: 100% accuracy. It finds the exact string.
Vector/Embedding: Variable. It might find "SKU-99281-Y" because the vectors are mathematically close, which is a failure in a logistics context.
Conceptual Match (e.g., "portable workstation" vs "laptop"):
Lexical: 0% accuracy (unless you have a manually maintained synonym list).
Vector/Embedding: High accuracy. It understands the semantic relationship.
The Hybrid AI Workflow #
The real "pro" move isn't choosing one or the other; it's implementing a hybrid search architecture. This is the only way to achieve production-grade reliability.
A typical high-performance deployment looks like this:
-
Parallel Retrieval: The system sends the query to both a BM25 lexical index and a vector database (like Milvus or Pinecone).
-
Reciprocal Rank Fusion (RRF): The results are merged using a scoring algorithm to balance exact matches and semantic relevance.
-
LLM Reranking: A small subset of the top 20-50 results is passed to a cross-encoder or an LLM to determine the final order based on the specific user intent.
For those building an LLM agent or a RAG (Retrieval-Augmented Generation) pipeline, relying solely on embeddings often leads to "hallucinations by retrieval"—where the model gets a semantically similar but factually wrong document. Integrating a lexical step fixes this.
If you're configuring a hybrid search system, avoid the mistake of giving equal weight to both scores. I've found that biasing toward lexical results for short queries (1-2 words) and biasing toward semantic results for long-tail queries (natural language questions) yields the best precision.
search_params:
lexical_weight: 0.7 # High weight for keyword precision
semantic_weight: 0.3 # Supplemental context
rerank_model: "cross-encoder/ms-marco-MiniLM-L-6-v2"
top_k_retrieval: 100
final_top_n: 5
The "magic" of the modern search experience isn't that the old tech disappeared; it's that we've finally found a way to layer LLMs on top of a foundation of inverted indices.
Next Claude Code vs GPT-4o: Adversarial Review Workflow →