Traditional RAG pipelines usually work deterministically by using only static vector search[cite: 1]. Because of this, vocabulary mismatches can occur when searching for exact words or technical keywords, and the agent may be unable to verify context[cite: 1].
In this research document, I explain the complete details of the Agentic Hybrid RAG architecture I engineered[cite: 1]. It combines dense semantic search (FAISS IndexFlatIP with L2-normalized embeddings) and exact keyword search (BM25Okapi), and operates through an autonomous LLM agent using the smolagents
framework[cite: 1]. It also describes in detail how I overcame issues encountered when running the model locally on CPU, such as Python sandbox import errors, extreme CPU time latency (>213 seconds per step), and model hallucinations, by using a cloud model (Qwen2.5-72B-Instruct) and dynamic score normalization ($\alpha=0.7$), along with the benchmark results[cite: 1].
Keywords: Agentic AI, Retrieval-Augmented Generation (RAG), Hybrid Search, FAISS, BM25, Smolagents, Knowledge Grounding, Score Normalization[cite: 1].
Across enterprise decision-support systems, the deployment of large language models (LLMs) has exposed significant architectural weaknesses: static parametric knowledge cutoffs, hallucinated facts, and an inability to access proprietary ground-truth records[cite: 1]. While standard retrieval-augmented generation (RAG) addresses these weaknesses by providing external context before generation, canonical RAG pipelines continue to operate in a rigid, deterministic manner[cite: 1].
In standard RAG, every query leads to a vector lookup that is not much different[cite: 1]. As a result, pipelines fail when queries involve exact alphanumeric codes or rare technical terms (vocabulary mismatch problem), or when multiple stages of logical analysis are needed before retrieval[cite: 1].
This study aims to design, build, and evaluate an autonomous "agentic hybrid RAG" system that functions as a rational, reasoning-capable system[cite: 1]. Its main objectives are:
This system includes three mutually complementary components:
Architectural Workflow Summary:
User Query
--> `Agent Reasoning`
--> `Code Tool Call: knowledge_base_search(query)`
--> `Dual Index: FAISS Vector (cos θ) + BM25 Lexical`
--> `Min-Max Score Normalization`
--> `Weighted Fusion (α=0.7)`
--> `Top-K Passages`
--> `Grounded Answer Synthesis`
[cite: 1].
We divide the source documents $D={d_1, d_2, \dots, d_N}$ using recursive character boundary splitting[cite: 1]. To maintain semantic continuity, we use a windowing function:
Equation (1): C = Chunk(di, W_size = 300, W_overlap = 50)
Here, a chunk size of 300 characters and an overlap of 50 characters are used; this overlap prevents semantic discontinuity between chunk boundaries and ensures completeness of information[cite: 1].
Each part (chunk) $c_j$ in $C$ is embedded into a dense vector $e_j \in \mathbb{R}^{384}$ using `all-MiniLM-L6-v2`
, and then L2-normalized efficiently[cite: 1]:
Equation (2): ê_j = e_j / ||e_j||_2, such that ||ê_j||_2 = 1.0
The vectors are indexed using faiss.IndexFlatIP
[cite: 1]. For a normalized query vector $\hat{q}$, the inner product exactly computes the directional cosine similarity[cite: 1]:
Equation (3): Sim(q̂, ê_j) = q̂ · ê_j = Σ q̂_k · ê_j,k = cos(θ)
IndexFlatIP
, by avoiding lossy quantization or clustering-based approximate calculations, guarantees 100% accuracy in vector space[cite: 1].
Concurrently, chunks are tokenized after stop-word sanitation[cite: 1]. The lexical relevance score of query tokens $Q={t_1, \dots, t_m}$ against chunk $c_j$ is computed via the probabilistic BM25 model, with saturation parameter $k_1=1.5$ and length normalization $b=0.75$[cite: 1]. Since the distributions of dense vector inner products and BM25 scores do not match, directly combining them causes a large imbalance or skew[cite: 1]. Therefore, we apply feature-level min-max scaling[cite: 1]:
Equation (5): Ŝ(x) = [S(x) - min(S)] / [max(S) - min(S) + ε]
The final unified hybrid ranking score is computed as[cite: 1]:
Equation (6): S_hybrid(c_j) = α · Ŝ_vector(c_j) + (1 - α) · Ŝ_keyword(c_j)
where $\alpha=0.7$ prioritizes semantic context while allocating 30% weight to exact keyword preservation[cite: 1].
The shift from an interactive notebook to a practical standalone system in VS Code has revealed four critical failure points[cite: 1]:
| Challenge / Failure Mode | Observed Root Cause | Engineering Countermeasure |
|---|---|---|
1. Sandbox Import Violation[cite: 1] |
Small LLM (1.5B) hallucinated external APIs (`requests` , `wolframalpha` )[cite: 1]. |
Enforced explicit tool prompting; transitioned backbone to Qwen2.5-72B-Instruct[cite: 1]. |
2. CPU Compute Latency[cite: 1] |
Local inference exceeded 213 seconds per reasoning step on consumer CPU[cite: 1]. | Offloaded LLM reasoning to Hugging Face hosted GPU endpoints (InferenceClientModel )[cite: 1]. |
3. API Interface Drift[cite: 1] |
smolagents renamed HfApiModel to InferenceClientModel across package updates[cite: 1]. |
Implemented multi-version dynamic try-except fallback import wrapper[cite: 1]. |
4. Vocabulary Mismatch[cite: 1] |
BM25 failed on conceptual synonyms; pure vector missed exact alphanumeric IDs[cite: 1]. | Engineered dual-engine weighted hybrid fusion ($\alpha=0.7$) and Min-Max scaling[cite: 1]. |
When running a 1.5B model locally, the agent made uncontrolled network calls (for example: import requests; request.get('http://api.wolframalpha.com/...')
)[cite: 1]. Since the CodeAgent
is running in a secure sandbox that allows only standard math libraries (math
, re
, collections
), an Interpreter Error
occurred and execution stopped[cite: 1].
Solution: The agent's reasoning backbone was upgraded to Qwen2.5-72B-Instruct
, and that resolved this issue[cite: 1]. This large model, which follows instructions accurately, strictly adhered to the given system contract and made precise calls to knowledge_base_search(query=...)
without unnecessary assumptions or hallucinations about external dependencies[cite: 1].
When running the first stage of the iterative generation on local CPU hardware, an unacceptable time of 213.85 seconds was required[cite: 1]. By shifting model inference to high-capacity cloud endpoints, the latency at each stage dropped from over 210 seconds to under 2.4 seconds; this indicates an approximately 89-fold increase in speed[cite: 1].
To measure retrieval precision and the ability to control hallucination, the hybrid agentic pipeline was evaluated across three different search categories[cite: 1]:
| Retrieval Paradigm | Semantic Recall | Exact Identifier | Composite MRR@3 |
|---|---|---|---|
Pure Vector (IndexFlatIP)[cite: 1] |
0.94[cite: 1] | 0.62[cite: 1] | 0.81[cite: 1] |
Pure Keyword (BM25Okapi)[cite: 1] |
0.48[cite: 1] | 0.96[cite: 1] | 0.74[cite: 1] |
Hybrid Fusion ($\alpha=0.7$)[cite: 1] |
0.95[cite: 1] |
0.93[cite: 1] |
0.96[cite: 1] |
In benchmark tests, the agent achieved 100% grounding verification on specific target queries[cite: 1]. When answering the question, "What is RAG, and why are embeddings important in a RAG system?", the agent used a hybrid search tool and analyzed information from the RAG001
and EMB001
chunks, producing a comprehensive response that clearly explained vector conversion, semantic indexing, and grounded context synthesis without factual errors[cite: 1].
To scale this architecture into enterprise production, we are implementing four key enhancements[cite: 1]:
This research demonstrated the full reality of an agentic hybrid RAG architecture[cite: 1]. By combining FAISS dense vector retrieval, BM25 sparse lexical indexing, and an autonomous code-executing LLM agent, the system overcomes the limitations of static RAG pipelines[cite: 1]. We addressed core engineering challenges related to sandbox execution safety, inference latency, and library version compatibility[cite: 1].
The empirical results confirm that integrating hybrid retrieval with autonomous agent decision-making delivers better grounding, stronger keyword precision, and highly fast, factual synthesis, establishing a modular framework for modern enterprise AI architectures[cite: 1].