Most RAG applications follow a simple path:
Retrieve a few documents → send them to an LLM → generate an answer.
That works for straightforward lookup questions. But it becomes fragile when the question spans multiple manuals, needs numerical reasoning, contains tables, or asks for an engineering recommendation grounded in plant documentation.
I built an Agentic RAG system for Small Hydropower Plant (SHP) engineering to explore a harder question:
Can a multi-agent retrieval workflow improve the quality of answers from private technical documents without depending on paid inference APIs?
The answer, based on my evaluation so far, is nuanced: it improves retrieval precision, but comes with a substantial latency cost and does not automatically outperform a well-designed plain RAG pipeline on faithfulness.
The complete implementation is available here: Agentic RAG for Small Hydropower.
One of my goals was to make this project accessible to students and independent learners who cannot afford recurring API bills.
The core RAG pipeline runs locally using Ollama and open-weight models. Document processing, embeddings, retrieval, answer generation, and runtime quality checks do not require paid cloud inference APIs. Once the required models and dependencies are downloaded, you can experiment on compatible hardware without paying per question or per token.
For offline benchmarking, I used Gemini and Groq through their available free tiers to evaluate answers against the golden dataset. Those services are optional: evaluation can also use a local Ollama judge. Free-tier quotas and availability may change, so they should not be treated as an unlimited resource.
This does not mean running the system has zero cost — hardware, electricity, and inference time still matter. But if you already have a suitable computer, it removes a major barrier to learning: a recurring inference bill.
For me, that makes the project useful beyond hydropower. It provides a practical starting point for learning document ingestion, hybrid retrieval, agent orchestration, tool use, and evaluation within a limited budget.
Small hydropower projects involve a mix of technical manuals, regulatory guidelines, turbine-selection references, hydrology documents, financial policies, scanned reports, tables, and drawings.
A typical question may require several kinds of evidence at once:
A traditional RAG pipeline may retrieve relevant passages, but it usually has no mechanism to decide whether the retrieved context is sufficient, whether a calculation should use a tool, or whether the generated answer contains unsupported claims.
That is where the agentic approach becomes useful.
The first challenge was preparing the documents. Hydropower knowledge arrives in different formats: digital PDFs, scanned reports, Word documents, presentations, spreadsheets, and images. Each format needs a different extraction approach.
I built a multi-tier parsing pipeline that converts these files into Markdown before chunking and embedding.
For formats with accessible document structure, the pipeline uses dedicated libraries:
This avoids sending every file through a more expensive OCR process.
For PDFs, the pipeline first uses pymupdf4llm to extract text into Markdown, aiming to retain headings and table structure.
If the extracted output contains at least 80 characters, the current implementation accepts it. If the output is shorter — or extraction fails — the document moves to the Docling fallback.
This is a practical routing heuristic, but it has a limitation: a PDF containing both digital text and scanned pages may pass the threshold even when some pages need OCR. Page-level quality checks would improve this approach.
Scanned PDFs and failed PDF extractions are processed with Docling, with OCR and table-structure recovery enabled when the required backend is available. Images also enter the Docling conversion path.
The PDF OCR configuration prefers EasyOCR, with RapidOCR as an alternative. It uses CUDA acceleration when available and otherwise falls back to CPU execution.
The aim is to recover readable text and useful table structure. Complex drawings, dense tables, and formulas remain difficult, so extracted Markdown should not be assumed to reproduce every detail correctly.
Alongside the Markdown, the parser records source information such as the filename, file type, content hash, parsing duration, and page count where available. Parsing failures are recorded explicitly.
The content hash also supports caching: an unchanged document can reuse its existing Markdown output, avoiding unnecessary processing.
The resulting flow is:
Raw documents → format-specific parsing or OCR → Markdown and metadata → chunking → embeddings and indexing
This stage directly affects everything that follows. If a table loses its units or OCR misreads a formula, the retrieval and generation stages may work with incorrect evidence. Improving parsing is therefore part of improving answer quality.
Binary SHA-256 deduplication is an efficient first line of defense to prevent wasting heavy parsing and OCR compute on identical files, but for the same converted file, Binary deduplication will NOT be applicable — the system will treat them as two completely separate files
An actual record saved in data/parsed/metadata.json{ "source_path": "D:\\GenAi_Code\\Agentic_RAG\\data\\raw\\AK - Hydrology of Ungauged Catchment.pdf", "file_name": "AK - Hydrology of Ungauged Catchment.pdf", "file_type": "PDF", "file_hash": "127653ce9e3151fd6732fd83bfc66e570ab762bdafb22d568fcef524ce0d19bc", "page_count": 0, "parsed_at": 1793168584.5575593, "parse_duration_s": 0.0, "parsed_md_path": "D:\\GenAi_Code\\Agentic_RAG\\data\\parsed\\AK - Hydrology of Ungauged Catchment__127653ce.md", "error": null}
After parsing, the documents are available as Markdown. The next challenge is dividing them into passages that are useful for retrieval without losing their context. I implemented a Markdown-aware hierarchical chunker. It uses document structure and text length for boundaries, rather than an LLM or embedding-based semantic segmentation model.
The chunker uses a 4-pass rule-based pipeline rather than an LLM or arbitrary character slicing:
[Raw Markdown Document] │ ▼ ┌─────────┐ │ Pass 1 │ Scan whole document → Divide into Structural Sections └─────────┘ │ Output: `sections` list ▼ ┌─────────┐ │ Pass 2 │ Process all sections → Split into Target-Sized Chunks (1,800 chars) └─────────┘ │ Output: `raw_chunks` list ▼ ┌─────────┐ │ Pass 3 │ Scan all raw chunks → Merge Tiny Leftover Fragments (< 150 chars) └─────────┘ │ Output: `merged_chunks` list ▼ ┌─────────┐ │ Pass 4 │ Enrich all chunks → Inject Citation Banners & Compute Token Counts └─────────┘ │ Output: `final_chunks` list (Written to `chunks.jsonl`) ▼[Final Output]
Instead of chopping text blindly every NNN characters, the chunker walks through the document line-by-line, tracking markdown headings (#, ##, ###, ####) like a tree or outline. Whenever a heading or a horizontal divider (---) appears, it finishes the current section. It also checks if the heading mentions a slide or page number.
When a heading includes a label such as “Slide 12” or “Page 8,” that location is captured as metadata. Page references therefore depend on what the parsing stage preserves; they are not automatically available for every chunk.
Within a section, long text is divided at paragraph boundaries. Oversized paragraphs are further split at sentence boundaries.
In simple words: If a text section is too long (>1,800 characters):
The default settings are:
The overlap is conditional, rather than a fixed sliding window: the previous paragraph is repeated only when it fits within the overlap allowance.
The 1,800-character setting is a target, not a strict maximum. Long individual sentences, table rows, fragment merging, and added source metadata can produce larger chunks.
max_chars = 1800overlap_chars = 150Imagine a document section with four paragraphs:Paragraph 1 (900 chars): Details about Francis turbine runner design.Paragraph 2 (800 chars): Details about guide vanes and water flow.Paragraph 3 (85 chars): "Cavitation occurs when local water pressure drops below vapor pressure."Paragraph 4 (600 chars): Explaining preventative coatings and blade maintenance.What happens:1. Building Chunk 1: Adds Para 1 (900 chars) + Para 2 (800 chars) + Para 3 (85 chars) = 1,785 chars. Current chunk is full (≈1800).2. Along comes Paragraph 4 (600 chars): 1,785+600=2,385 chars (exceeds 1,800!). So Chunk 1 is saved containing: [Para 1 + Para 2 + Para 3].3. Now, how should Chunk 2 start? The code looks at the last paragraph of Chunk 1 (Paragraph 3). Paragraph 3 is 85 chars long. Is 85≤150 (overlap_chars)? YES! Because Paragraph 3 is a short, complete sentence, the code carries it over as a natural bridge into Chunk 2:Chunk 2 begins with:"Cavitation occurs when local water pressure drops below vapor pressure.[Paragraph 4 text about coatings and blade maintenance...]" Result: Chunk 2 now has immediate context on why coatings are needed, without cutting any words or sentences in half!What if Paragraph 3 had been huge (e.g. 600 chars)? If Paragraph 3 had been 600 chars, len(current_chunk[-1]) <= 150 would be False. Repeating a massive 600-character block would waste one-third of Chunk 2's capacity. The condition fails, so Chunk 2 starts cleanly with Paragraph 4 alone.
Tables need different handling from narrative text. In standard chunkers, a long table gets sliced across arbitrary character limits, leaving half the data rows orphaned without column names. This chunker: If a table has 40 rows and exceeds 1,800 characters, it slices it into smaller tables, repeating the top column headers on every slice:
When an embedding model or LLM sees an isolated chunk like “The operating head is 45 meters”, it doesn’t know which plant or turbine that applies to. To fix this, the chunker injects a context banner directly at the top of the chunk’s text:
[Source: Turbine_Manual.pdf | Location: Slide/Page 12 | Section: Turbine Manual > Selection Guidelines > Francis Turbines]
The operating head is 45 meters...
In Pass 2:The chunker fills Chunk 1 up to the target limit (∼1,800 characters).There are 50 characters left over at the end of the section: example:"Refer to Figure 4 for full assembly diagrams." (49 chars)Pass 2 splits this leftover into Chunk 2.The Problem:If left alone, Chunk 2 would be indexed into Qdrant as its own chunk with only 49 characters! If a user searches for something, retrieving that tiny chunk alone is useless.Pass 3 to the rescue:Pass 3 looks at Chunk 2: Size = 49 characters. Is 49<150 (min_chars)? Yes! It appends that 49-character sentence back into Chunk 1.Result: Chunk 1 becomes 1,849 characters (which is completely fine because 1,800 is a soft target), and the tiny orphan chunk is eliminated!
Both the raw content and the banner-enriched content are saved.
These counts help inspect chunk sizes, but they are not exact token counts for the Qwen embedding model because the tokenizers differ.
The goal is to create passages that remain understandable after retrieval: a paragraph carries its section context, a table slice retains its headers, and each result can be traced back to its source.
After chunking, each passage is embedded locally using qwen3-embedding:0.6b. The indexer expects a 1,024-dimensional vector and stores it in a Qdrant collection named shp_knowledge. Every chunk is represented by a list of 1,024 numbers.
Each stored point has three parts:
point_uuid = manager.chunk_id_to_uuid("HDP_AHN511__9fe0f668__c0002")# Returns deterministic UUID5: "877250c9-fe3a-55c0-96a9-89d7d65a689a"
{ "chunk_id": "HDP_AHN511__9fe0f668__c0002", "source_file": "HDP_AHN511.ppt", ... "embedding": [ 0.002478, 0.005229, -0.005374, -0.025804, 0.025764, ... 1024 float numbers total ... ]}
point = models.PointStruct( # 1. Unique deterministic ID id="877250c9-fe3a-55c0-96a9-89d7d65a689a", # 2. The 1024-dim vector for Cosine similarity search vector=[0.002478, 0.005229, -0.005374, ...], # 3. Payload: Metadata + Context text for LLM retrieval and filtering payload={ "chunk_id": "HDP_AHN511__9fe0f668__c0002", "source_file": "HDP_AHN511.ppt", "file_type": "PowerPoint (legacy)", "is_table": "false", "page_or_slide": 1, "breadcrumb": "HDP_AHN511 > Slide 1: Dynamic Pricing of electrical power", "section_hierarchy": ["HDP_AHN511", "Slide 1: Dynamic Pricing of electrical power"], "token_count": 121, "char_count": 531, "content": "[Source: HDP_AHN511.ppt | Location: Slide/Page 1 | Section: ...]\n\n## Slide 1: ...", "text": "[Source: HDP_AHN511.ppt | Location: Slide/Page 1 | Section: ...]\n\n## Slide 1: ..." })
The embedding helps locate passages with similar meaning. The payload lets the application read the evidence, apply filters, and identify where it came from.
Metadata is assembled during parsing and chunking, then carried into the index. It is derived from filenames, document structure, extracted location labels, and chunk statistics rather than generated by an LLM.
This metadata helps the agents retrieve more selectively. A request for a discharge table can use a table filter, while a document-specific question can constrain retrieval to a source file.
The indexer reads embedded chunks from JSONL, checks that each vector has the expected dimensions, and uploads points in batches of 128 by default.
It converts each chunk ID into a deterministic UUID5 identifier. Re-indexing the same chunk updates the corresponding point instead of creating another copy.
That handles repeated indexing of unchanged chunks. Updating a document is a separate concern: changed content can produce new chunk IDs, so removing obsolete document versions requires an explicit cleanup strategy.
The collection uses cosine similarity to compare query vectors with stored chunk vectors. Its configuration requests on-disk vectors and payloads, and the indexer creates payload indexes for source_file, file_type, is_table, and page_or_slide.
Qdrant fits three requirements of this project.
Local deployment. The application can connect to a Qdrant server inside the organization’s infrastructure. The index manager also supports embedded local disk storage for development, reducing setup requirements for learners.
Similarity search with metadata filters. Engineering questions often need both semantic matching and precise constraints. Keeping vectors and source metadata together allows the retriever to combine those requirements.
A straightforward indexing workflow. Collections, payloads, deterministic point IDs, and batch upserts map naturally to the project’s document-and-chunk model.
These were practical reasons for choosing Qdrant; ChromaDB was also a viable option for local development and metadata-based retrieval. I chose Qdrant for its dedicated vector-search service and configurable payload indexes, which fit my document-filtering requirements. This was an architectural choice, rather than a claim that Qdrant universally outperforms ChromaDB. The project does not benchmark it against other vector databases.
Qdrant provides the dense retrieval component. A separate local BM25 retriever handles keyword matching, and the application combines their rankings using Reciprocal Rank Fusion.
Dense retrieval helps with conceptual similarity. BM25 helps with exact terminology, project names, and document identifiers. Combining the two gives the system complementary ways to find evidence before synthesis and answer generation.
The system is designed to run locally with Ollama and Qdrant, making it suitable for document sets that should remain inside an organization.
The high-level workflow is:
Here is Deatailed archietcture: User Query │ ▼ ┌─────────────────┐ │ Planner Agent │◄──────────────────────────────────┐ └────────┬────────┘ │ ┌─────────────────┼──────────────────┐ │ (direct_answer) (out_of_scope) (rag_retrieval) │ │ │ │ │ END │ ▼ │ │ ┌───────────────────────┐ │ │ │ Orchestrator Agent │ │ │ └───────────┬───────────┘ │ │ │ Send() API │ │ ┌────────────────┼────────────────┐ │ │ ▼ ▼ ▼ │ │ [Narrative] [Tabular] [MCP] │ │ (Manuals) (Tables) (Formulas) │ │ │ │ │ │ │ └────────────────┼────────────────┘ │ │ ▼ │ │ ┌───────────────────────┐ │ │ │ Aggregate Results │ │ │ └───────────┬───────────┘ │ │ ▼ │ │ ┌───────────────────────┐ │ │ │ Synthesizer Agent │ │ │ └───────────┬───────────┘ │ │ ▼ │ │ ┌───────────────────────┐ │ │ │ Retrieval Grader │ │ │ │ (Gate 1: Pre-Gen) │ │ │ └───────────┬───────────┘ │ │ │ │ │ (Insufficient │ │ │ & retry < max) │ (Sufficient or | | |Insufficient&retry>=max| │ ┌─────────┴─────────┐ │ │ │ ▼ │ │ │ ┌───────────────────┐ │ │ │ │ Generator Agent │ │ │ │ └─────────┬─────────┘ │ │ │ ▼ │ │ │ ┌───────────────────┐ │ │ │ │Generation Guardail│ │ │ │ │(Gate 2: Post-Gen) │ │ │ │ └─────────┬─────────┘ │ │ │ │ │ │ │ (Grounded) │ │ │ │ ┌───────────┴─────────┐ │ │ │ ▼ │ │ │ │ END │ │ │ │ (Ungrounded | | │ │ & retry < max)| | │ ▼ ▼ │ │ ┌───────────────────────────────────────┴┐ │ │ Feedback Agent │ │ │ (Diagnoses query gaps & missing terms)│ │ └────────────────────────────────────────┘ │ │ (Ungrounded & retry >= max) ▼ ┌────────────────────────────┐ │ Fallback Agent │ │ (Circuit Breaker) │ │ - Out-of-scope refusal │ │ - Partial answer + warning │ └─────────────┬──────────────┘ ▼ END
Each stage has a focused responsibility.
The Planner Agent first classifies the request into one of three routes:
For engineering questions, it elaborates domain terms and decomposes broad queries into focused sub-queries with targeted metadata filters (e.g., tagging a discharge inquiry with is_table: true).
The Orchestrator dispatches sub-queries to specialized workers using LangGraph’s dynamic Send() API:
You can add multiple workers as you want. For document retrieval, the system combines dense semantic search (Qdrant) and sparse keyword search (BM25) using Reciprocal Rank Fusion (RRF). Exact equipment designations, Indian Standards (IS codes), and plant names receive exact-match boosting.
The Synthesizer performs deterministic evidence assembly without an additional LLM call. It gathers worker outputs, removes duplicate chunks, balances evidence across sub-queries, and constructs a context dossier with source references for the generator. This stage still takes processing time, but avoids the inference overhead of another model call.
To improve answer reliability, the system implements two independent verification gates:
To make the system practical for plant engineers and operators, the user-facing interface is completely decoupled from the underlying agent execution engine.
┌────────────────────────────────────────────────────────┐ │ Streamlit Web Frontend (Port 8601) │ │ [Engineering Chat] [Hydro Calculator] [Citations] │ └───────────────────────────┬────────────────────────────┘ │ JSON REST HTTP Requests (session_id, query, health probes, uploads) │ ▼ ┌────────────────────────────────────────────────────────┐ │ FastAPI Server (Port 8600) │ │ - Pre-compiles LangGraph Singleton at startup │ │ - Manages session threads & MemorySaver checkpointing│ │ - Probes Ollama, Qdrant, and LangSmith health │ └───────────────────────────┬────────────────────────────┘ │ ▼ [Compiled LangGraph Multi-Agent Engine]
The system decouples the presentation layer from the agent execution engine and calculation services across three dedicated, non-conflicting service ports (8600–8602):
The frontend interacts with the agent engine across three primary endpoints:
The Streamlit interface organizes engineering workflows into focused tabs:
A core design goal of this project is that the RAG system can run fully on-premises without depending on external inference APIs.
At runtime, the architecture uses local components:
This means that plant manuals, DPRs, maintenance reports, drawings, and operational documents can remain within the organization’s own infrastructure.
External APIs such as Gemini and Groq were used only in the offline evaluation workflow as LLM-as-a-judge providers for scoring the golden dataset. They are not required for the runtime RAG pipeline.
For environments with stricter data policies, the evaluation layer can also be replaced with a local Ollama-based judge. The trade-off is that external judge models can provide an additional independent perspective during benchmarking, while a fully local evaluation setup preserves complete data isolation.
I used two evaluation layers: checks inside the running agent workflow, and an offline benchmark against a golden dataset. They answer different questions.
┌────────────────────────────────────────────────────────────────────────┐│ 1. Runtime In-Flight Evaluation (Zero External APIs) ││ - Gate 1: Retrieval Sufficiency Grader (qwen3.5:9b) ││ - Gate 2: Faithfulness & Numerical Guardrail (qwen3.5:9b, ≥ 0.90) ││ - Feedback Loop & Circuit Breaker (Bounded retry budget) │└────────────────────────────────────────────────────────────────────────┘ │ ▼┌────────────────────────────────────────────────────────────────────────┐│ 2. Offline Benchmark: Agentic RAG vs. Plain Hybrid-RAG ││ - 10 Head-to-Head Policy & Licensing Benchmark Cases ││ - Multi-provider LLM-as-a-Judge (Google Gemini 2.5 Flash & Groq) ││ - Evaluates: Faithfulness, Precision, Recall, Relevancy, Latency │└────────────────────────────────────────────────────────────────────────┘
During execution, the system uses the local qwen3.5:9b model as a judge at two stages.
Both checks share a bounded retry budget (cycle_count < MAX_CYCLES). When the budget is exhausted, insufficient retrieval proceeds to best-effort generation; a failed faithfulness check then routes to a fallback response with a warning.
These checks do not require a golden answer or an external inference API. However, an LLM judge is still fallible: passing the check is a probabilistic model assessment, not a mathematical proof of factual truth
For empirical benchmarking, I evaluated the updated Agentic RAG workflow against a standard Plain Hybrid-RAG baseline (single dense vector search + BM25 keyword search without agents or decomposition) across 10 complex Small Hydropower policy and licensing questions from the golden dataset.
Here, “offline” means evaluation outside the live request workflow — not disconnected from the internet. API-based judging sends the evaluation content to those providers. A fully local evaluation setup can instead use an Ollama-based judge. Evaluations were performed using automated LLM-as-a-Judge scoring via Google Gemini 2.5 Flash with automated failover to Groq (openai/gpt-oss-120b).
The first four metrics are LLM-assigned scores on a 0–1 scale in this benchmark. They should be interpreted as judge estimates, rather than exact measurements of factual accuracy.
Hallucination rate is reported separately from average faithfulness; it is not calculated as 1 − faithfulness. Latency is measured from execution, with mean and median used to compare the two pipelines.
Together, these measures help distinguish a grounded but incomplete answer from a comprehensive answer containing unsupported claims — and show how much processing time each architecture requires.
The updated head-to-head comparison covers the same ten policy/licensing questions for both systems. It does not establish performance across all engineering tasks.
The benchmark also uses multiple judge models through provider fallback, which can introduce scoring variation. A stronger follow-up would use a fixed judge configuration, repeated runs, and human review of difficult cases, especially calculations, tables, and conflicting evidence.
I compared the updated Agentic RAG workflow with a plain hybrid-RAG baseline across 10 policy and licensing questions.
The result surprised me in a useful way.
That is the central lesson from this project:
More agents do not automatically mean more reliable answers.
They add value when they improve retrieval quality, tool selection, and safe failure behavior. But every extra reasoning step adds latency, complexity, and more opportunities for error.
This project also taught me where Agentic RAG remains difficult.
An earlier version of the numerical guardrail incorrectly flagged heading labels such as 3.1 and 4.3 as unsupported engineering claims. That created false failures and unnecessary fallback responses. The fix was not simply “use another LLM.” It required better structure-aware preprocessing before evaluating numbers.
Multiple LLM calls, retrieval operations, validation checks, and correction loops add processing overhead. In my local benchmark, this made the agentic workflow substantially slower than a single retrieval-and-generation pipeline.
This latency is not necessarily a fixed limitation. More capable GPUs — either on-premises or cloud-hosted — and an optimized inference engine such as vLLM could reduce model-serving overhead. Prefix caching can avoid reprocessing shared prompt prefixes, while efficient batching can improve throughput when serving concurrent requests. However, higher throughput does not automatically mean a faster answer for an individual user. vLLM optimization documentation
In one benchmark question, the generated answer was grounded in the retrieved content, but the context lacked the expected preference criteria. The system was faithful, but incomplete. This distinction is important:
A trustworthy system needs both.
Future improvements should focus on practical reliability, speed, and testing, rather than merely adding more agents.
The 3.1× latency penalty observed during local testing is a solvable engineering challenge:
Not every question wants a multi-agent swarm:
Building Agentic RAG is not about making a chatbot appear more intelligent.
It is about deciding when to retrieve, when to calculate, when to retry, when to abstain, and how to show the user what evidence supports an answer.
My current system is not a finished production engineering advisor. It is an evaluated local-first prototype that demonstrates both the promise and the trade-offs of Agentic RAG for domain-specific technical knowledge.
The biggest result was not that Agentic RAG “beat” plain RAG.
It was learning exactly where it helped: retrieval precision and structured workflows — and where it still needs work: latency, recall, testing, and reliable engineering-grade verification.
Agentic RAG vs. Plain Hybrid RAG: What My Hydropower Benchmark Revealed was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.