# What It Takes to Make a RAG Assistant Work > Source: > Published: 2026-10-21 19:01:01+00:00 “*Who can authorize an Atlas Analytics pipeline rollback?*” In an earlier live check of the knowledge assistant I built, the response was straightforward: *Only the incident commander may authorize an Atlas Analytics pipeline rollback.* Underneath it, the application displayed a supporting quote and the document section it came from. I could expand the passage and compare the answer with the evidence. That was a useful milestone. It showed that the deployed application could retrieve a relevant passage, call the model, and return a response with a valid source reference. It didn't establish how reliably the assistant would behave on other questions. In my [**previous article**](https://pub.towardsai.net/building-a-knowledge-assistant-with-rag-and-python-4ccee61062a5), I described a Python RAG architecture for operational documents. This follow-up examines the working implementation, including the choices needed to make it inspectable, what the initial tests actually checked, and what deployment still leaves unresolved. The [**source code is available on GitHub**](https://github.com/anwarsan/rag-knowledge-assistant), alongside a [** hosted Streamlit demo**](https://knowledge-assistant-ai.streamlit.app/). All included procedures are synthetic. Atlas Analytics is a fictional service, and the application has not been deployed for a real support team. The use case focuses on helping a support team find procedures across operational documentation. The corpus includes incident escalation, stakeholder communications, database recovery guidance, and an Atlas Analytics SOP. The Atlas document defines who can approve a rollback, when to escalate, and how to verify recovery. It also deliberately omits information such as a monthly uptime SLA and on-call phone numbers. That gives the prototype both answerable and unanswerable questions. A broad chatbot can impress with fluent responses. This assistant must also let readers verify where an answer came from and avoid filling gaps with plausible operational advice. The initial hosted check loaded four active documents into 19 passages. That collection is sufficient to exercise the workflow, but far too small to establish performance on a large documentation estate. The implementation separates ingestion and indexing from question answering. During preparation, the application selects active documents, reads their text, creates chunks, attaches source metadata, and generates embeddings for storage in Chroma. During question answering, it retrieves candidate passages, combines the retrieval results, assembles evidence, and requests a structured response from the model. This distinction also appears in the interface. The user checks or builds the vector index before asking questions. When the same complete index remains available, subsequent questions reuse it instead of re-embedding the documents. Each RAG question still requires API calls to embed the query and generate an answer. Reusing document embeddings does not make question answering free. The project provides an **Evidence preview** mode as well. It uses local BM25 keyword search and displays passages without calling a model. That makes retrieval easier to inspect independently of generation and allows readers to explore the corpus without an API key. The original conceptual walkthrough used a simulated document reader. The runnable application needs actual file handling and meaningful source locations. For Markdown, the loader splits around headings before applying character windows within each section. For PDFs, it extracts text page by page and retains the actual page number. It rejects empty or unreadable PDF pages so that an extraction failure does not silently become an incomplete knowledge base. Each chunk carries a document identifier, source path, version, location, and stable content-derived ID. This metadata lets the interface show more than a quote floating beneath an answer. For the rollback example, the useful reference is the Atlas SOP, version 1.0, under “Recovery approval and verification.” That is something a reader can inspect. The current implementation uses windows of up to 1,800 characters with 200-character overlap. These are starting choices for the demo, not optimized settings. Character boundaries can still split a relationship or condition inside a section. The Markdown parser is intentionally simple, and PDF extraction doesn't preserve every table or layout. Even a well-written prompt cannot compensate for important information lost during document extraction or chunking. The repository includes both current and archived incident procedures. Searching everything indiscriminately could return an obsolete instruction that looks relevant to the question. The application uses a manifest to select active documents. An entry looks like this: ``` { "document_id": "atlas-operations", "version": "1.0", "status": "active", "path": "documents/atlas-analytics-sop.md"} ``` Archived entries are excluded before indexing, and the loader rejects multiple active versions for the same document ID. This simple governance rule keeps archived procedures out of the evidence supplied to the model through this ingestion path. The vector collection is also selected through a fingerprint of the corpus content, chunk metadata, and embedding model. Changing those inputs selects a different collection instead of silently searching the previous corpus snapshot. There are limits. Someone still has to approve the documents and maintain their status. A fingerprint cannot tell whether a procedure is correct, and old collections remain on disk until removed. These mechanisms address version selection and index reuse, they don't replace document ownership or authorization. The hybrid retriever combines two rankings: They can complement each other. A specific service name or procedure identifier may benefit from lexical matching, while a paraphrased question may benefit from semantic similarity. The implementation combines the rankings using reciprocal rank fusion. Each passage receives contributions based on its position in each list. This avoids directly treating a BM25 score as comparable to a vector distance. The default interface returns four passages. Internally, the hybrid retriever requests larger candidate lists before fusion. The retrieval count and fusion constant are configuration choices to evaluate, not evidence of optimality. There is no separate cross-encoder reranker in this prototype. The earlier architectural discussion included reranking as an option, the runnable repository implements a smaller subset. I would add another retrieval stage only after examining failures. If a question retrieves the wrong document version, fix version selection. If it misses an exact identifier, inspect lexical matching. If relevant passages consistently rank below irrelevant ones, a reranker becomes a candidate worth testing. The application requests a structured answer through the OpenAI Responses API. Its Pydantic schema contains an answer, an evidence-sufficiency flag, and citations with source IDs and quotes. The model is instructed to use only the supplied evidence, treat document content as untrusted, and abstain when evidence is insufficient or procedures conflict. The application then checks the response. It withholds an answer if the structured result is missing, a claimed supported answer has no citations, a source ID is unknown, or a quote does not appear in its referenced passage. These checks reject invented document references rather than accepting them as valid citations. However, source validity and claim support are different questions. A model could return a genuine quote and still draw an unsupported conclusion from it. It could omit an important condition or misunderstand a time interval. Structured outputs constrain format; they don't establish the truth of the answer. OpenAI’s [**structured outputs documentation**](https://developers.openai.com/api/docs/guides/structured-outputs) also distinguishes schema adherence from the correctness of generated content. The interface therefore exposes the underlying passages for review. That is useful even when validation passes. This limitation appeared in a later live run of the rollback question. The answer correctly identified the incident commander, but then described authorization as occurring after recovery confirmation. The source identifies the authorizing role and describes recovery verification; it does not support that added sequence. The returned quote was genuine, so the reference checks passed. The interface label “Supported by cited passages” therefore overstates what those checks establish: the answer still needs a claim-by-claim comparison with the source. The initial implementation passed 17 automated tests. They covered document-version selection, chunk metadata, PDF extraction, index persistence, hybrid retrieval wiring, response validation, and the Streamlit evidence-preview flow. Some tests use synthetic vectors and mocked model responses. They check implementation behavior without paid API calls. They don't measure the model’s semantic understanding or establish reliable abstention in live use. The original five-case BM25 evaluation found the expected document IDs within the top four results for each case. That is a coarse retrieval check on a tiny synthetic corpus. Finding the correct document does not prove that all necessary passages were retrieved. For manual testing, I use questions with explicit expected behaviors: These are test cases, not a table of measured live results. An earlier hosted check returned the correct role; a later run exposed the chronology error discussed above. These observations do not establish accuracy across the set. The remaining cases provide a repeatable review set rather than a claim of universal correctness. For the timing question, the Atlas SOP requires an initial stakeholder update within 10 minutes of declaration, then updates every 20 minutes after that initial update. An answer that simply says “every 20 minutes” loses part of the procedure, even if its citation is valid. Deploying on Streamlit introduced decisions that were easy to leave implicit in a local script. **Credentials belong in the hosting configuration.** The app reads an API key from environment variables or Streamlit Secrets. The actual key is not committed to GitHub. Streamlit documents this separation in its **secrets-management guide****.** **An API key and available API credit are separate requirements.** In the initial setup, a key was configured before the account had usable credit. A deployment can load its interface successfully while paid model requests remain unavailable. **A public repository and an accessible app are separate checks.** After the repository became public, a signed-out browser still encountered an access page for the Streamlit deployment. An authenticated check confirmed that the app worked. Streamlit’s [sharing settings](https://docs.streamlit.io/deploy/streamlit-community-cloud/share-your-app) control who can view the hosted application, so opening the repository alone was not sufficient evidence of public app access. **Local persistence has a lifecycle.** Chroma stores this demo’s index on local disk. If those files are lost during hosting lifecycle events, the current collection must be rebuilt. A build button checks readiness and reuses a complete index when present. It doesn't need to recreate embeddings for every question. For broader sharing, the owner’s API key also means the owner’s account pays for visitor requests. The prototype doesn't yet implement request limits or an application-level spending control. The prototype now has a working path from document ingestion to a cited response. It also exposes enough of that path to investigate failures. It doesn't yet provide application-level authentication, per-user document permissions, a durable hosted vector service, or production monitoring. A prompt that treats retrieved text as untrusted is only one part of defending against document-based prompt injection. My next priority would be a larger, human-reviewed evaluation set, including realistic paraphrases, incomplete procedures, conflicting documents, and questions that require multiple conditions to be preserved. That would make it possible to decide whether chunking, retrieval, or generation deserves the next change. To explore the current implementation, open the [**GitHub repository**](https://github.com/anwarsan/rag-knowledge-assistant) or the [** Streamlit deployment**](https://knowledge-assistant-ai.streamlit.app/). In Hybrid RAG mode, check the index, ask the rollback question, and inspect the supporting passage. Then ask for an SLA that the document never specifies. Compare what the application says with what its sources actually establish. That comparison is the most useful next test of whether the assistant works. [What It Takes to Make a RAG Assistant Work](https://pub.towardsai.net/what-it-takes-to-make-a-rag-assistant-work-df494ebc2c78) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.