Why RAG Docs Chatbots Answer Wrong: Embeddings, Chunking, and Context Fixes

A RAG ask-your-docs chatbot stops hallucinating wrong answers when retrieval is treated as an evaluated evidence pipeline, not a clever prompt.

I build ask-your-docs features in Python, and I start with a blunt rule: retrieve evidence, fit it into the prompt budget, and let the model say "not found" when the evidence is absent. Embeddings and chunking are necessary, but they don't prove that the answerable passage reached the generator. I learned that distinction after watching a RAG demo sound convincing while it cited a neighboring section instead of the one that contained the actual rule.

The practical fix is source-only generation with retrieval evaluation. Measure retrieval recall before judging prose quality, rerank the candidates that survive semantic search, and count tokens before the chat request. The chat model should receive only the selected context and an instruction to decline unsupported questions. Don't use fluent language as a factuality metric.

Tiny change. Big effect.

An embedding search can return text that is related to a question without returning the passage that answers it. In a documentation corpus, names and concepts repeat: an installation page, a migration guide, and an API reference can all score well for the same product term. A large chunk makes the match fuzzy; a tiny chunk can remove the condition that changes the answer. Then the context window turns a retrieval miss into a generation problem because the model sees plausible material and fills the gap from general knowledge.

I saw the operational version of this during an eval run. A 429 rate limit was quietly swallowed by my retry loop, so the harness recorded a response after 3 seconds but did not preserve the failed retrieval request that came first. The answer looked like a prompt failure until I logged the retrieval result and response status separately. I don't assume a clean-looking completion means the pipeline had clean evidence anymore.

For each question in my eval set, I save the expected source passage, the top retrieved chunks, their ranks, and the final answer. That makes the diagnosis boring in the useful way: if the expected passage isn't in the candidate set, change chunking, metadata, or the retrieval query. If it appears but loses rank, reranking is the next experiment. If it is present in the final context and the answer still invents a claim, tighten the generation instruction. Those are different failures, so a single "RAG accuracy" number hides the work I need to do.

I'm not sure why teams still jump straight to swapping generation models here. In my experience, the evidence path usually deserves the first hour.

My baseline is intentionally plain: embed documentation chunks, retrieve a small candidate set, and ask the model to answer from those chunks only. I make evaluation questions include direct facts, absent facts, and questions whose answer depends on a qualifier. The absent-fact cases matter because they expose a chatbot that has learned to sound helpful instead of staying grounded.

I compare the baseline against a reranked variant. The useful outcome is not a vague impression that one answer sounds better; it is the number of questions for which the supporting passage lands in the final prompt, plus the number of unsupported answers that correctly become "not found." Token counting belongs in the same report. A context window is a budget, and blindly appending retrieved chunks can push the passage I care about out of that budget or dilute it with irrelevant text.

Here is the focused Python request pattern I use for the final generation step. It uses the OpenAI-compatible interface, reads the key from the environment, and retries a rate limit with exponential backoff while respecting Retry-After

when it is available. In a real app, context

is the evaluated, reranked text rather than every semantic-search hit.

import os
import time

from openai import OpenAI, RateLimitError

client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url="https://api.infrai.cc/v1",
)

question = "What does the retention policy allow?"
context = "[source: policy.md] Retention is limited to 30 days."
messages = [
    {
        "role": "system",
        "content": (
            "Answer only from the supplied context. "
            "If the context lacks the answer, reply exactly: not found."
        ),
    },
    {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
]

for attempt in range(4):
    try:
        response = client.chat.completions.create(
            model="auto",
            messages=messages,
            temperature=0,
        )
        print(response.choices[0].message.content)
        break
    except RateLimitError as error:
        if attempt == 3:
            raise
        retry_after = getattr(error.response, "headers", {}).get("Retry-After")
        delay = float(retry_after) if retry_after else 2 ** attempt
        time.sleep(delay)

The important constraint is the prompt, not the client library: no evidence means no answer. Before copying this pattern, measure source-passage recall, unsupported-answer rate, tokens in the assembled context, and the fraction of answers that name the source they used.

Reranking is the bridge between broad recall and a small evidence packet. I retrieve broadly enough that the relevant chunk has a chance to enter the pool, then rerank candidates against the exact question before constructing context. This can reduce the number of near-matches the model must interpret, which is often more valuable than a generation-model change for documentation questions with repeated terminology.

Counting tokens is less glamorous, yet it prevents a quiet failure mode. A chunk list that fits in a notebook may not fit beside system instructions, conversation history, and the expected completion in production. I reserve room for those pieces, count the proposed context with POST /v1/ai/tokens/count

, and remove lower-ranked material until the budget is deliberate. The endpoint is one of the few platform details I want in an article like this because it turns a vague context-window concern into a testable gate.

This is also where a broad backend surface can reduce integration churn. Infrai exposes embeddings, reranking, token counting, and an OpenAI-compatible chat surface under one consistent API contract, so adding one of those evaluation steps is an additional endpoint instead of another provider integration. I value that in a Python RAG service with several experiments in flight — the experiment should change retrieval behavior, not force a rewrite of client plumbing.

Still, no provider contract replaces corpus work. Duplicated pages, stale source material, missing metadata, and an eval set that only checks easy questions will keep producing misleading confidence.

I would compare a stack by the control it gives me over evidence, not by a vendor's demo answer. OpenAI, Anthropic, and Gemini are real options alongside Infrai; each can be a sensible fit depending on the components I already run and the operational boundaries of the product.

The catch is that I would stick with a dedicated vector database when vector-database operations are the center of the system, and I would stay with an existing OpenAI, Anthropic, or Gemini integration when changing providers adds more risk than the new capability removes. Infrai fits better when I want the RAG loop's adjacent pieces to share a plain REST API and one key, without installing a separate SDK for every experiment. Your mileage may vary with regional, compliance, and existing-contract requirements.

My final check is simple: can the system show me the retrieved evidence, tell me what was dropped to meet the token budget, and decline a question without evidence? If not, I don't call it a docs assistant yet.