{"slug": "your-llm-trace-is-green-why-is-the-rag-answer-still-wrong", "title": "Your LLM Trace Is Green. Why Is the RAG Answer Still Wrong?", "summary": "A developer argues that model-centric observability is insufficient for debugging retrieval-augmented generation (RAG) systems, because a green LLM trace can still hide a wrong answer caused by upstream retrieval failures. The post outlines a complete evidence path—from query rewrite to evidence selection—and recommends tracing search decisions, query rewrites, retrieval metadata, and claim-to-evidence mappings to diagnose issues. It notes that emerging OpenTelemetry conventions for generative AI and LangSmith's retriever tracing provide a starting vocabulary but remain incomplete.", "body_md": "A user asks your AI assistant whether a product still supports a particular feature. The assistant responds confidently and links to the company’s documentation.\n\nThe model request succeeded. Latency was normal. Token usage stayed within budget. No tool call failed. Every indicator on the dashboard is green.\n\nThe answer is also six months out of date.\n\nThe model trace cannot tell you whether the system searched for the wrong phrase, preferred an old page, discarded a better result, or ignored the correct evidence. It only shows the context that eventually reached the model.\n\nThat is the blind spot in model-centred observability. For RAG applications and web-connected agents, the useful unit of observation is not the model call. It is the complete evidence path.\n\nA typical LLM trace records the prompt, response, model name, token consumption, latency, errors, and perhaps a tool invocation. That is useful for diagnosing slow requests, malformed inputs, and unexpectedly expensive generations.\n\nIt does not tell you whether the model received the right facts.\n\nIn a retrieval application, the final prompt is assembled by an upstream system. That system may rewrite the query, choose a search provider, apply time or domain filters, fetch pages, extract text, remove duplicates, rerank candidates, and select passages for the context window.\n\nThe model can behave exactly as instructed and still produce a bad answer because one of those earlier decisions was wrong.\n\nThe current [OpenTelemetry semantic conventions for generative AI](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) recognise retrieval from a vector database or search system as a separate span. They include concepts such as the data source, top-k, retrieved document IDs, scores, and the effective retrieval query.\n\nThe conventions are still marked **Development**, and potentially sensitive fields such as query text and returned documents are opt-in. They are useful as a shared vocabulary, but they should not be treated as a finished internal schema.\n\n[LangSmith’s retriever tracing documentation](https://docs.langchain.com/langsmith/log-retriever-trace) takes a similar approach by representing retrieval as its own run and attaching document metadata such as source URLs, chunk IDs, and scores.\n\nThe important question is not which tracing platform you use. It is what your trace can explain when an answer goes wrong.\n\nApplication code may represent web retrieval as a single tool call. Operationally, it looks more like this:\n\n**Question → query rewrite → search → page fetch → extraction → deduplication → reranking → evidence selection → answer**\n\nEvery transition introduces a different failure mode.\n\nA query rewrite can remove an important date or product qualifier. Search can return relevant-looking but obsolete pages. Extraction can miss the paragraph containing the answer. Several URLs can reproduce the same story and create the illusion of independent confirmation.\n\nA reranker may favour semantic similarity over freshness. The context builder may then drop the best passage to stay inside its token budget.\n\n“Search completed successfully” tells you almost nothing about those decisions.\n\nA search-aware trace should retain enough information to reconstruct how evidence moved through the pipeline.\n\n| Stage | Useful trace data | \n|---|---|\n| Search decision | Reason for searching, selected source, freshness policy | \n| Query | Original question, rewritten query, filters, strategy version | \n| Retrieval | Provider, top-k, rank, score, duration, retry, cache status | \n| Evidence | Canonical URL, publication time, retrieval time, content hash, selected passage | \n| Answer | Claim ID, evidence ID, citation mapping, model and prompt version | \n\nOnly some of these fields are part of emerging telemetry conventions. Canonical URLs, content hashes, cache status, freshness policies, and claim-to-evidence mappings are application-level metadata that teams need to design themselves.\n\nThis does not mean storing every complete webpage forever. Metadata, content hashes, and the passages supplied to the model are often enough for diagnosis. Sensitive queries and retrieved content still require redaction, access controls, and a defined retention period.\n\nThe goal is not to log everything. It is to preserve enough information to explain an answer.\n\nMany applications place a few URLs beside an answer and call that provenance. A URL identifies a location, but it does not identify the exact information the model used.\n\nPages change. Search snippets can differ from subsequently fetched content. Two URLs may contain the same syndicated article. A page’s publication date may also differ from its most recent update.\n\nIf you only retain the URL, opening it tomorrow may not reproduce the evidence used today.\n\nA better evidence record connects four things:\n\n**The source, the retrieved version, the selected passage, and the claim it supports.**\n\nThat normally requires a canonical URL, retrieval timestamp, publication date when available, content hash, and the passage sent to the model. A stable evidence ID can then follow that passage through reranking, context construction, and generation.\n\nThis becomes even more important in multi-agent systems. A shared URL cache may stop four agents from fetching the same page, but it does not demonstrate that they consumed the same version or selected the same passage.\n\nContent hashes and evidence IDs make that relationship visible. They also stop duplicated content from inflating confidence. Five URLs do not represent five independent sources when they all reproduce one original report.\n\nReturn to the outdated product answer.\n\nStart with the effective query. If the user asked about current support but the rewritten query dropped the word “current,” the problem began before retrieval.\n\nNext, inspect the returned sources. If the latest documentation never appeared, the trace points towards a discovery problem involving the query, filters, or search provider.\n\nIf the current page appeared but ranked below an older one, look at reranking and freshness weighting. If it survived reranking but its relevant paragraph was never selected, the problem is more likely to be extraction or context construction.\n\nOnly when the correct evidence reached the model and the answer still contradicted it does the incident begin to resemble a generation failure. Even then, context ordering, truncation, or conflicting evidence may still be involved.\n\nCitation failure is another category. An answer can be factually correct while citing a page that does not support the relevant claim. A generally related link is not evidence unless the cited passage actually supports what the answer says.\n\nThese distinctions matter because each problem has a different fix. A new system prompt cannot recover a source that was never retrieved. Increasing top-k may make duplicate evidence worse. Switching to a more expensive model will not make stale documentation current.\n\nA useful trace turns “the answer was wrong” into a testable hypothesis about a specific stage.\n\nTraces are useful for debugging individual incidents, but their larger value appears when you aggregate them.\n\nA team might define retrieval success rate as the percentage of runs in which at least one selected source meets its relevance threshold. Freshness-policy violation rate can measure how often the system uses evidence older than the question permits. Duplicate evidence ratio can show how many apparently different results collapse into one canonical or content-hash group.\n\nCitation coverage should represent the share of material claims connected to supporting evidence. It should not simply count links. An answer can contain five citations while leaving its most consequential statement unsupported.\n\nOne metric I would add is **cost per grounded answer**:\n\n`(search + extraction + reranking + generation cost) / answers that pass the grounding threshold`\n\nCost per model call is misleading when a cheap workflow repeatedly searches, retrieves duplicate pages, or generates responses that later fail verification.\n\nTraces should also feed evaluation rather than become an archive nobody examines. Failed production runs can become regression cases. After changing a query strategy, extraction rule, or reranker, the same cases can be run again to determine whether the evidence path improved.\n\n[MLflow’s current RAG evaluation documentation](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/rag/) separates retrieval relevance, groundedness, and sufficiency, and requires retrieval to appear explicitly in the trace. That separation matters because an end-to-end score alone cannot tell you what to fix.\n\nAutomated judges are imperfect, but they provide a useful loop: observe a failure, classify its likely stage, add it to an evaluation set, change one component, and measure the result.\n\nWhen an AI application produces a poor answer, the model is often the first component developers replace. They change prompts, increase context, or upgrade models without knowing whether generation caused the failure.\n\nFor a search-enabled RAG system, observability needs to begin earlier. Retrieval, reranking, and context selection should appear as separate nested steps under the same request.\n\nA production trace should be able to answer three questions:\n\n**What did the system ask? What evidence did it see? Why did that evidence become this answer?**\n\nUntil it can, you are monitoring your model—not observing your system.\n\nWhen a RAG answer is wrong, can your current trace tell whether the source was never found, ranked down, dropped from context, or ignored by the model? Which of those is hardest for your team to see today?", "url": "https://wpnews.pro/news/your-llm-trace-is-green-why-is-the-rag-answer-still-wrong", "canonical_source": "https://dev.to/cloudsway/your-llm-trace-is-green-why-is-the-rag-answer-still-wrong-41nk", "published_at": "2026-09-07 03:10:34+00:00", "updated_at": "2026-09-07 03:56:08.676471+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "mlops", "developer-tools"], "entities": ["OpenTelemetry", "LangSmith"], "alternates": {"html": "https://wpnews.pro/news/your-llm-trace-is-green-why-is-the-rag-answer-still-wrong", "markdown": "https://wpnews.pro/news/your-llm-trace-is-green-why-is-the-rag-answer-still-wrong.md", "text": "https://wpnews.pro/news/your-llm-trace-is-green-why-is-the-rag-answer-still-wrong.txt", "jsonld": "https://wpnews.pro/news/your-llm-trace-is-green-why-is-the-rag-answer-still-wrong.jsonld"}}