RAG Hallucination: 6 Ask-Your-Docs Invoice Traces Explaining Wrong Chatbot Answers in 2026

A developer outlines a trace-based approach to diagnosing wrong answers in ask-your-docs systems for supplier-invoice extraction, arguing that each answer should be treated as a traceable claim rather than a successful model call. The recommended setup records one trace per invoice with tenant-safe cost and retrieval attributes, requires an exact source span and page reference for accounting-critical fields, and returns an explicit abstention when retrieved evidence does not support a field. The writeup notes that better embeddings or larger context windows cannot fix a missing invoice page, ambiguous tenant policy, or a prompt that invites guessing.

Short answer: treat an ask-your-docs answer as a traceable claim, not a successful model call. For supplier-invoice extraction, the least complex useful setup records one trace per invoice, attaches tenant-safe cost and retrieval attributes, and refuses to fill a field when the retrieved evidence does not support it. Better embeddings or a larger context window cannot repair a missing invoice page, an ambiguous tenant policy, or a prompt that quietly asks the model to guess. Start with this decision table. Pick the smallest option that lets an operator connect a wrong field to its source evidence and its tenant cost. | Option | Pick this when | What it reveals | Main limit | |---|---|---|---| | Structured logs | One team owns a small pipeline | Inputs, selected passages, outcome, and cost units | Cross-step investigation becomes manual | | Correlated traces | Extraction spans OCR, retrieval, reranking, and generation | Where evidence disappeared or changed | Requires careful attribute design | | Offline evaluation set | You need release comparisons on known invoices | Regressions by field, document type, and tenant | Cannot explain a live incident alone | | Online sampled review | Production documents differ from the test set | Current failure shapes and abstention quality | Can miss rare tenant-specific cases | The practical answer is a combination: traces explain individual failures, while a small evaluation set tells you whether a change helped. Keep billing dimensions beside quality dimensions. Otherwise a globally healthy dashboard can hide one tenant that pays for repeated OCR, retrieves five irrelevant chunks, and receives an unsupported tax amount. A plausible passage is not necessarily evidence for the requested field. An invoice question such as "What is the payment due date?" may retrieve a tenant's payment policy, a supplier's standard terms, and the invoice footer. All three are semantically close. Only one may state the due date printed on this invoice. Chunk boundaries can remove the relationship that matters. The label "Total due" may be in one chunk while the amount, currency, and invoice identifier land in another. A bigger context window can carry both chunks, but it does not identify which amount belongs to which label. More room is storage, not judgment. Trace the claim. There is a quieter mismatch: extraction and question answering have different failure costs. Free-form prose can tolerate paraphrase. A payable amount cannot. For fields that trigger accounting work, require an exact source span and page reference; if either is absent, return an explicit abstention. That trade-off lowers apparent completion, but it makes errors inspectable. OWASP treats prompt injection, sensitive-information disclosure, excessive agency, and misinformation as distinct risks in LLM applications. That separation is useful here. A wrong invoice value may come from retrieval, generation, untrusted document text, or an action taken after extraction. One generic "hallucination" counter cannot tell those paths apart. Structured logs are enough when OCR, retrieval, and extraction run in one process and traffic is modest. Emit one completion event with a stable request ID. Include tenant id , invoice id , requested fields, evidence identifiers, abstention reason, latency, and normalized usage units. Do not log raw invoice text by default; invoices can contain names, addresses, bank details, and tax identifiers. This is the quick start. It has a ceiling. Once retries, queues, or separate workers appear, a completion log no longer proves which OCR output fed which retrieval attempt. Teams often compensate by searching timestamps. That works until concurrent retries interleave. Move to correlated traces when causal order matters, not when the log search becomes unbearable. Use one trace for the invoice job and one span for each meaningful transformation: ingest, text extraction, retrieval, evidence filtering, field generation, and validation. The diagram in words is simple: invoice bytes enter; page text comes out; candidates enter retrieval; cited passages come out; requested fields enter validation; supported values or abstentions leave. Consider a three-page invoice where page one names the supplier, page two contains line items, and page three states payment terms. Retrieval selects a tenant policy plus pages one and three. The trace should make that selection visible before generation, then show that due date cites page three while supplier name cites page one. If generation proposes a value with the policy document as evidence, validation abstains. If OCR retries page three, its extra usage stays attached to the same tenant and invoice. One path now explains quality, latency, and cost without storing the raw pages in telemetry. Record identifiers and measurements, not document bodies. A span can carry the tenant, document type, page count, candidate count, selected evidence IDs, retry count, usage units, and validation result. Keep high-cardinality identifiers available for investigation, but do not casually turn each identifier into a metric label. Here is a vendor-neutral TypeScript shape. The interfaces are deliberately small so the same event can go to logs or a tracing adapter. type ExtractedField = { name: string; value: string | null; evidenceIds: string ; status: "supported" | "abstained"; }; type ExtractionObservation = { traceId: string; tenantId: string; invoiceId: string; stage: "retrieve" | "generate" | "validate"; candidateCount: number; selectedEvidenceIds: string ; inputUnits: number; outputUnits: number; retryCount: number; fields: ExtractedField ; }; function observe event: ExtractionObservation : void { process.stdout.write ${JSON.stringify event }\n ; } The join matters. traceId connects stages, invoiceId connects the result to the business object, and tenantId makes cost allocation possible. Evidence IDs connect each field to retrieved material without copying sensitive text into telemetry. For cost visibility, aggregate usage units and retry counts by tenant and stage. Keep currency conversion outside this core event because commercial rates can change. This shows whether a tenant's spend is driven by long invoices, broad retrieval, verbose output, or retries without freezing an unstable price into application logic. Do not tune chunk size against memorable failures alone. Build a versioned set of representative invoices with expected fields, acceptable evidence spans, and cases that must abstain. Slice results by tenant, supplier layout, language, scan quality, and field. Then compare candidate changes on the same set. A useful result distinguishes three outcomes: correct value with supporting evidence, abstention, and unsupported value. Collapsing the last two into "incorrect" hides the operational difference. An abstention routes work to review. An unsupported value can enter an accounting system looking complete. Include adversarial document text in the set as well. Supplier files are untrusted input, and instructions embedded in a document should not override the extraction policy. The evaluation should verify that the system extracts invoice facts and ignores document-level attempts to redirect its behavior. OWASP's prompt-injection guidance provides the security rationale; your tenant policy supplies the exact expected behavior. The deepest implementation work belongs after generation. Validate every proposed field against selected evidence, and return a typed result that downstream code cannot mistake for a verified value. A model's confidence phrase is not evidence. Keep that boundary sharp. type Candidate = { field: string; value: string; evidenceId?: string }; type Evidence = { id: string; page: number; text: string }; type CheckedValue = | { status: "supported"; value: string; evidenceId: string; page: number } | { status: "abstained"; reason: "missing evidence" | "value not present" }; function checkCandidate candidate: Candidate, evidenceById: Map<string, Evidence , : CheckedValue { if candidate.evidenceId { return { status: "abstained", reason: "missing evidence" }; } const evidence = evidenceById.get candidate.evidenceId ; if evidence || evidence.text.includes candidate.value { return { status: "abstained", reason: "value not present" }; } return { status: "supported", value: candidate.value, evidenceId: evidence.id, page: evidence.page, }; } Exact containment is intentionally conservative and incomplete. Dates, decimal separators, currencies, and OCR substitutions need field-specific normalization. Add those rules one field at a time, test them against the evaluation set, and preserve both the original span and normalized value. Avoid a universal fuzzy-match threshold; the acceptable ambiguity for a supplier name is different from the acceptable ambiguity for a bank account or total due. Alert on symptoms an operator can act on: a tenant's abstention rate moves outside its reviewed baseline, retry volume rises at one stage, or unsupported values survive validation. A latency alert without the responsible stage sends people hunting. A global error-rate alert without a tenant dimension hides concentrated harm. Tracing does not make an answer true. It makes the route to that answer visible. Evaluation does not cover every supplier layout, and an exact evidence check can still accept the wrong occurrence of a repeated amount. Human review remains appropriate for high-impact fields and novel layouts. The final decision rule is compact: use logs while one event preserves causality; add traces when the pipeline crosses boundaries; require evidence-gated field results; compare changes on a versioned evaluation set; and allocate usage by tenant and stage. This turns "the chatbot hallucinated" into a specific, testable failure with an owner.