Corrective RAG for billing: the bug is not retrieval, it's the model narrating correct numbers wrong

An engineer at an ISP built a WhatsApp billing assistant using Corrective RAG (CRAG) and found that the main challenge was not retrieval but the model narrating correct numbers incorrectly. The system uses hybrid retrieval with pgvector and Postgres full-text search fused via Reciprocal Rank Fusion, then prunes irrelevant chunks before generation. A grounding verifier and forbidden-phrase detector escalate ungrounded or risky responses, ensuring accuracy for billing queries where users hold the invoice.

Most RAG demos are graded by an audience that cannot check the answer. Ask a docs bot something, get a fluent paragraph back, nobody in the room knows whether sentence three is invented. The demo lands. Billing is not like that. When a subscriber asks "why is my bill higher this month" they are holding the invoice. They will check. We built a WhatsApp assistant for an ISP that answers plan and billing questions. This post is about the two things that turned out to matter, neither of which is "we used CRAG": Code is from the real system. Comments translated from Spanish, otherwise untouched. inbound message ↓ state gate ──────────────► human agent active? bot writes nothing ↓ hybrid retrieval ├─ pgvector cosine ─┐ └─ Postgres FTS ────┴─► RRF fusion ─► CRAG prune ↓ capability call billing API ─► circuit breaker ↓ no context AND no tool data? ──────────► escalate before spending a token ↓ LLM, structured output ↓ grounding verifier ────────► not grounded? ─► escalate ↓ humanizer ─────────────────► forbidden phrase? ─► escalate ↓ send Every arrow pointing right is a refusal path. That is the design. Vector similarity and Postgres ts rank live on different scales. Adding them is meaningless. Reciprocal Rank Fusion only looks at position , so it fuses them without pretending the scores are commensurable: DEFAULT RRF K = 60 """Smoothing constant. Damps the weight of the top positions so a single ranking cannot dominate the fusion. 60 is the value from the original paper.""" def reciprocal rank fusion rankings: Sequence Sequence str , , k: int = DEFAULT RRF K - dict str, float : scores: dict str, float = {} for ranking in rankings: for position, id in enumerate ranking, start=1 : scores id = scores.get id , 0.0 + 1.0 / k + position return scores Ties break on the id, deterministically. A result that reorders between identical runs makes tests flaky and debugging impossible. Why full-text at all, when you have embeddings? Because account numbers, tax IDs and national ID numbers have no semantic meaning . An embedding cannot score digits. FTS is the only path by which those are ever retrieved. CRAG https://arxiv.org/abs/2401.15884 Yan et al., 2024 evaluates retrieval relevance and drops the noise before generating. The paper uses a separate lightweight evaluator model. We skipped the evaluator. The retriever already computed cosine similarity and knows whether full-text hit — running a per-chunk model call every turn is the expensive part of CRAG and the least useful part when you already have scores. The problem it solves is concrete: top k asks for 6 chunks and RRF always returns 6, whether or not the sixth makes sense. Fusion orders, it does not filter. That filler enters the prompt at full size, burns tokens every turn, and dilutes the good evidence. DEFAULT MARGIN = 0.12 """How far below the best chunk we still tolerate. Deliberately a relative margin. A fixed threshold behaves badly at both ends: on a well-answered query best = 0.80 it lets a 0.40 through as "good" when next to the best it's noise, and on a weak query best = 0.42 it would prune everything. The margin adapts: each chunk is compared to the best found on this turn.""" DEFAULT MIN CHUNKS = 2 def prune irrelevant Chunk: Scorable chunks: Sequence Chunk , , margin: float = DEFAULT MARGIN, min chunks: int = DEFAULT MIN CHUNKS, - list Chunk : if len chunks <= min chunks: return list chunks best = max c.vector score or 0.0 for c in chunks , default=0.0 cutoff = best - margin kept = {i for i, c in enumerate chunks if is relevant c, cutoff } Chunks arrive ordered best-to-worst, so backfilling from index 0 adds precisely the best of the discarded ones. for i in range len chunks : if len kept = min chunks: break kept.add i return c for i, c in enumerate chunks if i in kept def is relevant chunk: Scorable, cutoff: float - bool: """A full-text hit always survives; everything else competes on similarity. Full-text immunity is not a convenient exception: plainto tsquery requires the terms to actually appear in the chunk, so the hit is itself proof of relevance. It is also the only path by which account numbers and tax IDs are retrieved — digits carry no semantic meaning, so they arrive here with a low cosine that would otherwise condemn them. """ if chunk.matched by text: return True return chunk.vector score or 0.0 = cutoff Note the bias, because it is the opposite of the verifier below: when in doubt, keep. Over-pruning would delete the one line that answered the question and escalate a case the bot could have solved — worse than wasting a few tokens. The real balance is not in any document. It comes from the billing API, and it has to count as citable evidence or the verifier in step 4 would reject the correct answer: @dataclass frozen=True, slots=True class ToolResult: capability: str ok: bool evidence text: str | None = None """Text ready to pass to the LLM as evidence; None if there was no data.""" data: dict str, Any | None = None error: str | None = None should escalate: bool = False """ True if the failure is infrastructure billing API down → escalate. A "customer not found" does NOT escalate: it's data that simply isn't there.""" That last field is a distinction worth stealing. "The API is down" and "this customer doesn't exist" both produce no data, and they are not the same event. One is an outage, the other is an answer. except CrmBusinessError as exc: The API works and said "doesn't exist": missing data, not a hard escalation. return ToolResult capability=capability.name, ok=False, error=exc.message, should escalate=False except CRMError as exc: Infrastructure: timeout, 5xx, open circuit → escalate. return ToolResult capability=capability.name, ok=False, error=str exc , should escalate=True When the API really is down, retrying makes it worse — every inbound message burns 15s of timeout × 3 attempts, the worker queue backs up, and the API gets a stampede exactly while it is trying to recover. So there is a circuit breaker, and for the bot "API down" simply translates to escalate fast : php def record failure self - None: """A failure counts; in HALF OPEN it reopens immediately. Reopening on the first failure in HALF OPEN is deliberate: if the probe request fails, the API is still bad and there is no point spending the rest of the threshold to confirm it. """ self. failures += 1 if self. state is CircuitState.HALF OPEN or self. failures = self. threshold: self. state = CircuitState.OPEN self. opened at = self. now The clock is injected so tests can verify reopening without actually sleeping. A test that does sleep 60 gets deleted or marked slow, and the most delicate part of the system loses its coverage with it. Everything above is retrieval hygiene. This is the piece I would push hardest on in a review of anyone else's billing assistant, because it survives every model upgrade . The prompt asks the model not to invent. This verifies it: python class GroundingVerifier: def verify self, , answer: str, used chunk ids: list str , retrieved chunk ids: list str , evidence texts: list str , tool result texts: list str | None = None, - GroundingResult: Two checks: The bias is deliberate and the inverse of the pruner: when in doubt, reject. A rejected answer escalates to a human, which is annoying but safe. A hallucination that ships is expensive and costs trust. That module is held to 100% test coverage. Naive version: strip non-digits from the number, check if those digits appear in the evidence. That is wrong, and here is the counterexample that killed it: Evidence from the billing API : 45.00 Model says: 4.500 Digits of both: "4500" ✓ passes Two different amounts. Same digits. In Venezuelan notation 4.500 reads as four thousand five hundred; 45.00 is forty-five. Digit equality green-lights a hallucination that changes the magnitude of someone's bill by two orders of magnitude. So amounts are compared by canonical value , not by digits: AMOUNT = re.compile r"^\d{1,3} ?: .,\s \d{3} ., \d{1,2}$|^\d+ ., \d{1,2}$" def amount value token: str - str | None: """Normalise a currency token to canonical integer.decimals . Returns None if the token is not amount-shaped. This is what stops 45.00 and 4.500 being considered equal. Treats the LAST . / , as the decimal separator and the rest as thousands, which is the Venezuelan convention 1.234,56 and also the English one 1,234.56 . """ if not AMOUNT.match token : return None clean = token.replace " ", "" cut = max clean.rfind "." , clean.rfind "," integer = re.sub r"\D", "", clean :cut decimals = clean cut + 1: return f"{int integer or 0 }.{decimals}" Three rules, strictest first: php def number supported token, evidence, evidence digits, evidence amounts - bool: 1. Literal, with digit boundaries. 12345678 is NOT accepted just because it appears inside a longer 123456789 . if re.search r" ?< \d " + re.escape token + r" ? \d ", evidence : return True 2. Amount value. 45,00 Venezuelan decimal comma matches 45.00 from the API, but 4.500 does NOT match 45.00 — different values. value = amount value token if value is not None: return value in evidence amounts 3. Digit equality — ONLY for identifiers phones, accounts, tax IDs: 7+ digits , where separators are cosmetic and don't change the value. if len digits only token = MIN IDENTIFIER DIGITS: return digits only token in evidence digits return False Rule 3 is deliberately narrow. A short number that matched neither literally nor as an amount is rejected — 4.500 must not pass just because it shares digits with 45.00 . The instinct is to treat escalations as the failure rate and drive them to zero. That optimises the wrong thing. Every escalation you remove by loosening the verifier becomes a confident answer about a bill, and some of those are wrong in a way the customer notices. Each reason carries a summary the human agent reads, so they don't re-read the whole thread: AGENT SUMMARY: dict EscalationReason, str = { EscalationReason.NO CONTEXT: "The customer asked something that is not in the knowledge base.", EscalationReason.UNGROUNDED OUTPUT: "The bot was about to give an unsupported figure; it was stopped for safety.", EscalationReason.USER REQUEST: "The customer asked to speak to a person.", EscalationReason.TOOL ERROR: "Could not query the billing system possible outage .", EscalationReason.CAPABILITY MISSING: "The customer wants data the bot cannot query yet.", } USER REQUEST fires unconditionally. There is always pressure to have the bot try once more before releasing the conversation — it improves your containment metric. It also means the subscriber who typed "I want to talk to a human" gets a paragraph they didn't ask for. On a channel people use to talk to people, ignoring that request teaches them the bot is an obstacle, not a front door. One honest note on scope: this is one system for one operator, described as architecture rather than as a benchmark. We are not going to show you a deflection-rate chart from a sample of one. Also worth stating plainly: the billing API integration is contract-first . There is a documented HTTP contract and a simulator implementing it; the production system on the other side is still being built. Everything above — capability resolution, circuit breaker, tool-results-as-evidence — runs against that contract. We think that is the right order the contract is the thing both sides agree on , but you should know it when you read the code. Longer version, including why WhatsApp specifically removes every UI affordance you'd normally use to show uncertainty: Corrective RAG for Billing Questions on WhatsApp https://softronic.dev/blog/crag-whatsapp-billing-assistant