{"slug": "corrective-rag-for-billing-the-bug-is-not-retrieval-it-s-the-model-narrating", "title": "Corrective RAG for billing: the bug is not retrieval, it's the model narrating correct numbers wrong", "summary": "An engineer at an ISP built a WhatsApp billing assistant using Corrective RAG (CRAG) and found that the main challenge was not retrieval but the model narrating correct numbers incorrectly. The system uses hybrid retrieval with pgvector and Postgres full-text search fused via Reciprocal Rank Fusion, then prunes irrelevant chunks before generation. A grounding verifier and forbidden-phrase detector escalate ungrounded or risky responses, ensuring accuracy for billing queries where users hold the invoice.", "body_md": "Most RAG demos are graded by an audience that cannot check the answer. Ask a docs bot something, get a fluent paragraph back, nobody in the room knows whether sentence three is invented. The demo lands.\n\nBilling is not like that. When a subscriber asks *\"why is my bill higher this month\"* they are holding the invoice. They will check.\n\nWe built a WhatsApp assistant for an ISP that answers plan and billing questions. This post is about the two things that turned out to matter, neither of which is \"we used CRAG\":\n\nCode is from the real system. Comments translated from Spanish, otherwise untouched.\n\n```\ninbound message\n   ↓\nstate gate ──────────────► human agent active? bot writes nothing\n   ↓\nhybrid retrieval\n   ├─ pgvector cosine ─┐\n   └─ Postgres FTS ────┴─► RRF fusion ─► CRAG prune\n   ↓\ncapability call (billing API) ─► circuit breaker\n   ↓\nno context AND no tool data? ──────────► escalate (before spending a token)\n   ↓\nLLM, structured output\n   ↓\ngrounding verifier ────────► not grounded? ─► escalate\n   ↓\nhumanizer ─────────────────► forbidden phrase? ─► escalate\n   ↓\nsend\n```\n\nEvery arrow pointing right is a refusal path. That is the design.\n\nVector similarity and Postgres `ts_rank`\n\nlive on different scales. Adding them is meaningless. Reciprocal Rank Fusion only looks at *position*, so it fuses them without pretending the scores are commensurable:\n\n```\nDEFAULT_RRF_K = 60\n\"\"\"Smoothing constant. Damps the weight of the top positions so a single\nranking cannot dominate the fusion. 60 is the value from the original paper.\"\"\"\n\ndef reciprocal_rank_fusion(\n    rankings: Sequence[Sequence[str]], *, k: int = DEFAULT_RRF_K\n) -> dict[str, float]:\n    scores: dict[str, float] = {}\n    for ranking in rankings:\n        for position, id_ in enumerate(ranking, start=1):\n            scores[id_] = scores.get(id_, 0.0) + 1.0 / (k + position)\n    return scores\n```\n\nTies break on the id, deterministically. A result that reorders between identical runs makes tests flaky and debugging impossible.\n\nWhy full-text at all, when you have embeddings? Because **account numbers, tax IDs and national ID numbers have no semantic meaning**. An embedding cannot score digits. FTS is the only path by which those are ever retrieved.\n\n[CRAG](https://arxiv.org/abs/2401.15884) (Yan et al., 2024) evaluates retrieval relevance and drops the noise *before* generating. The paper uses a separate lightweight evaluator model.\n\nWe skipped the evaluator. The retriever already computed cosine similarity and knows whether full-text hit — running a per-chunk model call every turn is the expensive part of CRAG and the least useful part when you already have scores.\n\nThe problem it solves is concrete: `top_k`\n\nasks for 6 chunks and RRF always returns 6, whether or not the sixth makes sense. **Fusion orders, it does not filter.** That filler enters the prompt at full size, burns tokens every turn, and dilutes the good evidence.\n\n```\nDEFAULT_MARGIN = 0.12\n\"\"\"How far below the best chunk we still tolerate.\n\nDeliberately a *relative* margin. A fixed threshold behaves badly at both ends:\non a well-answered query (best = 0.80) it lets a 0.40 through as \"good\" when\nnext to the best it's noise, and on a weak query (best = 0.42) it would prune\neverything. The margin adapts: each chunk is compared to the best found on\n*this* turn.\"\"\"\n\nDEFAULT_MIN_CHUNKS = 2\n\ndef prune_irrelevant[Chunk: _Scorable](\n    chunks: Sequence[Chunk],\n    *,\n    margin: float = DEFAULT_MARGIN,\n    min_chunks: int = DEFAULT_MIN_CHUNKS,\n) -> list[Chunk]:\n    if len(chunks) <= min_chunks:\n        return list(chunks)\n\n    best = max((c.vector_score or 0.0 for c in chunks), default=0.0)\n    cutoff = best - margin\n\n    kept = {i for i, c in enumerate(chunks) if _is_relevant(c, cutoff)}\n\n    # Chunks arrive ordered best-to-worst, so backfilling from index 0\n    # adds precisely the best of the discarded ones.\n    for i in range(len(chunks)):\n        if len(kept) >= min_chunks:\n            break\n        kept.add(i)\n\n    return [c for i, c in enumerate(chunks) if i in kept]\n\ndef _is_relevant(chunk: _Scorable, cutoff: float) -> bool:\n    \"\"\"A full-text hit always survives; everything else competes on similarity.\n\n    Full-text immunity is not a convenient exception: `plainto_tsquery` requires\n    the terms to actually appear in the chunk, so the hit is itself proof of\n    relevance. It is also the only path by which account numbers and tax IDs are\n    retrieved — digits carry no semantic meaning, so they arrive here with a low\n    cosine that would otherwise condemn them.\n    \"\"\"\n    if chunk.matched_by_text:\n        return True\n    return (chunk.vector_score or 0.0) >= cutoff\n```\n\nNote the bias, because it is the opposite of the verifier below: **when in doubt, keep.** Over-pruning would delete the one line that answered the question and escalate a case the bot could have solved — worse than wasting a few tokens.\n\nThe real balance is not in any document. It comes from the billing API, and it has to count as citable evidence or the verifier in step 4 would reject the *correct* answer:\n\n```\n@dataclass(frozen=True, slots=True)\nclass ToolResult:\n    capability: str\n    ok: bool\n    evidence_text: str | None = None\n    \"\"\"Text ready to pass to the LLM as evidence; None if there was no data.\"\"\"\n\n    data: dict[str, Any] | None = None\n    error: str | None = None\n    should_escalate: bool = False\n    \"\"\"`True` if the failure is infrastructure (billing API down) → escalate.\n    A \"customer not found\" does NOT escalate: it's data that simply isn't there.\"\"\"\n```\n\nThat last field is a distinction worth stealing. \"The API is down\" and \"this customer doesn't exist\" both produce no data, and they are not the same event. One is an outage, the other is an answer.\n\n```\nexcept CrmBusinessError as exc:\n    # The API works and said \"doesn't exist\": missing data, not a hard escalation.\n    return ToolResult(capability=capability.name, ok=False,\n                      error=exc.message, should_escalate=False)\nexcept CRMError as exc:\n    # Infrastructure: timeout, 5xx, open circuit → escalate.\n    return ToolResult(capability=capability.name, ok=False,\n                      error=str(exc), should_escalate=True)\n```\n\nWhen the API really is down, retrying makes it worse — every inbound message burns 15s of timeout × 3 attempts, the worker queue backs up, and the API gets a stampede exactly while it is trying to recover. So there is a circuit breaker, and for the bot \"API down\" simply translates to *escalate fast*:\n\n``` php\ndef record_failure(self) -> None:\n    \"\"\"A failure counts; in HALF_OPEN it reopens immediately.\n\n    Reopening on the first failure in HALF_OPEN is deliberate: if the probe\n    request fails, the API is still bad and there is no point spending the\n    rest of the threshold to confirm it.\n    \"\"\"\n    self._failures += 1\n    if self._state is CircuitState.HALF_OPEN or self._failures >= self._threshold:\n        self._state = CircuitState.OPEN\n        self._opened_at = self._now()\n```\n\nThe clock is injected so tests can verify reopening without actually sleeping. A test that does `sleep(60)`\n\ngets deleted or marked slow, and the most delicate part of the system loses its coverage with it.\n\nEverything above is retrieval hygiene. This is the piece I would push hardest on in a review of anyone else's billing assistant, because **it survives every model upgrade**.\n\nThe prompt *asks* the model not to invent. This *verifies* it:\n\n``` python\nclass GroundingVerifier:\n    def verify(\n        self, *, answer: str, used_chunk_ids: list[str],\n        retrieved_chunk_ids: list[str], evidence_texts: list[str],\n        tool_result_texts: list[str] | None = None,\n    ) -> GroundingResult:\n```\n\nTwo checks:\n\nThe bias is deliberate and the inverse of the pruner: **when in doubt, reject.** A rejected answer escalates to a human, which is annoying but safe. A hallucination that ships is expensive and costs trust. That module is held to 100% test coverage.\n\nNaive version: strip non-digits from the number, check if those digits appear in the evidence.\n\nThat is wrong, and here is the counterexample that killed it:\n\n```\nEvidence (from the billing API):  45.00\nModel says:                       4.500\nDigits of both:                   \"4500\"  ✓ passes\n```\n\nTwo different amounts. Same digits. In Venezuelan notation `4.500`\n\nreads as four thousand five hundred; `45.00`\n\nis forty-five. Digit equality green-lights a hallucination that changes the magnitude of someone's bill by two orders of magnitude.\n\nSo amounts are compared **by canonical value**, not by digits:\n\n```\n_AMOUNT = re.compile(r\"^\\d{1,3}(?:[.,\\s]\\d{3})*[.,]\\d{1,2}$|^\\d+[.,]\\d{1,2}$\")\n\ndef _amount_value(token: str) -> str | None:\n    \"\"\"Normalise a currency token to canonical `integer.decimals`.\n\n    Returns None if the token is not amount-shaped. This is what stops `45.00`\n    and `4.500` being considered equal.\n\n    Treats the LAST `.`/`,` as the decimal separator and the rest as thousands,\n    which is the Venezuelan convention (`1.234,56`) and also the English one\n    (`1,234.56`).\n    \"\"\"\n    if not _AMOUNT.match(token):\n        return None\n    clean = token.replace(\" \", \"\")\n    cut = max(clean.rfind(\".\"), clean.rfind(\",\"))\n    integer = re.sub(r\"\\D\", \"\", clean[:cut])\n    decimals = clean[cut + 1:]\n    return f\"{int(integer or 0)}.{decimals}\"\n```\n\nThree rules, strictest first:\n\n``` php\ndef _number_supported(token, evidence, evidence_digits, evidence_amounts) -> bool:\n    # 1. Literal, with digit boundaries. `12345678` is NOT accepted just because\n    #    it appears inside a longer `123456789`.\n    if re.search(r\"(?<!\\d)\" + re.escape(token) + r\"(?!\\d)\", evidence):\n        return True\n\n    # 2. Amount value. `45,00` (Venezuelan decimal comma) matches `45.00` from\n    #    the API, but `4.500` does NOT match `45.00` — different values.\n    value = _amount_value(token)\n    if value is not None:\n        return value in evidence_amounts\n\n    # 3. Digit equality — ONLY for identifiers (phones, accounts, tax IDs:\n    #    7+ digits), where separators are cosmetic and don't change the value.\n    if len(_digits_only(token)) >= _MIN_IDENTIFIER_DIGITS:\n        return _digits_only(token) in evidence_digits\n    return False\n```\n\nRule 3 is deliberately narrow. A short number that matched neither literally nor as an amount is rejected — `4.500`\n\nmust not pass just because it shares digits with `45.00`\n\n.\n\nThe instinct is to treat escalations as the failure rate and drive them to zero. That optimises the wrong thing. Every escalation you remove by loosening the verifier becomes a confident answer about a bill, and some of those are wrong in a way the customer notices.\n\nEach reason carries a summary the human agent reads, so they don't re-read the whole thread:\n\n```\n_AGENT_SUMMARY: dict[EscalationReason, str] = {\n    EscalationReason.NO_CONTEXT:\n        \"The customer asked something that is not in the knowledge base.\",\n    EscalationReason.UNGROUNDED_OUTPUT:\n        \"The bot was about to give an unsupported figure; it was stopped for safety.\",\n    EscalationReason.USER_REQUEST:\n        \"The customer asked to speak to a person.\",\n    EscalationReason.TOOL_ERROR:\n        \"Could not query the billing system (possible outage).\",\n    EscalationReason.CAPABILITY_MISSING:\n        \"The customer wants data the bot cannot query yet.\",\n}\n```\n\n`USER_REQUEST`\n\nfires unconditionally. There is always pressure to have the bot try once more before releasing the conversation — it improves your containment metric. It also means the subscriber who typed *\"I want to talk to a human\"* gets a paragraph they didn't ask for. On a channel people use to talk to people, ignoring that request teaches them the bot is an obstacle, not a front door.\n\nOne honest note on scope: this is one system for one operator, described as architecture rather than as a benchmark. We are not going to show you a deflection-rate chart from a sample of one.\n\nAlso worth stating plainly: the billing API integration is **contract-first**. There is a documented HTTP contract and a simulator implementing it; the production system on the other side is still being built. Everything above — capability resolution, circuit breaker, tool-results-as-evidence — runs against that contract. We think that is the right order (the contract is the thing both sides agree on), but you should know it when you read the code.\n\nLonger version, including why WhatsApp specifically removes every UI affordance you'd normally use to show uncertainty: [Corrective RAG for Billing Questions on WhatsApp](https://softronic.dev/blog/crag-whatsapp-billing-assistant)", "url": "https://wpnews.pro/news/corrective-rag-for-billing-the-bug-is-not-retrieval-it-s-the-model-narrating", "canonical_source": "https://dev.to/softronic/corrective-rag-for-billing-the-bug-is-not-retrieval-its-the-model-narrating-correct-numbers-wrong-4938", "published_at": "2026-07-30 16:49:38+00:00", "updated_at": "2026-07-30 17:02:04.451927+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "natural-language-processing", "ai-infrastructure"], "entities": ["CRAG", "pgvector", "Postgres", "WhatsApp", "ISP", "Yan et al."], "alternates": {"html": "https://wpnews.pro/news/corrective-rag-for-billing-the-bug-is-not-retrieval-it-s-the-model-narrating", "markdown": "https://wpnews.pro/news/corrective-rag-for-billing-the-bug-is-not-retrieval-it-s-the-model-narrating.md", "text": "https://wpnews.pro/news/corrective-rag-for-billing-the-bug-is-not-retrieval-it-s-the-model-narrating.txt", "jsonld": "https://wpnews.pro/news/corrective-rag-for-billing-the-bug-is-not-retrieval-it-s-the-model-narrating.jsonld"}}