Most RAG demos are graded by an audience that cannot check the answer. Ask a docs bot something, get a fluent paragraph back, nobody in the room knows whether sentence three is invented. The demo lands.
Billing is not like that. When a subscriber asks "why is my bill higher this month" they are holding the invoice. They will check.
We built a WhatsApp assistant for an ISP that answers plan and billing questions. This post is about the two things that turned out to matter, neither of which is "we used CRAG":
Code is from the real system. Comments translated from Spanish, otherwise untouched.
inbound message
↓
state gate ──────────────► human agent active? bot writes nothing
↓
hybrid retrieval
├─ pgvector cosine ─┐
└─ Postgres FTS ────┴─► RRF fusion ─► CRAG prune
↓
capability call (billing API) ─► circuit breaker
↓
no context AND no tool data? ──────────► escalate (before spending a token)
↓
LLM, structured output
↓
grounding verifier ────────► not grounded? ─► escalate
↓
humanizer ─────────────────► forbidden phrase? ─► escalate
↓
send
Every arrow pointing right is a refusal path. That is the design.
Vector similarity and Postgres ts_rank
live on different scales. Adding them is meaningless. Reciprocal Rank Fusion only looks at position, so it fuses them without pretending the scores are commensurable:
DEFAULT_RRF_K = 60
"""Smoothing constant. Damps the weight of the top positions so a single
ranking cannot dominate the fusion. 60 is the value from the original paper."""
def reciprocal_rank_fusion(
rankings: Sequence[Sequence[str]], *, k: int = DEFAULT_RRF_K
) -> dict[str, float]:
scores: dict[str, float] = {}
for ranking in rankings:
for position, id_ in enumerate(ranking, start=1):
scores[id_] = scores.get(id_, 0.0) + 1.0 / (k + position)
return scores
Ties break on the id, deterministically. A result that reorders between identical runs makes tests flaky and debugging impossible.
Why full-text at all, when you have embeddings? Because account numbers, tax IDs and national ID numbers have no semantic meaning. An embedding cannot score digits. FTS is the only path by which those are ever retrieved.
CRAG (Yan et al., 2024) evaluates retrieval relevance and drops the noise before generating. The paper uses a separate lightweight evaluator model.
We skipped the evaluator. The retriever already computed cosine similarity and knows whether full-text hit — running a per-chunk model call every turn is the expensive part of CRAG and the least useful part when you already have scores.
The problem it solves is concrete: top_k
asks for 6 chunks and RRF always returns 6, whether or not the sixth makes sense. Fusion orders, it does not filter. That filler enters the prompt at full size, burns tokens every turn, and dilutes the good evidence.
DEFAULT_MARGIN = 0.12
"""How far below the best chunk we still tolerate.
Deliberately a *relative* margin. A fixed threshold behaves badly at both ends:
on a well-answered query (best = 0.80) it lets a 0.40 through as "good" when
next to the best it's noise, and on a weak query (best = 0.42) it would prune
everything. The margin adapts: each chunk is compared to the best found on
*this* turn."""
DEFAULT_MIN_CHUNKS = 2
def prune_irrelevant[Chunk: _Scorable](
chunks: Sequence[Chunk],
*,
margin: float = DEFAULT_MARGIN,
min_chunks: int = DEFAULT_MIN_CHUNKS,
) -> list[Chunk]:
if len(chunks) <= min_chunks:
return list(chunks)
best = max((c.vector_score or 0.0 for c in chunks), default=0.0)
cutoff = best - margin
kept = {i for i, c in enumerate(chunks) if _is_relevant(c, cutoff)}
for i in range(len(chunks)):
if len(kept) >= min_chunks:
break
kept.add(i)
return [c for i, c in enumerate(chunks) if i in kept]
def _is_relevant(chunk: _Scorable, cutoff: float) -> bool:
"""A full-text hit always survives; everything else competes on similarity.
Full-text immunity is not a convenient exception: `plainto_tsquery` requires
the terms to actually appear in the chunk, so the hit is itself proof of
relevance. It is also the only path by which account numbers and tax IDs are
retrieved — digits carry no semantic meaning, so they arrive here with a low
cosine that would otherwise condemn them.
"""
if chunk.matched_by_text:
return True
return (chunk.vector_score or 0.0) >= cutoff
Note the bias, because it is the opposite of the verifier below: when in doubt, keep. Over-pruning would delete the one line that answered the question and escalate a case the bot could have solved — worse than wasting a few tokens.
The real balance is not in any document. It comes from the billing API, and it has to count as citable evidence or the verifier in step 4 would reject the correct answer:
@dataclass(frozen=True, slots=True)
class ToolResult:
capability: str
ok: bool
evidence_text: str | None = None
"""Text ready to pass to the LLM as evidence; None if there was no data."""
data: dict[str, Any] | None = None
error: str | None = None
should_escalate: bool = False
"""`True` if the failure is infrastructure (billing API down) → escalate.
A "customer not found" does NOT escalate: it's data that simply isn't there."""
That last field is a distinction worth stealing. "The API is down" and "this customer doesn't exist" both produce no data, and they are not the same event. One is an outage, the other is an answer.
except CrmBusinessError as exc:
return ToolResult(capability=capability.name, ok=False,
error=exc.message, should_escalate=False)
except CRMError as exc:
return ToolResult(capability=capability.name, ok=False,
error=str(exc), should_escalate=True)
When the API really is down, retrying makes it worse — every inbound message burns 15s of timeout × 3 attempts, the worker queue backs up, and the API gets a stampede exactly while it is trying to recover. So there is a circuit breaker, and for the bot "API down" simply translates to escalate fast:
def record_failure(self) -> None:
"""A failure counts; in HALF_OPEN it reopens immediately.
Reopening on the first failure in HALF_OPEN is deliberate: if the probe
request fails, the API is still bad and there is no point spending the
rest of the threshold to confirm it.
"""
self._failures += 1
if self._state is CircuitState.HALF_OPEN or self._failures >= self._threshold:
self._state = CircuitState.OPEN
self._opened_at = self._now()
The clock is injected so tests can verify reopening without actually sleeping. A test that does sleep(60)
gets deleted or marked slow, and the most delicate part of the system loses its coverage with it.
Everything above is retrieval hygiene. This is the piece I would push hardest on in a review of anyone else's billing assistant, because it survives every model upgrade.
The prompt asks the model not to invent. This verifies it:
class GroundingVerifier:
def verify(
self, *, answer: str, used_chunk_ids: list[str],
retrieved_chunk_ids: list[str], evidence_texts: list[str],
tool_result_texts: list[str] | None = None,
) -> GroundingResult:
Two checks:
The bias is deliberate and the inverse of the pruner: when in doubt, reject. A rejected answer escalates to a human, which is annoying but safe. A hallucination that ships is expensive and costs trust. That module is held to 100% test coverage.
Naive version: strip non-digits from the number, check if those digits appear in the evidence.
That is wrong, and here is the counterexample that killed it:
Evidence (from the billing API): 45.00
Model says: 4.500
Digits of both: "4500" ✓ passes
Two different amounts. Same digits. In Venezuelan notation 4.500
reads as four thousand five hundred; 45.00
is forty-five. Digit equality green-lights a hallucination that changes the magnitude of someone's bill by two orders of magnitude.
So amounts are compared by canonical value, not by digits:
_AMOUNT = re.compile(r"^\d{1,3}(?:[.,\s]\d{3})*[.,]\d{1,2}$|^\d+[.,]\d{1,2}$")
def _amount_value(token: str) -> str | None:
"""Normalise a currency token to canonical `integer.decimals`.
Returns None if the token is not amount-shaped. This is what stops `45.00`
and `4.500` being considered equal.
Treats the LAST `.`/`,` as the decimal separator and the rest as thousands,
which is the Venezuelan convention (`1.234,56`) and also the English one
(`1,234.56`).
"""
if not _AMOUNT.match(token):
return None
clean = token.replace(" ", "")
cut = max(clean.rfind("."), clean.rfind(","))
integer = re.sub(r"\D", "", clean[:cut])
decimals = clean[cut + 1:]
return f"{int(integer or 0)}.{decimals}"
Three rules, strictest first:
def _number_supported(token, evidence, evidence_digits, evidence_amounts) -> bool:
if re.search(r"(?<!\d)" + re.escape(token) + r"(?!\d)", evidence):
return True
value = _amount_value(token)
if value is not None:
return value in evidence_amounts
if len(_digits_only(token)) >= _MIN_IDENTIFIER_DIGITS:
return _digits_only(token) in evidence_digits
return False
Rule 3 is deliberately narrow. A short number that matched neither literally nor as an amount is rejected — 4.500
must not pass just because it shares digits with 45.00
.
The instinct is to treat escalations as the failure rate and drive them to zero. That optimises the wrong thing. Every escalation you remove by loosening the verifier becomes a confident answer about a bill, and some of those are wrong in a way the customer notices.
Each reason carries a summary the human agent reads, so they don't re-read the whole thread:
_AGENT_SUMMARY: dict[EscalationReason, str] = {
EscalationReason.NO_CONTEXT:
"The customer asked something that is not in the knowledge base.",
EscalationReason.UNGROUNDED_OUTPUT:
"The bot was about to give an unsupported figure; it was stopped for safety.",
EscalationReason.USER_REQUEST:
"The customer asked to speak to a person.",
EscalationReason.TOOL_ERROR:
"Could not query the billing system (possible outage).",
EscalationReason.CAPABILITY_MISSING:
"The customer wants data the bot cannot query yet.",
}
USER_REQUEST
fires unconditionally. There is always pressure to have the bot try once more before releasing the conversation — it improves your containment metric. It also means the subscriber who typed "I want to talk to a human" gets a paragraph they didn't ask for. On a channel people use to talk to people, ignoring that request teaches them the bot is an obstacle, not a front door.
One honest note on scope: this is one system for one operator, described as architecture rather than as a benchmark. We are not going to show you a deflection-rate chart from a sample of one.
Also worth stating plainly: the billing API integration is contract-first. There is a documented HTTP contract and a simulator implementing it; the production system on the other side is still being built. Everything above — capability resolution, circuit breaker, tool-results-as-evidence — runs against that contract. We think that is the right order (the contract is the thing both sides agree on), but you should know it when you read the code.
Longer version, including why WhatsApp specifically removes every UI affordance you'd normally use to show uncertainty: Corrective RAG for Billing Questions on WhatsApp