I work at a B2B telecom consultancy. I'm not the one auditing the bills, but every month I
watch how it's done: open the invoice PDF, check every line against the signed contract,
compare it with what the account used in earlier cycles, and write up whatever doesn't add
up.
It's slow, it doesn't scale, and it's the first task dropped when the month gets busy —
which is exactly when the money leaks. And outside a consultancy it's worse: most companies
just pay the bill because it arrived.
So I built an agent that does the whole job. One invoice PDF in; a dispute letter for the
carrier and an executive summary for the customer out, with nobody in the loop. It runs on
Gemini 3.5 Flash and the Google ADK, on Cloud Run and Firestore, and it's open source:
github.com/Bren0-lz/invoice-sentinel. This post is about the one decision that shaped everything else, and about the four defects
that only showed up when I stopped reading my own code and started attacking the running
service.
An agent that writes a dispute letter is writing a document addressed to a third party and
signed by the customer. If a figure in it is invented, the customer doesn't lose a feature
— they lose credibility with their own supplier, and they lose it in writing.
So the rule is absolute, and it isn't a line in a prompt. It's enforced in three layers,
each of which would have to fail independently:
Structurally. The rule engine is pure Python with Decimal
. No module under rules/
imports an LLM client. Five rules across three families, running concurrently under a
ParallelAgent
.
In the tool signatures. No auditor tool accepts a monetary value as an argument.
flag_anomaly(finding_id, rationale)
cannot be talked into disputing four thousand reais
that nobody computed, because there is no parameter to put it in. A test asserts this with
inspect.signature
, so the guarantee survives someone adding a tool later.
In the generated prose. amount_guard
extracts every money-shaped number from the
letter and checks it against the set the engine actually computed. Invented one? The model
is told which number and rewrites it. Insisted past the rewrite budget? The dispute is
stored as blocked
— never as a draft
somebody might later mistake for reviewed.
The third layer is the one people forget, and the one that matters most. The first two make
the model unable to author a figure; only the third checks what it actually wrote.
The agent ships layout profiles for two carriers. Send it a bill from a third and it
refuses, rather than reading it with someone else's separator hints — because a Brazilian
1.234,56
read as American is 1.234
, and that is the shape of error nothing downstream
catches. It's plausible. It's wrong. It's a thousand times too small.
The same principle runs through the whole system:
That last one was a real defect, and it's the best lesson in the project.
Every one of these came from using the deployed service the way an evaluator would, with
documents nobody prepared knowing what would be checked.
With no contract on file, three of five rules are skipped and the findings list comes back
empty. The prompt then told the model to say the invoice "looks clean".
A brand-new account is the first path anyone walks. It got that certificate over a bill
that had a real overcharge — which appeared the moment the contract was filed. Worse, the
rule for escalating a missing contract existed ten lines below in the same prompt and was
unreachable: escalation needs a finding_id
, and there are no findings.
Silence from a rule that never ran is not evidence that the bill is correct. Empty now
branches on whether a contract exists, before it can mean anything.
Transcribing a three-plan contract, it filed six — helpfully re-adding each plan under the
abbreviated name the invoice prints, so whoever had to match the two documents would have
an easier time.
The contract is the baseline every figure is computed against. It now contained terms
nobody signed. Fixed with a validator that rejects duplicate plans and a repair loop that
sends the model back, not with a sterner prompt.
A Brazilian invoice was transcribed perfectly and still reported 149.42 out of balance.
ICMS, PIS, COFINS, FUST and FUNTTEL are computed inside the price here and itemised only
because Lei 12.741/2012 requires it. Summing them double-counts. The schema now declares
the tax regime per carrier — inclusive in Brazil, additive in the US — so the balance
warning means what it says instead of firing on every bill from a country whose invoices
are all built that way.
This is my favourite, because I found it by going looking.
The parser that coerces a transcribed amount into Decimal
strips currency noise with a
regex that keeps only digits, separators and an ASCII hyphen. So:
| Printed on the bill | Parsed as | Should be |
|---|---|---|
−50,00 (U+2212, what a properly typeset PDF prints) |
||
+50.00 |
||
-50.00 |
||
–50,00 (en dash, from a layout pasted out of a word processor) |
||
+50.00 |
||
-50.00 |
||
—50,00 (em dash) |
||
+50.00 |
||
-50.00 |
||
(50,00) (accounting notation) |
||
+50.00 |
||
-50.00 |
A fifty-real credit became a fifty-real charge, and the invoice total it reconciles against
moved by a hundred.
Here's what makes it worth writing about: nothing downstream could have caught it. The
three layers protecting the money protect against the model authoring a figure. They say
nothing about a figure entering wrong. amount_guard
compares the letter against what the
engine computed — it cannot compare the engine against the page. The sign was gone before
the value was ever a Decimal
.
The parser had no tests at all. The deterministic safety net under the project's central
claim, with zero coverage.
It now normalises the dash forms and reads parentheses as negative. And a value that states
its sign twice — (-50,00)
— is refused rather than resolved, because picking one
would be guessing about money.
Make the guarantee structural, not textual. Every time I needed something to be true, a
prompt was the weakest place to put it. A validator, a function signature or a type has to
be actively defeated; a prompt just has to be forgotten.
Test the deployed thing, with documents you didn't make. My whole synthetic dataset
comes from one generator, so it proves the rule engine agrees with the generator — a weaker
claim than it looks. The first time I audited an invoice built independently, three defects
surfaced at once, and all three were false positives: findings that weren't there. That
document is now committed as a regression fixture, as the false-positive control with the
generator taken out of the loop.
Watch what a refusal actually stops. I added a refusal at the intake stage and it
changed nothing: sibling stages of a SequentialAgent
can't be skipped from outside, and
the extractor was reading the attachment on its own and extracting anyway, right past the
"no". A refusal is a text in the transcript until something downstream asks.
Count your model calls, honestly. Only three of eleven pipeline stages are LlmAgent
s.
Deciding whether an attachment is a contract or a bill is pattern matching; so is picking a
carrier profile off a letterhead. A model call whose only job is to invoke a function is
tokens spent on nothing, and it dresses deterministic work up as reasoning.
Measured on 15 synthetic invoices, 4 accounts, 2 visually distinct carrier layouts:
Clone it and run pytest
; the accuracy claims verify themselves against extractions
committed in the repo.
Honest limitations, since I'd rather say them than have you find them: two carriers only,
the dataset is synthetic, and the audited PDF isn't stored yet, so an audit can't be
re-checked against the document that produced it.
Built for the All Things Agentic Hackathon, track