{"slug": "the-agent-that-refuses-to-guess", "title": "The agent that refuses to guess", "summary": "A developer at a B2B telecom consultancy built an open-source AI agent, Invoice Sentinel, that automates telecom invoice auditing and dispute letter generation. The agent, running on Gemini 3.5 Flash and Google ADK, enforces a strict no-guessing rule through structural, tool-signature, and prose-checking layers to prevent invented figures. The developer detailed four defects discovered through adversarial testing, including a false 'looks clean' verdict when no contract was on file.", "body_md": "I work at a B2B telecom consultancy. I'm not the one auditing the bills, but every month I\n\nwatch how it's done: open the invoice PDF, check every line against the signed contract,\n\ncompare it with what the account used in earlier cycles, and write up whatever doesn't add\n\nup.\n\nIt's slow, it doesn't scale, and it's the first task dropped when the month gets busy —\n\nwhich is exactly when the money leaks. And outside a consultancy it's worse: most companies\n\njust pay the bill because it arrived.\n\nSo I built an agent that does the whole job. One invoice PDF in; a dispute letter for the\n\ncarrier and an executive summary for the customer out, with nobody in the loop. It runs on\n\nGemini 3.5 Flash and the Google ADK, on Cloud Run and Firestore, and it's open source:\n\n[github.com/Bren0-lz/invoice-sentinel](https://github.com/Bren0-lz/invoice-sentinel).\n\nThis post is about the one decision that shaped everything else, and about the four defects\n\nthat only showed up when I stopped reading my own code and started attacking the running\n\nservice.\n\nAn agent that writes a dispute letter is writing a document addressed to a third party and\n\nsigned by the customer. If a figure in it is invented, the customer doesn't lose a feature\n\n— they lose credibility with their own supplier, and they lose it in writing.\n\nSo the rule is absolute, and it isn't a line in a prompt. It's enforced in three layers,\n\neach of which would have to fail independently:\n\n**Structurally.** The rule engine is pure Python with `Decimal`\n\n. No module under `rules/`\n\nimports an LLM client. Five rules across three families, running concurrently under a\n\n`ParallelAgent`\n\n.\n\n**In the tool signatures.** No auditor tool accepts a monetary value as an argument.\n\n`flag_anomaly(finding_id, rationale)`\n\ncannot be talked into disputing four thousand reais\n\nthat nobody computed, because there is no parameter to put it in. A test asserts this with\n\n`inspect.signature`\n\n, so the guarantee survives someone adding a tool later.\n\n**In the generated prose.** `amount_guard`\n\nextracts every money-shaped number from the\n\nletter and checks it against the set the engine actually computed. Invented one? The model\n\nis told *which* number and rewrites it. Insisted past the rewrite budget? The dispute is\n\nstored as `blocked`\n\n— never as a `draft`\n\nsomebody might later mistake for reviewed.\n\nThe third layer is the one people forget, and the one that matters most. The first two make\n\nthe model unable to author a figure; only the third checks what it actually wrote.\n\nThe agent ships layout profiles for two carriers. Send it a bill from a third and it\n\nrefuses, rather than reading it with someone else's separator hints — because a Brazilian\n\n`1.234,56`\n\nread as American is `1.234`\n\n, and that is the shape of error nothing downstream\n\ncatches. It's plausible. It's wrong. It's a thousand times too small.\n\nThe same principle runs through the whole system:\n\nThat last one was a real defect, and it's the best lesson in the project.\n\nEvery one of these came from using the deployed service the way an evaluator would, with\n\ndocuments nobody prepared knowing what would be checked.\n\nWith no contract on file, three of five rules are skipped and the findings list comes back\n\nempty. The prompt then told the model to say the invoice \"looks clean\".\n\nA brand-new account is the first path anyone walks. It got that certificate over a bill\n\nthat had a real overcharge — which appeared the moment the contract was filed. Worse, the\n\nrule for escalating a missing contract existed ten lines below in the same prompt and was\n\n**unreachable**: escalation needs a `finding_id`\n\n, and there are no findings.\n\n**Silence from a rule that never ran is not evidence that the bill is correct.** Empty now\n\nbranches on whether a contract exists, before it can mean anything.\n\nTranscribing a three-plan contract, it filed six — helpfully re-adding each plan under the\n\nabbreviated name the invoice prints, so whoever had to match the two documents would have\n\nan easier time.\n\nThe contract is the baseline every figure is computed against. It now contained terms\n\nnobody signed. Fixed with a validator that rejects duplicate plans and a repair loop that\n\nsends the model back, not with a sterner prompt.\n\nA Brazilian invoice was transcribed perfectly and still reported 149.42 out of balance.\n\nICMS, PIS, COFINS, FUST and FUNTTEL are computed *inside* the price here and itemised only\n\nbecause Lei 12.741/2012 requires it. Summing them double-counts. The schema now declares\n\nthe tax regime per carrier — inclusive in Brazil, additive in the US — so the balance\n\nwarning means what it says instead of firing on every bill from a country whose invoices\n\nare all built that way.\n\nThis is my favourite, because I found it by going looking.\n\nThe parser that coerces a transcribed amount into `Decimal`\n\nstrips currency noise with a\n\nregex that keeps only digits, separators and an **ASCII hyphen**. So:\n\n| Printed on the bill | Parsed as | Should be |\n|---|---|---|\n`−50,00` (U+2212, what a properly typeset PDF prints) |\n`+50.00` |\n`-50.00` |\n`–50,00` (en dash, from a layout pasted out of a word processor) |\n`+50.00` |\n`-50.00` |\n`—50,00` (em dash) |\n`+50.00` |\n`-50.00` |\n`(50,00)` (accounting notation) |\n`+50.00` |\n`-50.00` |\n\nA fifty-real credit became a fifty-real charge, and the invoice total it reconciles against\n\nmoved by a hundred.\n\nHere's what makes it worth writing about: **nothing downstream could have caught it.** The\n\nthree layers protecting the money protect against the *model* authoring a figure. They say\n\nnothing about a figure entering wrong. `amount_guard`\n\ncompares the letter against what the\n\nengine computed — it cannot compare the engine against the page. The sign was gone before\n\nthe value was ever a `Decimal`\n\n.\n\nThe parser had no tests at all. The deterministic safety net under the project's central\n\nclaim, with zero coverage.\n\nIt now normalises the dash forms and reads parentheses as negative. And a value that states\n\nits sign twice — `(-50,00)`\n\n— is **refused** rather than resolved, because picking one\n\nwould be guessing about money.\n\n**Make the guarantee structural, not textual.** Every time I needed something to be true, a\n\nprompt was the weakest place to put it. A validator, a function signature or a type has to\n\nbe actively defeated; a prompt just has to be forgotten.\n\n**Test the deployed thing, with documents you didn't make.** My whole synthetic dataset\n\ncomes from one generator, so it proves the rule engine agrees with the generator — a weaker\n\nclaim than it looks. The first time I audited an invoice built independently, three defects\n\nsurfaced at once, and all three were *false positives*: findings that weren't there. That\n\ndocument is now committed as a regression fixture, as the false-positive control with the\n\ngenerator taken out of the loop.\n\n**Watch what a refusal actually stops.** I added a refusal at the intake stage and it\n\nchanged nothing: sibling stages of a `SequentialAgent`\n\ncan't be skipped from outside, and\n\nthe extractor was reading the attachment on its own and extracting anyway, right past the\n\n\"no\". A refusal is a text in the transcript until something downstream asks.\n\n**Count your model calls, honestly.** Only three of eleven pipeline stages are `LlmAgent`\n\ns.\n\nDeciding whether an attachment is a contract or a bill is pattern matching; so is picking a\n\ncarrier profile off a letterhead. A model call whose only job is to invoke a function is\n\ntokens spent on nothing, and it dresses deterministic work up as reasoning.\n\nMeasured on 15 synthetic invoices, 4 accounts, 2 visually distinct carrier layouts:\n\nClone it and run `pytest`\n\n; the accuracy claims verify themselves against extractions\n\ncommitted in the repo.\n\nHonest limitations, since I'd rather say them than have you find them: two carriers only,\n\nthe dataset is synthetic, and the audited PDF isn't stored yet, so an audit can't be\n\nre-checked against the document that produced it.\n\nBuilt for the [All Things Agentic\nHackathon](https://allthingsagentichackathon.devpost.com/), track", "url": "https://wpnews.pro/news/the-agent-that-refuses-to-guess", "canonical_source": "https://dev.to/breno_luiz_ab86a62d8c5c7c/the-agent-that-refuses-to-guess-2364", "published_at": "2026-08-31 03:24:41+00:00", "updated_at": "2026-08-31 03:51:32.133612+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "developer-tools"], "entities": ["Gemini 3.5 Flash", "Google ADK", "Cloud Run", "Firestore", "Invoice Sentinel"], "alternates": {"html": "https://wpnews.pro/news/the-agent-that-refuses-to-guess", "markdown": "https://wpnews.pro/news/the-agent-that-refuses-to-guess.md", "text": "https://wpnews.pro/news/the-agent-that-refuses-to-guess.txt", "jsonld": "https://wpnews.pro/news/the-agent-that-refuses-to-guess.jsonld"}}