I don’t trust my own AI extractor. So I made it prove every number.

A developer building an LLM-based PDF-to-spreadsheet tool for bank statements created a mathematical verification method that checks each extracted row against the statement's running balance, ensuring 99% accuracy without human review. The system rejects any extraction where the balance chain is inconsistent, catching single-digit misreads that would otherwise go unnoticed. The approach uses the statement's own redundant data as a self-checking mechanism, scoring extractions by the fraction of adjacent row pairs that satisfy the balance identity.

Back to Blog /blog I don’t trust my own AI extractor. So I made it prove every number. I build LLM extraction systems for a living. Which is exactly why, when I built a tool to convert PDF bank statements into spreadsheets, I refused to trust the model. Here’s the failure mode that keeps me up at night. You feed a bank statement PDF to a vision-language model. It returns 700 clean-looking rows. 699 are perfect. One has 1,240.50 where the statement says 1,340.50 — a single digit, misread. Nothing about that row looks wrong. It sorts fine, imports fine, sums to a plausible total. Weeks later a bookkeeper’s reconciliation is off by 100 and nobody knows why. Wrong-but-plausible output is worse than an error , because an error you catch and a plausible lie you don’t. A tool that fails loudly is annoying. A tool that’s confidently, quietly wrong on financial data is dangerous. So the interesting engineering problem in this product was never “extract the transactions.” Any decent model does that. The problem was: how do you know the extraction is right, per document, without a human checking it? The answer turned out to be sitting in the statement itself. A correctly parsed statement is internally consistent Every bank statement prints a running balance next to each transaction. That running balance is a redundant, self-checking signal the bank hands you for free. In the statement’s native newest-first order, this identity holds for every adjacent pair of rows: balance i + amount i-1 === balance i-1 The older row’s balance, plus the newer row’s amount, equals the newer row’s balance. It has to — that’s what a running balance is . And here’s the property that makes it useful as a gate: a correctly extracted statement satisfies this for every row; a misread amount or balance breaks it. The model can’t hallucinate a value that happens to keep the whole chain consistent — the constraint is over-determined. So the accuracy check is just: what fraction of adjacent row pairs satisfy that identity? export function reconcile transactions, options = {} { const { tolerance = 0.01 } = options; // Only rows with both a numeric amount and balance can be reconciled. const rows = transactions || .filter t = t && typeof t.amount === 'number' && typeof t.balance === 'number' ; if rows.length < 2 { return { score: 1, ok: 0, total: 0, mismatches: }; } let ok = 0; for let i = 1; i < rows.length; i++ { const newer = rows i - 1 ; const older = rows i ; const diff = Math.abs older.balance + newer.amount - newer.balance ; if diff < tolerance ok++; } const total = rows.length - 1; return { score: ok / total, ok, total }; } The tolerance is 0.01 — a cent — to absorb floating-point noise, not real error. The score is a fraction of adjacent pairs. On a real 16-page Handelsbanken statement with 742 transactions, this scores 740/741 = 0.9986 . Nothing ships below 0.99 . That single number does triple duty: it’s the verify step when I’m building a new parser, the regression gate in CI, and the only marketing claim I’ll make — reconciled, not guessed . It’s not an adjective. It’s a measurement, recomputed on every conversion. Twist 1: banks don’t agree on which way time flows The identity above assumes newest-first ordering, which is what Nordic banks print. Most US banks print oldest-first. That’s a layout choice, not a data difference — but it flips the sign of the check. I could detect the ordering heuristically. Instead I lean on a nice property of the constraint: a correctly parsed statement reconciles in exactly one of the two orders, and wrong magnitudes fail in both. So score it both ways and keep the better one: js export function reconcileBestDirection transactions, options = {} { const forward = reconcile transactions, options ; const backward = reconcile transactions.slice .reverse , options ; return backward.score forward.score ? { ...backward, reversed: true } : { ...forward, reversed: false }; } This removes the newest-first assumption without weakening the gate. A genuinely wrong extraction can’t score high in either direction — there’s no ordering that makes bad numbers foot. Getting the direction “for free” costs one extra pass and buys correctness across banking conventions I’ve never seen. Twist 2: some statements barely print balances at all Plenty of layouts — UK/Commonwealth two-column statements, day-grouped formats — print the running balance only on the last row of each day, leaving most rows blank. Too few balances for full row-by-row continuity. But the printed balances that are there are still hard anchors: the signed amounts between two consecutive printed balances must move you from one to the other. So there’s a second reconciler that walks the rows, accumulates signed amounts, and checks each segment against the next printed balance — seeded by the statement’s opening balance: js for const row of seq { acc += row.amount; if typeof row.balance === 'number' { total++; if Math.abs anchor + acc - row.balance <= tolerance ok++; anchor = row.balance; // re-anchor acc = 0; } } This is strictly stronger than footing the account summary opening + total debits + total credits = closing , because it localizes the error to a segment instead of just telling you the whole thing doesn’t add up. It precisely catches the classic OCR failure — a flipped sign — because the one segment containing the flip won’t foot, even if everything else does. The part I care about most: the gate refuses to bless what it can’t verify Here’s the subtle bug that a naive version of this ships with. Look back at reconcile : with fewer than two usable rows, it returns score: 1 . Nothing to contradict, so it “passes.” Which means an empty extraction, or one where the model returned rows but no balances, scores a perfect 1.0 and sails through. That’s exactly backwards. An unverifiable result is not a verified one. So pass/fail requires evidence — at least one pair actually checked: export function reconciliationGate transactions, options = {} { const { threshold = 0.99, ...rest } = options; const result = reconcile transactions, rest ; return { ...result, pass: result.total 0 && result.score = threshold }; } result.total 0 is the whole ethic. If the tool couldn’t verify your statement, it doesn’t quietly hand you numbers and hope — it labels the output UNVERIFIED , and that label follows the data everywhere: into a banner row at the top of the file, the download filename, the HTTP response header, and the API JSON. You always know whether you got a checked result or an unchecked one. The tool is never confidently wrong; at worst it’s honestly unsure. Why this beats “99% accurate” Every converter in this space claims high accuracy. Nobody can show it for the document in front of you , because their accuracy is a number from someone else’s test set, not yours. The reconciliation gate is different in kind: it’s computed on your statement, from your statement’s own printed balances, at conversion time. The claim is checkable by construction. If every row of your statement foots, you can see that. If it comes back UNVERIFIED, you know not to trust it before it touches your books. It also means the model can be swapped, upgraded, or fail in a novel way, and the safety property still holds — because the check is external to the model. The reconciliation math is ground truth the model doesn’t get a vote on. It can’t game a constraint it doesn’t control. What it doesn’t do because that matters too - It verifies internal consistency , not truth against the bank’s servers. If a statement PDF were itself doctored so that its fake transactions still foot, reconciliation would pass it. It catches extraction errors, not forgery. - Rows with no amount or no balance can’t be reconciled and don’t count toward the score — which is why the “require evidence” rule matters. - It’s a per-statement check, not a per-cell guarantee. A 99.86% score means one adjacent pair out of 741 didn’t foot within a cent — worth surfacing, not necessarily worth panicking over. You still get to look. Takeaway If you’re putting an LLM anywhere near numbers people will act on, look for the redundant signal already in your input. Bank statements carry a running balance; invoices carry line-item-plus-tax-equals-total; ledgers carry double entries. That redundancy is a free, deterministic check you can wrap around a probabilistic model — and it lets you make a promise most AI tools can’t: if I’m not sure, I’ll tell you, instead of guessing.