When I started building StatementDecoder, I thought the difficult part would be reading bank statements.
It wasn't.
Reading them turned out to be relatively easy.
The difficult part was knowing when the output was wrong.
Like most people building with LLMs, my first version was straightforward. Feed the statement to the model and ask for structured JSON. For a while it felt almost magical. It handled tables I'd never written parsers for. It coped with different layouts. It even did a decent job on scanned PDFs.
Then I stopped testing with the handful of sample statements I'd been using and started collecting real ones.
Everything fell apart.
A PDF from one UK bank looked nothing like another. Wise statements had quirks I'd never seen before. Revolut CSVs weren't as simple as I expected. Then there were OFX files that looked like they hadn't been touched since the late 90s.
The model still did a good job.
Just not a perfect one.
And that's the problem.
Imagine a 14-page statement with a few hundred transactions. The model gets every row right except one. Maybe it swaps two digits in an amount. Maybe a transaction wrapped across two pages and it guessed the wrong date. Maybe it silently merged two rows together.
The JSON still looks completely reasonable.
Nothing says, "By the way, transaction 187 is wrong."
If you're importing the data into bookkeeping software, that one mistake is enough to make the whole export untrustworthy. My first reaction was the obvious one: improve the prompt.
Then I tried changing models.
Then I spent time tweaking extraction logic.
Eventually I realised I was solving the wrong problem.
I was asking the model to tell me the answer and trusting the same model to tell me whether the answer was correct.
That's a terrible idea.
The thing that finally fixed the architecture wasn't a better prompt. It was adding something that didn't care where the data came from.
Bank statements are actually quite nice because they contain their own consistency checks.
The opening balance, every transaction and the closing balance all have to agree.
Running balances have to reconcile as well.
That means every extraction, whether it came from an LLM, OCR, a CSV import or an OFX parser, can go through exactly the same validation step.
If everything adds up, great.
If it doesn't, we know something is wrong before the user ever exports the data.
Sometimes it's obvious what's happened. A page is missing. A debit became a credit. A row was duplicated. Sometimes the validator can repair the issue automatically. Sometimes it can't.
Either way, the system doesn't pretend everything is fine.
That ended up changing how I think about building with AI.
The model doesn't need to be perfect.
It just needs to produce something that can be verified independently.
Some of the bugs that led us there were surprisingly mundane.
One Wise statement looked as though the description column had been extracted perfectly. Every row contained text. Unfortunately, the text was mostly internal transaction IDs. The merchant names lived elsewhere on the page.
Revolut caught me for a different reason. Their CSV exports include a Type column with values like CARD_PAYMENT
and TOPUP
. I initially assumed that was the transaction direction. It wasn't. Once I stopped trying to infer debits and credits from that field and trusted the signed amounts instead, the imports became much more reliable.
UK bank statements introduced another edge case. Quite a few of them don't use negative numbers at all. Instead they have separate Money In and Money Out columns. One parser decided the mostly empty Money In column wasn't part of the table and quietly removed every transaction direction.
Then there was an Australian OFX export that wrapped CDATA inside SGML. OFX 1.02 already isn't proper XML, so by the time I'd finished preprocessing it I'd written two tiny functions that completely eliminated what I'd assumed would require another AI pass.
That probably taught me the biggest lesson of the project.
Not every difficult-looking problem needs AI.
Sometimes it just needs twenty lines of code.
The same thing happened with scanned PDFs.
Originally we'd send every PDF through the same pipeline. If the document had no text layer, we'd waste tokens asking the model to extract text that simply wasn't there before eventually failing.
Now the first thing we do is check whether any text exists at all.
If it doesn't, we route the document straight to vision. It's faster, cheaper and produces better results.
Looking back, I don't think the interesting part of StatementDecoder is that it uses AI.
Lots of products use AI.
The interesting part is deciding where not to trust it.
The model is responsible for understanding messy documents.
Deterministic code is responsible for parsing structured formats.
The validator is responsible for deciding whether the result is believable.
Those are three separate jobs.
Keeping them separate made the system dramatically more reliable than trying to make the model do everything.
If you're building any kind of extraction pipeline, not just for bank statements, that's probably the idea I'd steal. Don't ask the model to prove it's correct.
Give it something independent to answer to.