# I built a vision AI that reads Brazilian boletos and invoices, and code that checks every digit it…

> Source: <https://blog.devgenius.io/i-built-a-vision-ai-that-reads-brazilian-boletos-and-invoices-and-code-that-checks-every-digit-it-22b4b8ce2cc2?source=rss----4e2c1156667e---4>
> Published: 2026-11-04 16:15:52+00:00

Every month I copy numbers out of documents. The digit line of a boleto into the bank app, the amounts of a nota fiscal into a spreadsheet, and, once a year, every line of each broker’s informe de rendimentos into the IRPF program. Vision models can read all of these now. What they can’t do is tell me when they read one digit wrong, and a boleto with one wrong digit pays someone else, or the wrong amount.

So I built **Doc Extraction Copilot**: upload a photo or PDF of a boleto, a DANFE or an informe de rendimentos, and a vision model returns every field as typed JSON. Code then checks everything it can, and I only review the fields that failed a check or that the model said it was unsure of.

The design goals:

The stack is Next.js 16 with React 19 and TypeScript, the Vercel AI SDK 6 with Google Gemini, Zod, Tailwind, Vitest and Playwright.

The three documents have nothing in common: a boleto has nine fields, a DANFE has a list of products and five totals, and an informe has three lists. One schema for all of them would be mostly nulls, so the app first asks which kind of document it is, then extracts it with that type’s schema:

``` js
const type = opts.type ?? (await classify(file, model))if (type === 'other') throw new NotSupportedError('This does not look like a boleto, a DANFE or an informe de rendimentos.')const { output } = await generateText({  model,  output: Output.object({ schema: SCHEMAS[type] }),  system: SYSTEM,  messages: [{ role: 'user', content: [{ type: 'text', text: TYPE_HINTS[type] }, { type: 'file', data: file.data, mediaType: file.mediaType }] }],  timeout: { totalMs: 90_000 },  maxRetries: 2,})
```

Images and PDFs go in the same way, as a file part. The AI SDK turns the Zod schema into Gemini’s response schema, and validates the answer against it.

**Why the prompt forbids corrections:** the most important line of the system prompt is “Never correct a digit, a check digit, a date or a total, even if it looks wrong: separate code checks them.” A model that silently fixes a total that doesn’t add up hides exactly the problem I want to see. The schema also has an uncertain list, where the model names every field it couldn't read clearly.

Most numbers on these documents carry their own check digits, so a misread can usually be caught without knowing the right answer:

``` js
function bankBarcodeDigit(digits43: string): number {  const dv = 11 - (weightedSum11(digits43) % 11)  return dv === 0 || dv === 10 || dv === 11 ? 1 : dv}
```

One detail cost me some time. The due date is a factor: days since 1997–10–07. It reached 9999 on 2025–02–21 and started again at 1000 the next day, so the same factor now stands for two dates 9000 days apart. The code takes the one closer to today.

Every field starts as *not checked*. A check that passes marks the fields it covers as *checked*, and a check that fails, or the model listing the field as uncertain, marks it for *review*. Review always wins: one failed check is enough.

``` js
for (const check of checks) {  for (const path of check.paths) {    const f = fields.get(path)    f.notes.push(`${check.name}: ${check.detail}`)    if (!check.ok) f.status = 'review'    else if (f.status === 'unchecked') f.status = 'ok'  }}
```

This is a tampered boleto from the samples: the printed amount says R$ 2.449,98, but the digit line encodes R$ 3.062,47. Every check digit is valid, so the digit line itself passes, but the amount doesn’t match it, and it is the one field sent to review:

The JSON can only be downloaded once nothing is left to review. Correcting a field reruns the checks in the browser, and a field the checks can’t confirm can be marked “It’s right” by hand.

**Why lines that pass are trusted:** when a DANFE’s products don’t add up to the products total, the obvious move is to flag every line. But a line whose quantity × unit price equals its total passed its own check, so only the lines that didn’t, and the total, are flagged. The review then points at the line that is actually wrong.

**Why “not checked” is shown:** names and descriptions have no check digits. The app doesn’t call them confirmed, it says they weren’t checked, so I know what I’m trusting.

“It worked on my three samples” says nothing, and I won’t upload real boletos and tax documents to test it. So a script generates 36 fictional documents from a seeded random generator, 12 of each type, and renders them with headless Chromium: 35% clean PNGs, 30% PDFs and 35% phone-photo-style JPEGs, rotated, blurred, low contrast and on a table:

Every document is valid by construction: real check digits, a digit line that encodes the printed amount, totals that add up. The last two of each type have a planted inconsistency: a boleto whose printed amount is 80% of the one in its digit line, a DANFE with a line total off by R$ 10, and an informe whose exempt-income total is R$ 100 too high. Those must be flagged even when the model reads them perfectly.

The eval scores each document against its answers and reports field accuracy, exact matches, flag precision and recall, and the number that matters most: **silent errors**, wrong fields nobody was asked to look at.

I ran it with the default model and with the previous Flash Lite generation. Both read almost everything right, and in both, every wrong field was sent to review. Gemini 3.1 Flash Lite misread three access keys and one CNPJ, and the check digits caught all four. Its lower precision comes from listing correct CNPJs as uncertain, which is the safe kind of mistake.

The one misread with the default model was on a rotated photo of a utility boleto: it dropped one zero from a run of zeros, reading 83660000000-1 as 8366000000-1. That is the kind of mistake a person makes too, and the digit line check caught it because the line was one digit short.

**Why the scorer matters too:** the first run said the bank was wrong on nine of the twelve boletos. It wasn’t: the model wrote “Bradesco” or “077–9” where the label said “Bradesco 237–2”. The field is there to identify the bank, so the scorer now compares bank codes, the same way the check does. An eval can be wrong in either direction, so I read every error before trusting the totals.

These documents are synthetic and cleaner than real photos, so I read the numbers as an upper bound. What they do show is that the checks work: every planted inconsistency was caught, and no wrong field went through unflagged.

The validators are tested against published examples, like a real Banco do Brasil digit line, and against every single-digit misread of a valid line, all of which must fail. Every synthetic document must produce zero flags, and every planted one must be flagged on exactly its planted fields.

The end-to-end tests run Playwright against the production build with AI_PROVIDER=mock, a scripted model from the AI SDK's test utilities that returns each sample with one deliberate misread. The test clicks the boleto sample, sees "1 field to review", types the amount from the check's note, and expects the download to be enabled. CI runs all of it without an API key.

I wrote it with an AI coding assistant, and the decisions stayed mine: that the model reads and code judges, the review rules, which checks matter for each document, and building an eval set before trusting any number. The assistant was fastest at the parts with clear specs, like the FEBRABAN check digit rules and the document templates, and the tests caught it when it got them wrong.

There are 35 Vitest tests and 4 Playwright tests.

Documents are sent to Google’s Gemini API, and the free tier may use them to improve Google’s products, so I only use fictional documents on a free key. Names and descriptions can’t be checked, only read. The eval is synthetic.

Next, I want a harder eval set with real-world photo problems (glare, folds, cut-off edges), comparisons with other vision models, and an export to the IRPF program’s import format.

The code is on GitHub: [github.com/Eric-Souza/doc-extraction-copilot](https://github.com/Eric-Souza/doc-extraction-copilot). If you’ve built document extraction for Brazilian documents, or have a boleto layout it gets wrong, I would like to hear about it in the comments.

[I built a vision AI that reads Brazilian boletos and invoices, and code that checks every digit it…](https://blog.devgenius.io/i-built-a-vision-ai-that-reads-brazilian-boletos-and-invoices-and-code-that-checks-every-digit-it-22b4b8ce2cc2) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.