Two AI readers agreed—and both were wrong

My workflow in TransferDue extracts unit numbers, floor areas, and prices from Thai property documents. A misread digit can still look perfectly reasonable, so I added a second reading. Then two readers agreed on the same wrong answer.

Gemini read the number correctly, but my code accepted the answer with the most votes.

I changed the rule for accepting an answer. Across 315 checks of extracted values, wrong answers accepted without a warning fell from 57 to 35—a 38.6% reduction. Correct answers barely increased. More mistakes became visible, but more values also needed manual checking.

A wrong unit number, floor area, or sale price can change how someone understands a transaction. I wanted extra checks on those values.

The first model reads the document and extracts the numbers. A second model reads it separately, without seeing the first answer. If they disagree, the workflow can ask for a third reading. Gemini also reads the documents, giving me an answer from a different model family.

By “model family,” I mean related models, such as Claude models from Anthropic. Running them in separate sessions keeps them from copying each other's answers. It doesn't stop them from misreading the same character.

Here is one example. The unit number on a poor-quality image of a Thai condo title deed was 8B:

The first two Claude readings agreed, so the workflow would skip its third Claude reader. Its decision came down to two votes for 4B and one for 8B. It accepted 4B.

The extra Claude reading in the table came from a separate recorded check. It wasn't part of that decision, but it showed why asking Claude again wouldn't have helped in this case.

I stopped treating repeated answers from one model family as enough reason to overrule another family.

If the Claude readers agree but Gemini disagrees, the system can now flag the value for checking. In the code, that status is called `family_conflict`.

For the 8B example, the report still keeps the first answer, **4B**, but warns that the readings disagree. **It hasn't found the right answer. It has stopped presenting the wrong one as checked.**

I didn't make Gemini the deciding vote either. Gemini can also be wrong. In this example, its answer alone cannot overturn the two matching Claude answers. If Mistral also reads 8B, two different families support that value, allowing the rules to correct it. Otherwise, this disagreement stays a warning.

A Python script applies these rules. Its decision depends on the recorded values and which readers produced them.

I compared the old and new rules using the same saved model answers. That lets me see what the rule change did without new model responses changing the comparison.

There were 315 checks of extracted values, such as a unit number or sale price. Each check ended with a correct answer, a wrong answer with or without a warning, or no answer.

The new rules produced 22 fewer wrong answers without warnings. They corrected only one additional answer. Most of the benefit came from making mistakes visible.

But a warning creates work for whoever checks it:

That's 30 more warnings, including nine more correct answers that now need checking. A disagreement doesn't prove an answer is wrong. It means the system doesn't have enough support to call it checked.

The second family matters. When I replayed the same answers with Gemini's readings removed, wrong answers without a warning rose to 66, more than the 57 under the old rules. Most of the warnings came from Gemini disagreeing.

The models read the documents and return values. I give each reader the information it needs without showing it the answers it is supposed to check.

The business rules decide whether to accept, correct, or flag a value. These rules are ordinary code, so I can change them and compare the results using saved answers. I don't need another model to judge whether two readings agree.

The orchestrator runs the steps: it starts readers in parallel, waits for their results, and requests another reading when the rules call for one. It saves the results so I can see whether a mistake came from a document reading, an acceptance rule, or a step that failed to run.

Gemini gives me a reading from a different model family. That doesn't guarantee a correct answer, but it can reveal a mistake that repeated Claude readings miss.

When the readings still disagree, an optional Mistral OCR step can help. OCR means reading text from document images. Mistral reads the relevant documents, and the script checks its answers against the others. If it returns nothing useful or the answers still disagree, the warning stays.

Another reader can also make the decision worse. Adding Mistral's saved readings increased correct answers, but it also caused three more wrong answers to pass without warnings. That tradeoff matters more than the number of models in the workflow.

Model choices and the Mistral comparison The current workflow uses Claude Opus 5.5 to extract important figures, read them again for checking, and assess risks. Claude Sonnet 5.5 classifies documents, extracts identity details, runs other checks, and writes the report. Gemini 3.5 Flash provides readings from a different model family.

I chose this combination after comparing models and document-reading tools on my Thai documents, including clear documents and poor scans. It gave me the best results for these tasks among the options I tried.

In the comparison using earlier saved answers, adding Mistral readings for the flagged values changed the results:

Mistral produced 19 additional correct answers. It also added three wrong answers without warnings, all on degraded generated scans. On the clean PDFs, it produced 10 additional correct answers without adding any wrong answers accepted without warnings.

Every new model generation needs another check. A newer model might read more values correctly but make a new kind of mistake, or repeat mistakes made by the other readers. I need to check both the individual models and the rules that combine their answers. An upgrade can make the overall result better or worse.

The screenshots and video below show real models processing generated sample documents in Docker. They show a separate run from the earlier 8B/4B example. This time, the readers agreed on the unit number but disagreed on the deposit amount and title-deed number.

The checking script skipped the third Claude reading because the first two agreed. The script that combines all the readings checks disagreements with Gemini later.

The run took about five minutes, including the first document check. It kept the deposit amount and title-deed number unchanged and marked both for review.

Gemini returned useful readings. The Mistral step returned no usable readings, shown in the execution record as zero_reads, so it didn't resolve either disagreement.

TransferDue's admin view shows the two warnings:

The buyer's report explains what to do next:

The report finished, but it didn't hide the parts that still needed checking.

Here is a 30-second walkthrough of the completed run and report:

English captions are burned into the video. No audio. The video shows inspection after the run finished.

[Open the full-size workflow image](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/qq410o524pz4yrhly7sa.png).

For this workflow, I chose more warnings over silently accepting disputed numbers. The change reduced wrong answers without warnings from 57 to 35, but also flagged nine additional correct answers. That's the tradeoff: fewer hidden mistakes, more manual checking.

Adding more readers only helps if the rules handle their answers sensibly. I need to decide what counts as agreement, when another reading is worth requesting, and when the system should ask someone to check the original document.

If you build document pipelines, how do you tell genuine agreement from the same mistake repeated twice? How much extra checking are you willing to add? I run this pipeline with NRFLO, the open-source workflow engine I build. It handles parallel reads, conditional steps, and saved results; the document-checking rules belong to TransferDue. The document-intake example is a smaller starting point for combining scripts and model steps.

About the examples: Sample documents go through the same upload and processing steps as buyer documents; their accuracy can differ. The numerical comparison uses saved answers from earlier models on generated and real documents across repeated runs, and covers the rules for accepting extracted numbers. It doesn't measure overall report accuracy or the current Opus 5.5 / Sonnet 5.5 workflow shown in the captures.