{"slug": "two-ai-readers-agreed-and-both-were-wrong", "title": "Two AI readers agreed—and both were wrong", "summary": "A developer building TransferDue, a workflow that extracts unit numbers, floor areas and prices from Thai property documents, found that two AI readers from the same model family can agree on the same wrong value, after Claude twice read a condo title deed unit number as 4B while Gemini correctly read 8B. The developer changed the acceptance rule so agreement within one model family no longer overrules a different family, flagging such cases as family_conflict; across 315 saved checks, wrong answers accepted without a warning fell from 57 to 35 (38.6%), while warnings rose from 13 to 43 and correct answers improved by only one.", "body_md": "My workflow in [TransferDue](https://transferdue.com/how-it-works?utm_source=devto&utm_medium=referral&utm_campaign=launch2026&utm_content=two-ai-readers-20261030) extracts unit numbers, floor areas, and prices from Thai property documents. A misread digit can still look perfectly reasonable, so I added a second reading. Then two readers agreed on the same wrong answer.\n\nGemini read the number correctly, but my code accepted the answer with the most votes.\n\nI changed the rule for accepting an answer. Across 315 checks of extracted values, **wrong answers accepted without a warning fell from 57 to 35—a 38.6% reduction**. Correct answers barely increased. More mistakes became visible, but more values also needed manual checking.\n\nA wrong unit number, floor area, or sale price can change how someone understands a transaction. I wanted extra checks on those values.\n\nThe first model reads the document and extracts the numbers. A second model reads it separately, without seeing the first answer. If they disagree, the workflow can ask for a third reading. Gemini also reads the documents, giving me an answer from a different model family.\n\nBy “model family,” I mean related models, such as Claude models from Anthropic. Running them in separate sessions keeps them from copying each other's answers. It doesn't stop them from misreading the same character.\n\nHere is one example. The unit number on a poor-quality image of a Thai condo title deed was **8B**:\n\n| Reader | Unit number | \n|---|---|\n| First Claude reading | 4B | \n| Second Claude reading | 4B | \n| Extra Claude reading, saved separately and not counted in this decision | 4B | \n| Gemini | 8B | \n| Value on the document | **8B** | \n\nThe first two Claude readings agreed, so the workflow would skip its third Claude reader. Its decision came down to two votes for **4B** and one for **8B**. It accepted **4B**.\n\nThe extra Claude reading in the table came from a separate recorded check. It wasn't part of that decision, but it showed why asking Claude again wouldn't have helped in this case.\n\nI stopped treating repeated answers from one model family as enough reason to overrule another family.\n\nIf the Claude readers agree but Gemini disagrees, the system can now flag the value for checking. In the code, that status is called `family_conflict`.\n\nFor the 8B example, the report still keeps the first answer, **4B**, but warns that the readings disagree. **It hasn't found the right answer. It has stopped presenting the wrong one as checked.**\n\nI didn't make Gemini the deciding vote either. Gemini can also be wrong. In this example, its answer alone cannot overturn the two matching Claude answers. If Mistral also reads 8B, two different families support that value, allowing the rules to correct it. Otherwise, this disagreement stays a warning.\n\nA Python script applies these rules. Its decision depends on the recorded values and which readers produced them.\n\nI compared the old and new rules using the same saved model answers. That lets me see what the rule change did without new model responses changing the comparison.\n\nThere were **315 checks of extracted values**, such as a unit number or sale price. Each check ended with a correct answer, a wrong answer with or without a warning, or no answer.\n\n| Outcome | Old rules | New rules | \n|---|---|---|\n| Correct answer | 238 | 239 | \n| Wrong answer, no warning | 57 | 35 | \n| Wrong answer, with a warning | 12 | 33 | \n| No answer | 8 | 8 | \n| Total | 315 | 315 | \n\nThe new rules produced **22 fewer wrong answers without warnings**. They corrected only one additional answer. Most of the benefit came from making mistakes visible.\n\nBut a warning creates work for whoever checks it:\n\n| Review workload | Old rules | New rules | \n|---|---|---|\n| Values with a warning | 13 | 43 | \n| Correct values with a warning | 1 | 10 | \n\nThat's 30 more warnings, including nine more correct answers that now need checking. A disagreement doesn't prove an answer is wrong. It means the system doesn't have enough support to call it checked.\n\nThe second family matters. When I replayed the same answers with Gemini's readings removed, wrong answers without a warning rose to **66**, more than the 57 under the old rules. Most of the warnings came from Gemini disagreeing.\n\nThe models read the documents and return values. I give each reader the information it needs without showing it the answers it is supposed to check.\n\nThe business rules decide whether to accept, correct, or flag a value. These rules are ordinary code, so I can change them and compare the results using saved answers. I don't need another model to judge whether two readings agree.\n\nThe orchestrator runs the steps: it starts readers in parallel, waits for their results, and requests another reading when the rules call for one. It saves the results so I can see whether a mistake came from a document reading, an acceptance rule, or a step that failed to run.\n\nGemini gives me a reading from a different model family. That doesn't guarantee a correct answer, but it can reveal a mistake that repeated Claude readings miss.\n\nWhen the readings still disagree, an optional **Mistral OCR** step can help. OCR means reading text from document images. Mistral reads the relevant documents, and the script checks its answers against the others. If it returns nothing useful or the answers still disagree, the warning stays.\n\nAnother reader can also make the decision worse. Adding Mistral's saved readings increased correct answers, but it also caused three more wrong answers to pass without warnings. That tradeoff matters more than the number of models in the workflow.\n\nModel choices and the Mistral comparison\nThe current workflow uses **Claude Opus 5.5** to extract important figures, read them again for checking, and assess risks. **Claude Sonnet 5.5** classifies documents, extracts identity details, runs other checks, and writes the report. **Gemini 3.5 Flash** provides readings from a different model family.\n\nI chose this combination after comparing models and document-reading tools on my Thai documents, including clear documents and poor scans. It gave me the best results for these tasks among the options I tried.\n\nIn the comparison using earlier saved answers, adding Mistral readings for the flagged values changed the results:\n\n| Outcome | New rules | New rules + Mistral | \n|---|---|---|\n| Correct answer | 239 | 258 | \n| Wrong answer, no warning | 35 | 38 | \n| Wrong answer, with a warning | 33 | 11 | \n| No answer | 8 | 8 | \n| Total | 315 | 315 | \n\nMistral produced 19 additional correct answers. It also added three wrong answers without warnings, all on degraded generated scans. On the clean PDFs, it produced 10 additional correct answers without adding any wrong answers accepted without warnings.\n\nEvery new model generation needs another check. A newer model might read more values correctly but make a new kind of mistake, or repeat mistakes made by the other readers. I need to check both the individual models and the rules that combine their answers. An upgrade can make the overall result better or worse.\n\nThe screenshots and video below show **real models processing generated sample documents** in Docker. They show a separate run from the earlier 8B/4B example. This time, the readers agreed on the unit number but disagreed on the deposit amount and title-deed number.\n\n*The checking script skipped the third Claude reading because the first two agreed. The script that combines all the readings checks disagreements with Gemini later.*\n\nThe run took about five minutes, including the first document check. It kept the deposit amount and title-deed number unchanged and marked both for review.\n\nGemini returned useful readings. The Mistral step returned no usable readings, shown in the execution record as `zero_reads`, so it didn't resolve either disagreement.\n\nTransferDue's admin view shows the two warnings:\n\nThe buyer's report explains what to do next:\n\nThe report finished, but it didn't hide the parts that still needed checking.\n\nHere is a 30-second walkthrough of the completed run and report:\n\n*English captions are burned into the video. No audio. The video shows inspection after the run finished.*\n\n[Open the full-size workflow image](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/qq410o524pz4yrhly7sa.png).\n\nFor this workflow, I chose more warnings over silently accepting disputed numbers. The change reduced wrong answers without warnings from 57 to 35, but also flagged nine additional correct answers. That's the tradeoff: fewer hidden mistakes, more manual checking.\n\nAdding more readers only helps if the rules handle their answers sensibly. I need to decide what counts as agreement, when another reading is worth requesting, and when the system should ask someone to check the original document.\n\nIf you build document pipelines, how do you tell genuine agreement from the same mistake repeated twice? How much extra checking are you willing to add?\n\nI run this pipeline with [NRFLO](https://github.com/nrflo/nrflo), the open-source workflow engine I build. It handles parallel reads, conditional steps, and saved results; the document-checking rules belong to TransferDue. The [document-intake example](https://github.com/nrflo/nrflo/blob/master/doc/mixed-workflow.md) is a smaller starting point for combining scripts and model steps.\n\n**About the examples:** Sample documents go through the same upload and processing steps as buyer documents; their accuracy can differ. The numerical comparison uses saved answers from earlier models on generated and real documents across repeated runs, and covers the rules for accepting extracted numbers. It doesn't measure overall report accuracy or the current Opus 5.5 / Sonnet 5.5 workflow shown in the captures.", "url": "https://wpnews.pro/news/two-ai-readers-agreed-and-both-were-wrong", "canonical_source": "https://dev.to/anderfred/two-ai-readers-agreed-and-both-were-wrong-2ghe", "published_at": "2026-10-30 09:09:33+00:00", "updated_at": "2026-10-30 09:20:08.556037+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools"], "entities": ["TransferDue", "Gemini", "Claude", "Anthropic", "Mistral"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/two-ai-readers-agreed-and-both-were-wrong", "markdown": "https://wpnews.pro/news/two-ai-readers-agreed-and-both-were-wrong.md", "text": "https://wpnews.pro/news/two-ai-readers-agreed-and-both-were-wrong.txt", "jsonld": "https://wpnews.pro/news/two-ai-readers-agreed-and-both-were-wrong.jsonld"}}