FinRCA-Bench finds financial AI diagnosis depends heavily on evidence retrieval

FinRCA-Bench, a deterministic synthetic benchmark introduced in an arXiv preprint by Pratik Ghawate on Aug. 19, 2026, finds that financial AI diagnosis depends heavily on evidence retrieval, with macro required-record recall rising from 0.83% to 77.70% and exact accuracy across 16 classes from 2.05% to 72.44% when retrieval was improved. The benchmark, containing 2,250 synthetic cases with 1,500 injected failures across 15 causal categories, reports structural retrieval failures outnumbered reasoning failures 95 to 15, and strict returned-evidence contract accuracy was only 5.72%, suggesting that final answer accuracy alone overstates the quality of the supporting process.

What happened The arXiv preprint FinRCA-Bench introduces a deterministic synthetic benchmark for accounts-payable-to-bank reconciliation. It tests whether financial AI systems can retrieve the right records and identify the underlying cause of a transaction failure, while evaluating retrieval separately from reasoning. The arXiv record identifies FinRCA-Bench as a preprint submitted on Aug. 19, 2026, by Pratik Ghawate. Its stated purpose is to separate two capabilities that can be blurred in an end-to-end financial AI score: finding the evidence needed to diagnose a problem and reasoning from that evidence. The benchmark focuses on accounts-payable-to-bank reconciliation, where relevant information is distributed across invoices, purchase orders, approvals, allocations, payments, ledger entries, and bank activity. The source says these records are connected by transactional relationships rather than simply by textual similarity. The benchmark contains 2,250 deterministic synthetic cases. Of those, 1,500 contain injected failures spanning 15 causal categories, while 750 are legitimate cases or hard negatives. The source says the model does not see the root-cause labels or the record-level evidence contracts. Those hidden contracts allow the benchmark to assess whether a system retrieved the records required for a diagnosis, rather than measuring only whether its final label happened to be correct. The supplied source does not describe the individual failure categories, the data-generation process in detail, or how closely the synthetic records represent particular accounting systems. The study compares several approaches: Rules/SQL, classical machine learning, dense semantic retrieval, deterministic relational expansion, and Typed Provenance Graph Retrieval, or TPGR. The source describes TPGR as a typed traversal restricted to persisted transaction relationships. It reports that Rules/SQL reached 84.97% held-out exact accuracy and classical machine learning reached 95.44%. In a separate comparison, the researchers held the reasoning model, prompt, and generation settings fixed while changing only retrieval. Under that setup, macro required-record recall rose from 0.83% to 77.70%, while exact accuracy across 16 classes rose from 2.05% to 72.44%. The abstract reports that structural retrieval failures outnumbered reasoning failures when sufficient retrieval was available, by 95 to 15. It also reports 254 correct predictions despite incomplete retrieval, showing why a final answer alone may overstate the quality of the supporting process. Strict returned-evidence contract accuracy was only 5.72%. These are claims made by the preprint about its own benchmark; the supplied source does not independently verify the implementation, code, data, statistical analysis, or reported results. The arXiv page lists associated code and data, but the source text does not provide enough information to assess their availability or completeness. Read the primary source: arxiv.org ↗ https://arxiv.org/abs/2608.18534 Why it matters The benchmark’s results suggest that financial AI performance can be dominated by access to linked evidence rather than by the reasoning model alone. That distinction matters for systems expected to support decisions that must be checked against invoices, approvals, payments, ledgers, and bank activity. The practical significance of the paper is its separation of answer correctness from evidence completeness. A system can produce the right root-cause label while missing records that an auditor, accountant, or investigator would need to verify the conclusion. FinRCA-Bench’s reported 254 correct predictions with incomplete retrieval and 5.72% strict evidence-contract accuracy illustrate that these are not interchangeable outcomes within the benchmark. The source therefore challenges the use of a single end-to-end accuracy number as a complete description of financial AI reliability. The reported retrieval sensitivity also changes where an organization would look for failure. If the source’s benchmark results generalize, improving the language model or adjusting its prompt may not address a system whose main weakness is the way it traverses linked financial records. Retrieval design, transaction identifiers, relationship types, provenance, and coverage of required records could become as important to diagnosis quality as the model generating the explanation. This is an implication of the benchmark’s controlled comparison, not evidence that any particular deployed finance system has the same failure profile. The findings matter beyond reconciliation because they offer a framework for evaluating systems that must justify conclusions from distributed records. The benchmark hides both the cause and the evidence requirements, then scores retrieval separately from the final classification. That structure makes it possible to ask whether a system found the records it was supposed to use, whether it returned enough of them, and whether its conclusion remained correct when evidence was incomplete. The source does not claim that FinRCA-Bench applies to every financial workflow or to other industries, so broader conclusions remain unestablished. There are also important limits to the evidence. The cases are synthetic and deterministic, not demonstrated examples drawn from live accounting or banking operations. The source does not report how the benchmark handles changing schemas, missing records, duplicated transactions, access controls, confidential data, multilingual documents, or ambiguous human judgments. It also does not establish whether the reported classical machine-learning and Rules/SQL results are directly comparable to the retrieval-and-reasoning experiment in deployment cost, maintenance burden, latency, or explainability. The results support a research finding about this benchmark, not a general certification of any approach. What to watch next The central question is whether FinRCA-Bench’s findings hold beyond its synthetic cases. The source does not establish peer review, independent replication, performance on real financial records, or the operational cost and reliability of the proposed retrieval approaches. The first test is reproducibility. The arXiv record says code and data are associated with the article, but the supplied source does not identify a repository, license, instructions, baseline implementations, or independent replication. Those details would determine whether other researchers can recreate the 0.83% and 77.70% retrieval figures, the 2.05% and 72.44% exact-accuracy figures, and the evidence-contract results under the same conditions. The next test is external validity. Future evaluations should show whether the retrieval gap persists on real or independently constructed reconciliation data, across different enterprise schemas and document formats, and under realistic missing-data and permissions constraints. It would also be important to test whether the 15 injected causal categories cover the kinds of failures encountered in operational finance, or whether performance changes when cases are authored by organizations other than the benchmark’s creator. None of those facts is established by the supplied source. Evaluation practice is another area to watch. FinRCA-Bench suggests that reporting only a final root-cause label can hide incomplete evidence retrieval. Comparable systems may need separate measures for required-record recall, exact diagnosis, evidence-contract compliance, and errors attributable to retrieval or reasoning. The benchmark’s own result—that structural failures outnumbered reasoning failures with sufficient retrieval by 95 to 15—makes that separation a central claim to test, rather than an assumption to accept across all financial AI applications. Finally, deployment decisions should account for the difference between a plausible answer and an auditable record of how that answer was supported. Organizations considering AI for reconciliation would need evidence about access controls, data retention, latency, maintenance, human review, and the consequences of an incorrect or insufficiently supported diagnosis. The source provides no results on those operational questions, no user study, and no evidence of adoption. Until such work is available, FinRCA-Bench is best treated as a focused research benchmark with potentially important implications, not as proof that a particular retrieval architecture is ready for production.