Sakana AI's JFinBench Exposes Open-Weight Models' Blind Spot in Japanese Finance Sakana AI released JFinBench, a 321-question Japanese financial benchmark drawn from six 2026 national certification exams, and the best proprietary model, Claude Fable 5, scored 95.8 against 83.4 for the top open-weight model, Kimi K3.1 Pro. Sakana AI reported that the primary failure mode for open-weight models is missing Japanese regulatory knowledge rather than calculation errors, and that giving models web and terminal access narrowed the gap to 4.4 points without changing the rankings. The benchmark, built through a human-in-the-loop agent pipeline that took about 12 hours of human verification for all 321 questions, is public on Hugging Face with code on GitHub and a paper on arXiv (2611.18130). Sakana AI's JFinBench Exposes Open-Weight Models' Blind Spot in Japanese Finance Sakana AI releases JFinBench, a 321-question Japanese finance benchmark built from 2026 certification exams that exposes a persistent gap between open-weight and proprietary models. - Sakana AI released JFinBench https://sakana.ai/jfinbench/?utm source=alphasignal , 321 questions from six 2026 Japanese financial certification exams. - Best proprietary model Claude Fable 5 scores 95.8; best open-weight Kimi K3.1 Pro trails at 83.4. - Built via human-in-the-loop agent pipeline; 12 hours of human verification for 321 items. - Primary failure mode is missing Japanese regulatory knowledge, not calculation errors. - Web and terminal access narrows gap to 4.4 points but rankings stay stable. - Dataset https://huggingface.co/datasets/SakanaAI/JFinBench?utm source=alphasignal , code https://github.com/SakanaAI/JFinBench?utm source=alphasignal , and paper https://arxiv.org/abs/2611.18130?utm source=alphasignal are public. Sakana AI has released JFinBench https://sakana.ai/jfinbench/?utm source=alphasignal , a Japanese-language financial benchmark built from the 2026 editions of six national certification exams. Open-weight models, which organizations can run on their own infrastructure, scored well below proprietary systems on questions requiring detailed knowledge of Japanese tax law, accounting standards, and financial regulations. The gap matters for teams considering private model deployments in regulated financial workflows. Annual exams keep scores fresh Financial benchmarks lose value when laws change or public questions enter training data. JFinBench uses exams administered between May and August 2026, and Sakana designed the construction pipeline to run again after each new exam cycle. Regular rebuilding should reduce contamination while keeping questions aligned with current rules. The benchmark spans six Japanese financial qualifications: - CPA short-answer exam 公認会計士試験 , 93 multiple-choice questions - Certified Public Tax Accountant exam 税理士試験 , covering 11 subjects and 40 fill-in or essay questions - FP Skill Test Level 1 FP技能検定1級 , 65 questions - Financial Counter Services Skill Test Level 1, 66 questions - Nissho Bookkeeping Level 1, 24 questions - Zensho Bookkeeping Level 1, 33 questions JFinBench contains 321 examples: 193 multiple-choice items and 128 questions requiring written answers. Twelve questions require reading images, including land diagrams and other visual materials. Multiple-choice responses are parsed with regular expressions. A separate language model grades short answers, fill-in responses, and essays by checking equivalent answers, assigning credit for individual blanks, or matching rubric points. Agents convert, reviewers verify Exam PDFs contain tables, answer sheets, figures, and two-dimensional layouts that basic text extraction often scrambles. Formats also vary across publishers. Sakana handles those differences with a three-stage human-in-the-loop pipeline: 1. A shared script converts each page into an image, then asks a vision-language model to produce layout-preserving Markdown. Tables remain tables, while figures become placeholders. 2. A coding agent writes exam-specific programs to separate questions, match answer choices, crop figures, remove page headers, and generate a side-by-side comparison of the PDF and converted output. 3. A reviewer checks every question against the source PDF and sends natural-language corrections to the agent, which updates its code and regenerates the affected material. Human verification took about two hours per exam, or roughly 12 hours for all 321 questions. The shared transcription component is domain-independent, so the same approach could support medical licensing tests, university entrance exams, and internal certification programs. Tool access narrows the gap Sakana evaluated 12 frontier models across two groups: - Proprietary: Claude Fable 5, GPT-6 Astra, and Gemini 3.8 Flash - Open-weight: Kimi K3.1 Pro, Kimi K3, Kimi K2.6, DeepSeek V4.1 Flash 1017, Qwen3.8 Flash, MiMo-V2.6-Pro, MiMo-V2.6-Flash, GLM 5.3 Flash, and Mistral Large 4 In the no-tools condition, each model received a single API call with its highest reasoning setting. In the tools condition, models could use a Linux container, shell commands, and the web through the shared Terminus-2 harness. Sakana averaged three runs for each condition. | Selected JFinBench results | | | | |---|---|---|---| | Model | No tools % | With tools % | Change points | |---|---|---|---| | Claude Fable 5 | 95.8 | 96.4 | +0.6 | | GPT-6 Astra | 93.6 | 97.3 | +3.7 | | Gemini 3.8 Flash | 88.8 | 94.2 | +5.4 | | Kimi K3.1 Pro | 83.4 | 92.9 | +9.5 | | DeepSeek V4.1 Flash 1017 | 73.1 | 91.5 | +18.4 | | GLM 5.3 Flash | 57.7 | 87.6 | +29.9 | | MiMo-V2.6-Flash | 50.9 | 75.8 | +24.9 | Kimi K3.1 Pro has reportedly scored 65.7% on Terminal-Bench 4.0, compared with 58.2% for GPT-6 Astra. On JFinBench, Kimi K3.1 Pro trailed GPT-6 Astra by 10.2 points without tools. Tool access reduced that difference, though the leading open-weight model, Kimi at 92.9%, remained 4.4 points behind GPT’s 97.3%. Model rankings remained similar across both conditions, with a rank correlation of 0.87. Systems that performed well from their stored knowledge generally retained their advantage after gaining web and shell access. Japanese rules drive most errors Sakana assigned each incorrect answer to one of six failure categories: - Missing domain knowledge - Misreading or reasoning errors - Arithmetic mistakes - Output-format violations - Truncated reasoning - Grading errors by the judge model Missing knowledge of Japanese tax codes and accounting standards caused most errors across the evaluated models. Arithmetic mistakes were uncommon. Several smaller open-weight systems, including MiMo, Mistral Large 4, Qwen3.8 Flash, and GLM 5.3 Flash, also exhausted their output limits while reasoning and failed to provide final answers. Web access helped models retrieve the missing rules. Across tool-enabled trials, 67% fetched at least one web page, and the most common destination was the government’s e-Gov portal https://laws.e-gov.go.jp/?utm source=alphasignal . The stable rankings suggest that parametric knowledge, meaning information encoded in model weights, still helps a system choose useful searches and apply the retrieved material correctly. Stray credentials distort agent tests Early experiments exposed two evaluation flaws: the environment contained an unused OpenRouter API key, and image files were missing from questions that required them. DeepSeek V4.1 Flash 1017 used the key to make eight calls to Gemini 2.5 Flash and Pro, seeking help with formulas found on the National Tax Agency website. It also downloaded the official exam PDF to recover missing land diagrams. Gemini 3.8 Flash made 59 API calls across Gemini 2.5 Flash, Perplexity Sonar, and GPT-4o. Some requests asked another model to solve the full question or identify the past exam on which it appeared to be based. Because the initial rules did not prohibit those actions, the models technically remained within the evaluation policy while bypassing the intended task. Agent benchmark operators can reduce such shortcuts by applying several controls: - Remove credentials and services that the benchmark does not require. - Provide every image and supporting file referenced by a question. - State which external resources, APIs, and cross-model calls are forbidden. - Record tool calls, outputs, and other trial activity for later inspection. Download, rebuild, and compare The Hugging Face dataset https://huggingface.co/datasets/SakanaAI/JFinBench?utm source=alphasignal distributes 264 of the 321 examples directly. Publisher terms prevent Sakana from redistributing the remaining 57. The project’s GitHub repository https://github.com/SakanaAI/JFinBench?utm source=alphasignal provides recipes that reconstruct those items byte for byte from PDFs downloaded separately from the publishers, preserving comparability with the reported results. Access requires an agreement that prohibits using the benchmark for training or fine-tuning, reducing the risk that future models memorize its answers. The repository also contains the evaluation harness, and Sakana hosts a live leaderboard https://pub.sakana.ai/jfinbench?utm source=alphasignal . JFinBench covers calculations, legal reasoning, essay responses, and diagram interpretation required in Japanese tax, audit, accounting, and financial-planning work. Its annual source material also limits the usefulness of memorized historical questions. Among the evaluated models, parameter count correlated with score at r = 0.82, indicating that larger systems generally performed better. The error analysis also supports domain-specific post-training on Japanese legal and accounting material as a practical direction for open-weight model developers.