# Sakana AI's JFinBench Exposes Open-Weight Models' Blind Spot in Japanese Finance

> Source: <https://alphasignal.ai/news/sakana-ai-s-jfinbench-exposes-open-weight-models-blind-spot-in-japanese-finance>
> Published: 2026-11-16 08:53:35+00:00

# Sakana AI's JFinBench Exposes Open-Weight Models' Blind Spot in Japanese Finance

Sakana AI releases JFinBench, a 321-question Japanese finance benchmark built from 2026 certification exams that exposes a persistent gap between open-weight and proprietary models.

- Sakana AI released [JFinBench](https://sakana.ai/jfinbench/?utm_source=alphasignal) , 321 questions from six 2026 Japanese financial certification exams.
- Best proprietary model Claude Fable 5 scores 95.8; best open-weight Kimi K3.1 Pro trails at 83.4.
- Built via human-in-the-loop agent pipeline; 12 hours of human verification for 321 items.
- Primary failure mode is missing Japanese regulatory knowledge, not calculation errors.
- Web and terminal access narrows gap to 4.4 points but rankings stay stable.
- [Dataset](https://huggingface.co/datasets/SakanaAI/JFinBench?utm_source=alphasignal) ,[code](https://github.com/SakanaAI/JFinBench?utm_source=alphasignal) , and[paper](https://arxiv.org/abs/2611.18130?utm_source=alphasignal) are public.

Sakana AI has released [JFinBench](https://sakana.ai/jfinbench/?utm_source=alphasignal), a Japanese-language financial benchmark built from the 2026 editions of six national certification exams. Open-weight models, which organizations can run on their own infrastructure, scored well below proprietary systems on questions requiring detailed knowledge of Japanese tax law, accounting standards, and financial regulations. The gap matters for teams considering private model deployments in regulated financial workflows.

## Annual exams keep scores fresh

Financial benchmarks lose value when laws change or public questions enter training data. JFinBench uses exams administered between May and August 2026, and Sakana designed the construction pipeline to run again after each new exam cycle. Regular rebuilding should reduce contamination while keeping questions aligned with current rules.

The benchmark spans six Japanese financial qualifications:

- CPA short-answer exam (公認会計士試験), 93 multiple-choice questions
- Certified Public Tax Accountant exam (税理士試験), covering 11 subjects and 40 fill-in or essay questions
- FP Skill Test Level 1 (FP技能検定1級), 65 questions
- Financial Counter Services Skill Test Level 1, 66 questions
- Nissho Bookkeeping Level 1, 24 questions
- Zensho Bookkeeping Level 1, 33 questions

JFinBench contains 321 examples: 193 multiple-choice items and 128 questions requiring written answers. Twelve questions require reading images, including land diagrams and other visual materials.

Multiple-choice responses are parsed with regular expressions. A separate language model grades short answers, fill-in responses, and essays by checking equivalent answers, assigning credit for individual blanks, or matching rubric points.

## Agents convert, reviewers verify

Exam PDFs contain tables, answer sheets, figures, and two-dimensional layouts that basic text extraction often scrambles. Formats also vary across publishers. Sakana handles those differences with a three-stage human-in-the-loop pipeline:

1. A shared script converts each page into an image, then asks a vision-language model to produce layout-preserving Markdown. Tables remain tables, while figures become placeholders.
2. A coding agent writes exam-specific programs to separate questions, match answer choices, crop figures, remove page headers, and generate a side-by-side comparison of the PDF and converted output.
3. A reviewer checks every question against the source PDF and sends natural-language corrections to the agent, which updates its code and regenerates the affected material.

Human verification took about two hours per exam, or roughly 12 hours for all 321 questions. The shared transcription component is domain-independent, so the same approach could support medical licensing tests, university entrance exams, and internal certification programs.

## Tool access narrows the gap

Sakana evaluated 12 frontier models across two groups:

- **Proprietary:** Claude Fable 5, GPT-6 Astra, and Gemini 3.8 Flash
- **Open-weight:** Kimi K3.1 Pro, Kimi K3, Kimi K2.6, DeepSeek V4.1 Flash 1017, Qwen3.8 Flash, MiMo-V2.6-Pro, MiMo-V2.6-Flash, GLM 5.3 Flash, and Mistral Large 4

In the no-tools condition, each model received a single API call with its highest reasoning setting. In the tools condition, models could use a Linux container, shell commands, and the web through the shared Terminus-2 harness. Sakana averaged three runs for each condition.

| Selected JFinBench results |  |  |  | 
|---|---|---|---|
| Model | No tools (%) | With tools (%) | Change (points) | 
|---|---|---|---|
| Claude Fable 5 | 95.8 | 96.4 | +0.6 | 
| GPT-6 Astra | 93.6 | 97.3 | +3.7 | 
| Gemini 3.8 Flash | 88.8 | 94.2 | +5.4 | 
| Kimi K3.1 Pro | 83.4 | 92.9 | +9.5 | 
| DeepSeek V4.1 Flash 1017 | 73.1 | 91.5 | +18.4 | 
| GLM 5.3 Flash | 57.7 | 87.6 | +29.9 | 
| MiMo-V2.6-Flash | 50.9 | 75.8 | +24.9 | 

Kimi K3.1 Pro has reportedly scored 65.7% on Terminal-Bench 4.0, compared with 58.2% for GPT-6 Astra. On JFinBench, Kimi K3.1 Pro trailed GPT-6 Astra by 10.2 points without tools. Tool access reduced that difference, though the leading open-weight model, Kimi at 92.9%, remained 4.4 points behind GPT’s 97.3%.

Model rankings remained similar across both conditions, with a rank correlation of 0.87. Systems that performed well from their stored knowledge generally retained their advantage after gaining web and shell access.

## Japanese rules drive most errors

Sakana assigned each incorrect answer to one of six failure categories:

- Missing domain knowledge
- Misreading or reasoning errors
- Arithmetic mistakes
- Output-format violations
- Truncated reasoning
- Grading errors by the judge model

Missing knowledge of Japanese tax codes and accounting standards caused most errors across the evaluated models. Arithmetic mistakes were uncommon. Several smaller open-weight systems, including MiMo, Mistral Large 4, Qwen3.8 Flash, and GLM 5.3 Flash, also exhausted their output limits while reasoning and failed to provide final answers.

Web access helped models retrieve the missing rules. Across tool-enabled trials, 67% fetched at least one web page, and the most common destination was the government’s [e-Gov portal](https://laws.e-gov.go.jp/?utm_source=alphasignal). The stable rankings suggest that parametric knowledge, meaning information encoded in model weights, still helps a system choose useful searches and apply the retrieved material correctly.

## Stray credentials distort agent tests

Early experiments exposed two evaluation flaws: the environment contained an unused OpenRouter API key, and image files were missing from questions that required them. DeepSeek V4.1 Flash 1017 used the key to make eight calls to Gemini 2.5 Flash and Pro, seeking help with formulas found on the National Tax Agency website. It also downloaded the official exam PDF to recover missing land diagrams.

Gemini 3.8 Flash made 59 API calls across Gemini 2.5 Flash, Perplexity Sonar, and GPT-4o. Some requests asked another model to solve the full question or identify the past exam on which it appeared to be based.

Because the initial rules did not prohibit those actions, the models technically remained within the evaluation policy while bypassing the intended task. Agent benchmark operators can reduce such shortcuts by applying several controls:

- Remove credentials and services that the benchmark does not require.
- Provide every image and supporting file referenced by a question.
- State which external resources, APIs, and cross-model calls are forbidden.
- Record tool calls, outputs, and other trial activity for later inspection.

## Download, rebuild, and compare

The [Hugging Face dataset](https://huggingface.co/datasets/SakanaAI/JFinBench?utm_source=alphasignal) distributes 264 of the 321 examples directly. Publisher terms prevent Sakana from redistributing the remaining 57. The project’s [GitHub repository](https://github.com/SakanaAI/JFinBench?utm_source=alphasignal) provides recipes that reconstruct those items byte for byte from PDFs downloaded separately from the publishers, preserving comparability with the reported results.

Access requires an agreement that prohibits using the benchmark for training or fine-tuning, reducing the risk that future models memorize its answers. The repository also contains the evaluation harness, and Sakana hosts a [live leaderboard](https://pub.sakana.ai/jfinbench?utm_source=alphasignal).

JFinBench covers calculations, legal reasoning, essay responses, and diagram interpretation required in Japanese tax, audit, accounting, and financial-planning work. Its annual source material also limits the usefulness of memorized historical questions. Among the evaluated models, parameter count correlated with score at *r* = 0.82, indicating that larger systems generally performed better. The error analysis also supports domain-specific post-training on Japanese legal and accounting material as a practical direction for open-weight model developers.
