Most LLM evals reward accuracy. In financial documents the expensive mistake is different: confirming a wrong number.
I built KoFinClaim-Abstain — 500 Korean numeric claims from fictional filings. For each claim + 2–4 evidence sentences, a system must accept, reject, or hand off to a human. Scoring is by cost: a false accept costs 10, a review costs 1.
Five open models answering alone passed 7–30% of wrong claims (EXAONE 3.5 7.8B: 36/500, Qwen3 8B: 149/500). Adding a rule-based checker that re-verifies quotes, numbers, units, periods and conflicts cut Qwen3-14B’s false accepts from 46 to 1 — in-sample, and at the price of sending many correct answers to review. Both caveats are in the card.
Answers were hash-sealed before any run. CC BY 4.0. Feedback on the scoring and the labels is welcome.