{"slug": "kofinclaim-abstain-a-korean-benchmark-that-scores-llms-by-the-cost-of-false-on", "title": "KoFinClaim-Abstain: a Korean benchmark that scores LLMs by the cost of false accepts on financial numbers", "summary": "A new benchmark, KoFinClaim-Abstain, scores large language models on the cost of false accepts rather than raw accuracy, using 500 Korean numeric claims drawn from fictional financial filings where a false accept costs 10 and a human review costs 1. Five open models answering alone passed 7–30% of wrong claims, with EXAONE 3.5 7.8B accepting 36 of 500 and Qwen3 8B accepting 149 of 500, while adding a rule-based checker that re-verifies quotes, numbers, units, periods and conflicts cut Qwen3-14B's false accepts from 46 to 1 in-sample, at the cost of sending many correct answers to review. Answers were hash-sealed before any run and the benchmark is released under CC BY 4.0.", "body_md": "Most LLM evals reward accuracy. In financial documents the expensive mistake is different: confirming a wrong number.\n\nI built KoFinClaim-Abstain — 500 Korean numeric claims from fictional filings. For each claim + 2–4 evidence sentences, a system must accept, reject, or hand off to a human. Scoring is by cost: a false accept costs 10, a review costs 1.\n\nFive open models answering alone passed 7–30% of wrong claims (EXAONE 3.5 7.8B: 36/500, Qwen3 8B: 149/500). Adding a rule-based checker that re-verifies quotes, numbers, units, periods and conflicts cut Qwen3-14B’s false accepts from 46 to 1 — in-sample, and at the price of sending many correct answers to review. Both caveats are in the card.\n\nAnswers were hash-sealed before any run. CC BY 4.0. Feedback on the scoring and the labels is welcome.", "url": "https://wpnews.pro/news/kofinclaim-abstain-a-korean-benchmark-that-scores-llms-by-the-cost-of-false-on", "canonical_source": "https://discuss.huggingface.co/t/kofinclaim-abstain-a-korean-benchmark-that-scores-llms-by-the-cost-of-false-accepts-on-financial-numbers/191616#post_1", "published_at": "2026-11-10 01:25:37+00:00", "updated_at": "2026-11-10 01:49:40.172905+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["KoFinClaim-Abstain", "EXAONE 3.5 7.8B", "Qwen3 8B", "Qwen3-14B"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/kofinclaim-abstain-a-korean-benchmark-that-scores-llms-by-the-cost-of-false-on", "markdown": "https://wpnews.pro/news/kofinclaim-abstain-a-korean-benchmark-that-scores-llms-by-the-cost-of-false-on.md", "text": "https://wpnews.pro/news/kofinclaim-abstain-a-korean-benchmark-that-scores-llms-by-the-cost-of-false-on.txt", "jsonld": "https://wpnews.pro/news/kofinclaim-abstain-a-korean-benchmark-that-scores-llms-by-the-cost-of-false-on.jsonld"}}