{"slug": "hakembench-a-turkish-benchmark-of-typed-decisions", "title": "HakemBench: A Turkish Benchmark of Typed Decisions", "summary": "HakemBench, a Turkish benchmark of typed decisions released under CC BY 4.0 with 2,346 items and 4,275 choice, yes/no and score questions across seven tracks, scores a leader at a composite of 0.888 while the lab's own model ranks 7th at 0.660 on a board of 16 rows. The benchmark's gold labels come mostly from blind passes of one AI model family compared with votes from a panel of large language models from other families and are not human-verified, and the lab's own model is not blind because earlier runs' test results shaped its training data, flagging its guardrail, moderation and customer support numbers. Scored on the other four tracks only, that model's composite falls to 0.678, 6th of 16.", "body_md": "arXiv:2610.02293v1 Announce Type: new \nAbstract: HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0, with 2,346 items and 4,275 choice, yes/no and score questions in seven tracks (fact-check triage, education, guardrails, legal routing, moderation, spam and phishing, and customer support). One harness scores decision quality (macro F1), calibration (from the normalised Brier score) and selective automation (from the normalised area under the generalised risk-coverage curve), combines them by a geometric mean and reports intervals from 2,000 bootstrap draws; probes for option order, paraphrase, English translation and substituted names are reported alongside. Most gold labels come from blind passes of one AI model family compared with the votes of a panel of large language models from other model families; they are not human-verified. On a board of 16 rows the leader scores a composite of 0.888 and the lab's own model is 7th at 0.660. Its numbers are not blind. Earlier runs' test results shaped its training data, so its guardrail, moderation and customer support numbers are flagged; with every model scored on the other four tracks only, its composite is 0.678, 6th of 16.", "url": "https://wpnews.pro/news/hakembench-a-turkish-benchmark-of-typed-decisions", "canonical_source": "https://arxiv.org/abs/2610.02293", "published_at": "2026-10-05 04:00:00+00:00", "updated_at": "2026-10-05 04:14:38.199802+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "natural-language-processing"], "entities": ["HakemBench", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/hakembench-a-turkish-benchmark-of-typed-decisions", "markdown": "https://wpnews.pro/news/hakembench-a-turkish-benchmark-of-typed-decisions.md", "text": "https://wpnews.pro/news/hakembench-a-turkish-benchmark-of-typed-decisions.txt", "jsonld": "https://wpnews.pro/news/hakembench-a-turkish-benchmark-of-typed-decisions.jsonld"}}