A Japanese NLI checker for the space between generation and trust (sealed 0.961, zero-shot baselines 0.79–0.81) The Japan Open Data team released ja-nli-v1, a small Apache 2.0 Japanese natural language inference model that returns entailment, neutral and contradiction probabilities for premise-hypothesis pairs. It scored 0.9612 accuracy and Macro F1 on a sealed 5,000-pair test and 0.9567 on a 600-pair human-verified set, versus 0.79–0.81 for two zero-shot multilingual NLI models, running at about 109 ms per pair on CPU with an ONNX build. Trained without any existing NLI dataset for licensing reasons, the model was validated with a premise-shuffle diagnostic that drops accuracy to roughly 0.38, and the team cautions that both test sets are synthetic-derived, so real-world text should score lower. TL;DR — We released ja-nli-v1, a small Japanese natural language inference model: given a premise and a hypothesis, it returns probabilities for entailment / neutral / contradiction. On a sealed test of 5,000 pairs it scores accuracy and Macro F1 of 0.9612; on a 600-pair human-verified set, 0.9567, with entailment↔contradiction confusions at 3 0.5% . Two multilingual NLI models used zero-shot score 0.79–0.81 on the same sealed set. A premise-shuffle diagnostic confirms the model actually reads the premise: swap in an unrelated one and accuracy falls to ~0.38. Trained without any existing NLI dataset — licensing, not preference. Apache 2.0, ONNX included, ~109 ms per pair on CPU. Model page try the demo from there : https://huggingface.co/japan-opendata/ja-nli-v1 https://huggingface.co/japan-opendata/ja-nli-v1 Browser demo: https://huggingface.co/spaces/japan-opendata/ja-nli-demo https://huggingface.co/spaces/japan-opendata/ja-nli-demo 1. A verification part, meant to sit next to generation. Does the RAG answer contradict its source passage? Does the claim match the search result it cites? Those are premise–hypothesis questions, and in Japanese they usually get answered by a large multilingual model or not at all. This is the small, local, CPU-priced version. 2. The sanity checks are the feature. Shuffle the premise for an unrelated one and accuracy drops to ~0.38, with most outputs collapsing to neutral — the model is lost without the premise, which is what you want. A hypothesis-only probe can it guess the label from the hypothesis alone? was driven down to ~0.63 on the human-verified set by diversifying the training data, so the shortcut is weak. An NLI model that doesn't read its premise is a label guesser; we measured that ours does. 3. No existing NLI data was trained on. JSNLI, JNLI, JSICK, JaNLI, XNLI, ANLI — all excluded on license terms. The training pairs are synthetic, built from seed corpora limited to public-domain-licensed government documents PDL 1.0 and MIT / Apache-2.0 Japanese OSS documentation, generated by an open-weight LLM and blind-judged by a different model. The zero-shot baselines above were measured for reference only. Use it The ONNX build fp32, opset 17 matched PyTorch predictions exactly on 200 development and 200 sealed items. Speed: ~109 ms per pair on a Core i7-11700F, measured on inputs that include longer premises. No INT8 build is shipped. Measured results Sealed test 5,000 pairs, separated from training, single final run : - Accuracy 0.9612, Macro F1 0.9612 - Zero-shot comparison, same conditions: two multilingual mDeBERTa-family NLI models at 0.7920 and 0.8094 Human-verified evaluation 600 pairs : - Accuracy 0.9567; entailment↔contradiction confusions: 3 0.5% - The same zero-shot models: 0.8433 each Diagnostics: - Premise shuffled to an unrelated text: accuracy falls to ~0.38, most predictions collapse to neutral - Hypothesis-only probe: ~0.63 on the human-verified set kept low by design Training data Seed corpora: PDL 1.0 public documents, and Japanese OSS documentation with confirmed MIT / Apache-2.0 licenses — nothing else. Pairs were generated by an open-weight LLM and verified by blind judgment from a separate model. A random 200-pair audit found label validity 95.0% and Japanese naturalness 94.5%. What didn't work / Limitations - The evaluation ground is, again, ours. Both test sets are synthetic-derived, built and labeled under our pipeline. The zero-shot models were graded on Japanese pairs written to our distribution; on the standard Japanese NLI benchmarks they were built for, the comparison would look different. We did not train on those benchmarks, so we also cannot report on them as a neutral party — treat 0.9612 as performance on verified synthetic pairs, with real-world text expected to score lower. - The hypothesis-only probe is weakened, not dead. ~0.63 is well below the full model's 0.9567, but above the 0.33 chance level — some hypothesis-side signal remains, as it does in essentially every NLI dataset ever built, including the famous ones. The premise-shuffle collapse ~0.38 is the stronger evidence that the premise drives the decision. - ~109 ms is the slowest in our series so far. Premise-plus-hypothesis pairs with long premises cost more than single short texts. Fine for checking generated answers; budget accordingly for bulk scoring. - Truncation cuts the premise, never the hypothesis. Past 1,024 tokens, the end of a long source passage silently stops participating. For long-document checks, chunk the premise yourself rather than letting the tokenizer decide what matters. This is the eleventh in our line of small, single-purpose Japanese models. NLI is the one where "the model got a high score" is least trustworthy on its own — so this release leads with the diagnostics that try to catch the model cheating, and publishes what they found. If you check generated Japanese text against its sources today: dedicated NLI model, a prompt to a big LLM, or nothing yet — and which failure worries you most, a missed contradiction or a false alarm? Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ https://jpopendata.jp/ newsletter signup in the footer .