# A Japanese NLI checker for the space between generation and trust (sealed 0.961, zero-shot baselines 0.79–0.81)

> Source: <https://dev.to/japanopendatalab/a-japanese-nli-checker-for-the-space-between-generation-and-trust-sealed-0961-zero-shot-5h52>
> Published: 2026-11-21 14:31:55+00:00

**Key takeaway:** ja-nli-v1 scores 0.9612 Macro F1 on a sealed 5,000-pair Japanese NLI test versus 0.79–0.81 for zero-shot multilingual models, at ~109 ms per pair on CPU via ONNX.

**TL;DR — We released ja-nli-v1, a small Japanese natural language inference model: given a premise and a hypothesis, it returns probabilities for entailment / neutral / contradiction. On a sealed test of 5,000 pairs it scores accuracy and Macro F1 of 0.9612; on a 600-pair human-verified set, 0.9567, with entailment↔contradiction confusions at 3 (0.5%). Two multilingual NLI models used zero-shot score 0.79–0.81 on the same sealed set. A premise-shuffle diagnostic confirms the model actually reads the premise: swap in an unrelated one and accuracy falls to ~0.38. Trained without any existing NLI dataset — licensing, not preference. Apache 2.0, ONNX included, ~109 ms per pair on CPU.**

**Model page (try the demo from there): [https://huggingface.co/japan-opendata/ja-nli-v1](https://huggingface.co/japan-opendata/ja-nli-v1)**

**Browser demo: [https://huggingface.co/spaces/japan-opendata/ja-nli-demo](https://huggingface.co/spaces/japan-opendata/ja-nli-demo)**

1. 
**A verification part, meant to sit next to generation.** Does the RAG answer contradict its source passage? Does the claim match the search result it cites? Those are premise–hypothesis questions, and in Japanese they usually get answered by a large multilingual model or not at all. This is the small, local, CPU-priced version.
2. 
**The sanity checks are the feature.** Shuffle the premise for an unrelated one and accuracy drops to ~0.38, with most outputs collapsing to neutral — the model is lost without the premise, which is what you want. A hypothesis-only probe (can it guess the label from the hypothesis alone?) was driven down to ~0.63 on the human-verified set by diversifying the training data, so the shortcut is weak. An NLI model that doesn't read its premise is a label guesser; we measured that ours does.
3. 
**No existing NLI data was trained on.** JSNLI, JNLI, JSICK, JaNLI, XNLI, ANLI — all excluded on license terms. The training pairs are synthetic, built from seed corpora limited to public-domain-licensed government documents (PDL 1.0) and MIT / Apache-2.0 Japanese OSS documentation, generated by an open-weight LLM and blind-judged by a different model. The zero-shot baselines above were measured for reference only.

## 
  
  
  Use it

The ONNX build (fp32, opset 17) matched PyTorch predictions exactly on 200 development and 200 sealed items. Speed: ~109 ms per pair on a Core i7-11700F, measured on inputs that include longer premises. No INT8 build is shipped.

## 
  
  
  Measured results

Sealed test (5,000 pairs, separated from training, single final run):

- Accuracy 0.9612, Macro F1 0.9612
- Zero-shot comparison, same conditions: two multilingual mDeBERTa-family NLI models at 0.7920 and 0.8094

Human-verified evaluation (600 pairs):

- Accuracy 0.9567; entailment↔contradiction confusions: 3 (0.5%)
- The same zero-shot models: 0.8433 each

Diagnostics:

- Premise shuffled to an unrelated text: accuracy falls to ~0.38, most predictions collapse to neutral
- Hypothesis-only probe: ~0.63 on the human-verified set (kept low by design)

## 
  
  
  Training data

Seed corpora: PDL 1.0 public documents, and Japanese OSS documentation with confirmed MIT / Apache-2.0 licenses — nothing else. Pairs were generated by an open-weight LLM and verified by blind judgment from a separate model. A random 200-pair audit found label validity 95.0% and Japanese naturalness 94.5%.

## 
  
  
  What didn't work / Limitations

- 
**The evaluation ground is, again, ours.** Both test sets are synthetic-derived, built and labeled under our pipeline. The zero-shot models were graded on Japanese pairs written to our distribution; on the standard Japanese NLI benchmarks they were built for, the comparison would look different. We did not train on those benchmarks, so we also cannot report on them as a neutral party — treat 0.9612 as performance on verified synthetic pairs, with real-world text expected to score lower.
- 
**The hypothesis-only probe is weakened, not dead.** ~0.63 is well below the full model's 0.9567, but above the 0.33 chance level — some hypothesis-side signal remains, as it does in essentially every NLI dataset ever built, including the famous ones. The premise-shuffle collapse (~0.38) is the stronger evidence that the premise drives the decision.
- 
**~109 ms is the slowest in our series so far.** Premise-plus-hypothesis pairs with long premises cost more than single short texts. Fine for checking generated answers; budget accordingly for bulk scoring.
- 
**Truncation cuts the premise, never the hypothesis.** Past 1,024 tokens, the end of a long source passage silently stops participating. For long-document checks, chunk the premise yourself rather than letting the tokenizer decide what matters.

This is the eleventh in our line of small, single-purpose Japanese models. NLI is the one where "the model got a high score" is least trustworthy on its own — so this release leads with the diagnostics that try to catch the model cheating, and publishes what they found.

*If you check generated Japanese text against its sources today: dedicated NLI model, a prompt to a big LLM, or nothing yet — and which failure worries you most, a missed contradiction or a false alarm?*

*Get the next release notes and our Kaggle experiment logs by email: [https://jpopendata.jp/](https://jpopendata.jp/) (newsletter signup in the footer).*
