{"slug": "jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different", "title": "Jev vs a 310M encoder I trained myself: 750 rows, three tasks, two different winners", "summary": "A developer fine-tuned a 310M-parameter Japanese encoder (sbintuitions/modernbert-ja-310m) on 250 labeled rows per task and found it beat TypeSafe's Jev decision API by 12.0 points on 9-class livedoor topic classification (McNemar p=0.00007) while running 4–20× faster, but only tied Jev on the two polarity tasks. Zero-shot open-source classifiers fared worse across the board, with a GLiClass multilang-mini model scoring 62.8/87.2/44.8 versus Jev's 76.8/94.4/74.0. The developer concludes that task shape, not model trendiness, determines whether a small trained encoder or a zero-shot decision API wins.", "body_md": "**TL;DR**: I ran three Japanese classification tasks (250 rows each, same gold labels) through six systems. A 310M encoder I fine-tuned on 250 labels beat TypeSafe's Jev on topic classification by **+12.0 points** (McNemar p=0.00007) and was **4–20× faster** — but on the two polarity tasks it only tied Jev. The zero-shot open-source models everyone is writing about lost everywhere: a GLiNER-family classifier scored 62.8 / 87.2 / 44.8 against Jev's 76.8 / 94.4 / 74.0. The same architecture family, trained on my labels, scored 88.8 / 92.8 / 75.2.\n\n**One-line summary**: A decision API wins when you have no labels; a small trained encoder wins when you have a few hundred — and which one wins on a given task is decided by the shape of the task, not by which model is trending.\n\nJev (TypeSafe AI's System One) is genuinely good, and it is cheap: $0.042 per 1M input tokens with output free, roughly 32k of context, a confidence number in the response, and no training data required. If you read the last two weeks of posts about it, the conclusion is \"use Jev.\"\n\nI use Jev. But I had a pile of results from my own tasks that did not fit that conclusion, and one of them was embarrassing enough that I re-ran it twice:\n\nA 5-minute linear model beating a decision API on one task and losing badly on the next is not a story about either model. It is a story about **task shape**. So I designed the experiment I should have run first: hold the answers fixed, hold the rows fixed, and separate the two axes that actually differ — *labels or no labels*, and *topic or meaning*.\n\nThree Japanese datasets, 250 rows each, frozen:\n\n| Task | Shape | Classes | Chance | \n|---|---|---|---|\n| livedoor news corpus | long-document topic (mean 1,174 chars) | 9 | 11.1% | \n| rakuten reviews | short-text polarity (mean 138 chars) | 2 | 50.0% | \n| chabsa | financial-sentence polarity (mean 92 chars) | 3 | 33.3% | \n\nSix systems, two conditions:\n\n`sbintuitions/modernbert-ja-310m` with a classification head, and a char n-gram + logistic regression baseline.\nThe trained numbers are **5-fold cross-validated out-of-fold predictions**: train on 200, predict the held-out 50, five times, so all 250 reported predictions come from a model that never saw that row. That is the same protocol I use for the linear baseline, which means the trained row and the baseline row are directly comparable.\n\nEverything is compared on **identical row ids** with **McNemar's test**. This matters: at n=250 the independent-confidence interval is about ±6 points, so \"0.888 vs 0.944\" cannot be called a difference without a paired test.\n\n| System | Condition | livedoor (9) | rakuten (2) | chabsa (3) | p50 latency | \n|---|---|---|---|---|---|\n| **ModernBERT-ja-310m** | **trained, 250 labels** | **88.8%** | 92.8% | **75.2%** | **0.10–0.45s** | \n| Jev 1.13.0 | zero-shot | 76.8% | **94.4%** | 74.0% | 1.9–2.4s | \n| Gemma 4 26B-A4B (local) | zero-shot | 65.2% | **94.4%** | **75.2%** | 1.24s | \n| SemIf (Qwen3.5-4B) | zero-shot | 68.8% | 92.0% | 64.0% | 1.3–6.8s | \n| GLiClass multilang-mini | zero-shot | 62.8% | 87.2% | 44.8% | 0.04–0.42s | \n| Laya (typed decisions) | zero-shot | 29.2% | 60.0% | 40.0% | — | \n| char n-gram + LogReg | trained, 250 labels | 88.4% | 73.6% | 64.0% | ~0s | \n\nMcNemar, same ids:\n\n| Comparison | livedoor | rakuten | chabsa | \n|---|---|---|---|\n| **trained encoder − Jev** | **+12.0pt (p=0.00007)** | −1.6pt (p=0.50) | +1.2pt (p=0.83) | \n| trained encoder − local 26B MoE | +23.6pt (p<1e-15) | −1.6pt (p=0.45) | ±0.0pt (p=1.00) | \n| trained encoder − SemIf | +20.0pt (p<1e-15) | +0.8pt (p=0.83) | +11.2pt (p=0.007) | \n| **trained encoder − GLiClass (zero-shot)** | **+26.0pt (p<1e-15)** | +5.6pt (p=0.016) | **+30.4pt (p<1e-15)** | \n| trained encoder − char n-gram LogReg | +0.4pt | +19.2pt | +11.2pt | \n\nThis is the part I did not expect, and it is the most useful thing in the table.\n\nGLiClass is a GLiNER-family classifier — same lineage as the models currently being pitched as open-source Jev alternatives. Zero-shot, it scored **62.8 / 87.2 / 44.8**: it lost to Jev on all three tasks, and on the 3-class polarity task it was barely above chance (44.8% against 33.3%).\n\nTrained on 250 of my labels, the same generation of model scored **88.8 / 92.8 / 75.2**. The gap between zero-shot and trained is **+26.0 / +5.6 / +30.4 points**, all significant.\n\nSo \"is there an open-source model that replaces Jev?\" was the wrong question. The models are not the variable — **the labels are**. A 310M encoder with 250 labels beats a decision API on one task and ties it on two; the same architecture with zero labels loses to everything.\n\nI also paid for this lesson twice. Before this experiment I ran a proper evaluation of the two models most often named as Jev replacements — SemIf and Laya — on these same rows. **Both failed to replace anything.** Laya scored 29.2% on a 9-class task (chance 11.1%) with a median confidence of 0.029, i.e. it was visibly not deciding. Its zero-shot English number in its own documentation is 0.362, so this is not a Japanese-specific defect — it is a zero-shot defect. SemIf was closer but lost on all three tasks, two of them significantly.\n\nIndependent benchmarks showed the same thing. On a frozen-question benchmark, Laya's \"hard\" score was 34.1% while its model card implied a win over Jev, because the model had been fine-tuned on that benchmark's training split. Self-reported comparisons of fine-tuned models are not evidence about zero-shot behaviour.\n\nLook at where the 12-point win came from and where it did not.\n\nThe practical rule I now use: **if the label is visible in the words, train something small; if the label is a judgement about the words, use a decision API; if both plateau around 75%, stop shopping for models and fix the data or add an escalation stage.**\n\nLatency, single-threaded, same machine:\n\n|  | p50 per item | \n|---|---|\n| trained encoder (310M, CPU) | **0.10–0.45s** | \n| Jev | 1.9–2.4s | \n| local 26B MoE | 1.24s | \n| SemIf (4B) | 1.3–6.8s | \n\nThat is **4–20× faster than the API**, with zero marginal cost, no network dependency, and no data leaving the machine. The confidence score is a softmax you can calibrate yourself rather than a vendor's number.\n\nThis is the part that changes architecture. Once the local model is both faster and not worse, the API becomes the *fallback* rather than the primary path: trained encoder first, Jev only on the low-confidence tail, the novel-looking rows, and as a drift detector.\n\nI tried that too, on the same mini-PC (Ryzen 9 7940HS, Radeon 780M iGPU, `gfx1103`). Short answer: **it works, and it is not worth switching to.**\n\nGetting PyTorch onto that iGPU is now possible through AMD's device-specific wheels, but there are three traps in a row:\n\n`nvidia-*` packages.`rocm-bootstrap` version that the same index does not ship.\nOnce pinned properly, the iGPU is real: `torch 2.8.0+rocm7.14.0a20260519`, `arch=gfx1103`, fp16 matmul fine. The microbenchmark is impressive — **1.21 s/step against 16.41 s/step on CPU, 13.6×**.\n\nThe real workload does not deliver that:\n\n| Task (5-fold CV, 310M encoder) | CPU fp32 | iGPU fp16 | \n|---|---|---|\n| long documents (512 tokens) | 112 min | **44.5 min (2.5×)** | \n| short sentences | 21 min | 23 min ( **0.9×** ) | \n\nTwo reasons the microbenchmark lies. Short sequences leave the GPU idle while the CPU is comparatively efficient at small matrices, and per-fold fixed costs (model load, tokenisation, one-at-a-time evaluation) can exceed the training itself. Only long-document training showed a real gain.\n\nAnd it is not stable by default. Roughly every few minutes the driver resets the GPU — `amdgpu: ring gfx_0.0.0 timeout` → `GPU reset(8)` → then the process dies with `HIP error: unspecified launch failure`. The GPU edge temperature was 49°C, so this is a driver problem, not a thermal one. Serialising kernel dispatches and pinning the performance level plus running one fold per process with retries did get a full 5-fold run through cleanly, but fp16 also cost 2.8–4.4 points of accuracy (not significant at n=250, but negative on both tasks).\n\nSo: CPU for training, GPU for what already works — Vulkan inference. If you are training on your own private data, a documented 30 minutes per dataset on CPU is better than a 2.5× speedup that crashes.\n\nIf trained beats zero-shot this decisively, the obvious move is to generate labels with Jev and train on those. That works, with one trap I want to be explicit about, because it is the reason I did not just ship the first version:\n\n**If Jev alone produced the labels, Jev's accuracy is the ceiling of the student.** You are teaching a model to reproduce another model's mistakes. The fix is to spend the labels where they matter: run Jev over everything, then take the **low-confidence** rows and have a stronger model or a human relabel them. That is where the ceiling comes off, and it is also where the marginal cost is, because high-confidence rows can be accepted as-is.\n\nAt 250 labels the trained encoder is already at parity with Jev on the tasks Jev is strong at, and ahead where Jev is weak. Labels are the lever, and the second and third hundred labels are worth more than the first.\n\n| Situation | Use | \n|---|---|\n| No labels, short text, judge-shaped task | **Jev** (74–94% zero-shot, no setup) | \n| No labels, must stay offline or in-machine | local 26B MoE (ties Jev on both polarity tasks) | \n| A few hundred labels, topic-shaped task | **train a small encoder** (beats Jev by 12pt, 4–20× faster) | \n| A few hundred labels, meaning task | trained encoder, but accept parity with Jev | \n| Everything plateaus at ~75% | stop model-shopping; escalate the low-confidence tail | \n| Labels to be generated | Jev for volume, a stronger model or a human on the low-confidence rows | \n\n**Takeaway**: Zero-shot and trained are different products, not better and worse versions of the same one. If you have no labels, a decision API is the only thing that works today. If you have a few hundred labels and the label is visible in the text, a 310M encoder will beat it, cost nothing to run, and answer in a tenth of a second. Measure your task's shape before you pick a winner — and be suspicious of any \"open-source Jev replacement\" benchmark where the model was trained on the benchmark.\n\n**The API used here**: TypeSafe AI's Jev (System One, `jev-latest`) — text plus a list of questions in, a 0–1 score per question out, own confidence included, roughly 32k of context, $0.042 per 1M input tokens with output free. Both controlled and randomised question sets were frozen and committed before the runs, and every comparison above is on identical row ids.\n\nIf you have run the same shape of experiment on your own data, I want to know where a trained small model beat the API and where it did not — especially on tasks where both plateaued.", "url": "https://wpnews.pro/news/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different", "canonical_source": "https://dev.to/ikkun1222/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different-winners-242e", "published_at": "2026-09-21 02:20:55+00:00", "updated_at": "2026-09-21 02:52:53.509619+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "ai-research", "ai-tools"], "entities": ["TypeSafe AI", "Jev", "sbintuitions/modernbert-ja-310m", "GLiClass", "GLiNER", "Gemma 4 26B-A4B", "SemIf", "Qwen3.5-4B"], "alternates": {"html": "https://wpnews.pro/news/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different", "markdown": "https://wpnews.pro/news/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different.md", "text": "https://wpnews.pro/news/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different.txt", "jsonld": "https://wpnews.pro/news/jev-vs-a-310m-encoder-i-trained-myself-750-rows-three-tasks-two-different.jsonld"}}