{"slug": "ultrabert-jev-encoder-models-for-system-one-applications", "title": "UltraBERT-JEV: Encoder models for System One Applications", "summary": "A technical report on arXiv (2611.17867v1) finds that a fine-tuned UltraBERT-380M encoder scored 69.7% to 72.1% test accuracy on single-pass option selection, versus 67.6% for Qwen3-0.6B trained on the same data with the LLM-as-Jev recipe. Scoring each option by \"yes\" minus \"no\" rather than \"yes\" alone produced the larger gain, lifting accuracy from 69.7% to 71.3% on the test set and from 57.8% to 59.6% on external benchmarks, where the decoder reached 56.1%; sigmoid normalisation added 0.8 test points while a square-root softplus added none. On a single NVIDIA A100 processing one request at a time, the encoder took 29 ms per decision against 68 ms for the decoder but used more peak memory, 2.8 GiB versus 2.2 GiB, and an auxiliary head trained on human-marked evidence and labels selected with a reward adapted from SelfCite predicts in the same pass which sentences support the answer.", "body_md": "arXiv:2611.17867v1 Announce Type: new \nAbstract: We study encoder models that pick one of several options in a single forward pass, without generating text. This technical report compares recipes for fine-tuning UltraBERT-380M with one mask per option. The four recipes differ in how an option is scored and how the scores become probabilities; one follows the expert gate of DeepSeek-V3. Trained on the same data as Qwen3-0.6B with the LLM-as-Jev recipe (67.6% test accuracy), the encoder reaches 69.7% to 72.1%. Scoring an option by \"yes\" minus \"no\" instead of \"yes\" alone gives the larger gain, from 69.7% to 71.3% on the test set and from 57.8% to 59.6% on external benchmarks (56.1% for the decoder). Sigmoid normalisation adds 0.8 test points; a square-root softplus adds none. One request at a time on an NVIDIA A100, the encoder takes 29 ms per decision against 68 ms, but uses more peak memory (2.8 against 2.2 GiB). An auxiliary head, trained on human-marked evidence and on labels selected with a reward adapted from SelfCite, predicts in the same pass the sentences annotated as support for the answer.", "url": "https://wpnews.pro/news/ultrabert-jev-encoder-models-for-system-one-applications", "canonical_source": "https://www.machinebrief.com/news/ultrabert-jev-encoder-models-for-system-one-applications-pux5", "published_at": "2026-11-16 05:00:00+00:00", "updated_at": "2026-11-16 05:51:25.313499+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-research", "large-language-models"], "entities": ["UltraBERT-380M", "Qwen3-0.6B", "DeepSeek-V3", "SelfCite", "NVIDIA A100", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ultrabert-jev-encoder-models-for-system-one-applications", "markdown": "https://wpnews.pro/news/ultrabert-jev-encoder-models-for-system-one-applications.md", "text": "https://wpnews.pro/news/ultrabert-jev-encoder-models-for-system-one-applications.txt", "jsonld": "https://wpnews.pro/news/ultrabert-jev-encoder-models-for-system-one-applications.jsonld"}}