{"slug": "jev-and-the-return-of-ai-ml-engineering", "title": "Jev and the Return of AI/ML Engineering", "summary": "Independent testing found Jev's calibrated-probability claims fail in practice, with Valeriy M reporting calibration failures on 7 of 8 datasets across 16,500 predictions and the author's own coin-toss, two-dice and three UCI dataset experiments showing high expected calibration error. The author argues Jev's other three promises — structured outputs, low latency and low cost — are reproducible with off-the-shelf models, citing Eric Zhang of Modal building a Jev-compatible API with Qwen-3.6-35b-a3b and SGLang, and Matt Mastracci of Google matching Jev's latency with DiffusionGemma and vLLM on DGX Spark. The author concludes most capable teams should build their own models rather than rely on Jev's RLCD calibration.", "body_md": "# Jev and the return of AI/ML Engineering\n\nJev looks interesting, but its differentiator, probability caliberation, is underwhelming. Most would be better off using fast and customizable open source LLM instead, e.g.,Qwen-3.5-2b, or Flash models from DeepSeek and Qwen for those with GPUs.\n\nJev is marked as a 'System One' model. That's marketing. From thinking fast and slow, 'fast' models is system one, and 'slow' models is system 2. System one models ranges from plain 'o linear regression all the way to Transformers and future models. Ignoring this branding, Jev has 4 main promises:\n\n1. Calibrated Decisions, or caliberated probabilities.\n2. Structured outputs\n3. Low latency\n4. Low cost\n\nThe latter three can be replicated with off-the-shelf models.\n\n## Cost and latency.\n\nSmaller LLM have similar cost and latency profile vs Jev. Eric Zhang of @modal built a Jev-compatible API w/ Qwen-3.6-35b-a3b and SGLang.\n\nMatt Mastracci of Google it w/ DiffusionGemma and VLLM with similar latency vs Jev on DGX Spark.\n\nSo, fast \"decision\" APIs are reproducible with existing models. At the end of the day, for small prefills, the latency and cost gap between autoregressive and non-autoregressive models is negligible.\n\n## Structured outputs\n\nStructured outputs is a capability trained into all autoregressive language models since 2023 in the early days of function calling. Jev provides JSON as its output response with choice (choose an option from the list), score (score the state based on a rubric), and noul (boolean true/false), each with its answer, probability, and confidence. This is a similar abstractions to Delip Rao's [AutoRubrics](https://autorubric.org/) that's going to make a showing at COLM 2026 in San Francisco.\n\n## Calibrated decisions\n\nPerhaps I am not applied or product minded enough, or i've been thinking too much about evaluations and alignments, the most useful application for Jev is rubrics / preference model with its calibrated probabilities for LLM evaluation and alignments or search engine reranking and optimization. Its API design agrees with my intuition.\n\nBut I am skeptical about its broad calibration claims, which it calls RLCD (calibrated decisions). What does it calibrate to? Calibrations is highly dependent on the known probability of the distribution q. Valeriy M ran 16.5k predictions across 8 datasets and found caliberation failures on 7/8.\n\nMy own experiments, with coin toss, two-dice, and three separate UCI datasets shows that Jev has high expected caliberation errors (ECE). This is especially pronounced once we venture out of simulation based experiments.\n\nThere is real product value here. Pedram use it to run Poker Arena, [Shengyao Zhuang](https://www.linkedin.com/in/shengyaozhuangit/) used it as a re-ranker, and Peter Wang used it to play Warcraft 3. And if you play Wracraft, go look at the video and observe that damn Mountain King; its basically randomly jerking around doing random actions, and there's no sign of intelligence. I suspect most of these capabilities originate from Jev's base model, not from its RLCD calibration.\n\nMore reliable and calibrated probabilities is a huge durable advantage. This translates directly into better decision when paired with thresholds. This calibration is not yet observable with Jev, and most capable teams should be building their own models. Welcome to the return of AI/ML Engineering.", "url": "https://wpnews.pro/news/jev-and-the-return-of-ai-ml-engineering", "canonical_source": "https://leehanchung.github.io/blogs/2026/09/20/jev-return-ai-ml-engineering/", "published_at": "2026-09-27 13:48:51+00:00", "updated_at": "2026-09-27 14:01:30.599546+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-products"], "entities": ["Jev", "Qwen-3.5-2b", "Qwen-3.6-35b-a3b", "DeepSeek", "Eric Zhang", "Modal", "Matt Mastracci", "Google"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/jev-and-the-return-of-ai-ml-engineering", "markdown": "https://wpnews.pro/news/jev-and-the-return-of-ai-ml-engineering.md", "text": "https://wpnews.pro/news/jev-and-the-return-of-ai-ml-engineering.txt", "jsonld": "https://wpnews.pro/news/jev-and-the-return-of-ai-ml-engineering.jsonld"}}