mrs83 1
I was in the middle of a seven-day vacation, trying to disconnect, when my phone buzzes. Multiple contacts message me the exact same question: “What do you think about Jev?”
Away from my workstation, my first instinct was to check with Gemini on my mobile phone. The frontier model’s verdict? It politely told me I was probably hallucinating.
Fair enough. I opened LinkedIn to see what was actually going on, and sure enough, my feed was already flooded with people hyping it as the next revolutionary breakthrough in AI/ML. I then decided to take a look.
Lots of massive efficiency claims, but very little evaluation data. Plenty of words framing it as a "new paradigm” but thin technical detail on what is actually happening under the hood.
The concept is practical, but doesn’t look much different from what we already have.
Which makes me wonder: did someone actually build a genuine architectural breakthrough, or did they just repackage classic classification? Or it is just an LLM wrapper with clever positioning and marketing?
What do you think about Jev?
Screenshot from Linkedin Post: Jev does not remotely assign probabilities correctly even for the most simple examples. However, I like that they introduce that notion and future models will probably perform better on this. Or do I… | Florian Hönicke | 84 comments
nootxlm 2
Good question — and there’s enough independent measurement out there now to separate the two halves of the claim.
The structural part is real: Jev is genuinely non-autoregressive — one parallel pass, no text generation — so the speed and cost numbers come from the architecture, not the positioning. It’s not a wrapper. The accuracy-and-calibration part is where the measurement matters more than the marketing.
The sharpest check on any probability model is its calibration curve: the stated probabilities should match observed frequencies. On that, the independent checks disagree with the “calibrated” claim — one public claims-audit repo recomputed the calibration error against each study’s own noise floor and found it running 2–2.5× too high, with the probabilities compressed toward the middle (that’s the same symptom as the Florian Hönicke screenshot you linked). Interestingly, a single-parameter fix cut most of that away, which suggests the mechanism is fine and the calibration just isn’t finished.
So I’d read it as: genuinely fast classification with genuinely unfinished calibration. The best next step is the boring one — take your own task, score Jev against a plain fine-tuned small model (someone got ~93% on BANKING77 with a 4B model in about 30 minutes vs ~80% for Jev alone), and set your decision threshold off the measured calibration, not the brochure. If the numbers hold on your data, the speed is the actual product.
mrs83 3
Thanks! This confirms my read on the calibration trade offs.
Honestly, welcome back machine learning? I love this pivot back to narrow and structured tasks!!
I am tired by the endless parade of all-purpose multimodal instruction-tuned LLMs … anything resembling mid-2010s applied ML brings me joy
I’m heads-down on a few things through the end of the year, but I’d love to explore this direction further with folks in the EU timezone around January