{"slug": "benchmarking-jev-laya-and-five-open-models-under-three-stresses", "title": "Benchmarking Jev, Laya, and five open models under three stresses", "summary": "TypeSafe's Jev decision model answered 60% of items correctly at 128 candidate answers, versus 39% for Laya and 41% for the best open model, in a benchmark of seven systems run on a frozen dataset of 200 items per domain with pre-registered pass/fail rules. Jev also lost the least accuracy per doubling of the candidate list (-0.043, versus -0.073 for Laya and -0.048 for the best open model) and the least when distractors were made plausible (-0.105, versus -0.351 for Laya and -0.256 for the best open model), but changed its answer on 14.6% of decisions when option order was shuffled, while two open models changed 0.0% because they cannot. The author notes the idea appears older than Jev, and that Jev remains the better implementation by a wide margin.", "body_md": "Seven systems do the same job: take a piece of text, a question, and a list of candidate answers, and return a probability over those candidates. I measured them against each other as that job gets harder in the three ways it gets harder in production. The candidate list grows, the option order changes, and the wrong answers stop being obvious.\n\n**Short version.** TypeSafe's\n[Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) held up\nbest as the list grew and best when the wrong answers got plausible. Shuffle the\noption order, though, and it changes its answer on one decision in seven. Two of\nthe open models never change theirs, because they cannot.\n\n| at 64 to 128 candidates | Jev | [Laya](https://huggingface.co/convaiinnovations/laya) | best open | \n|---|---|---|---|\n| accuracy at 128 candidates | **60%** | 39% | 41% | \n| accuracy lost per doubling of the list | **-0.043** | -0.073 | -0.048 | \n| accuracy lost when distractors get hard | **-0.105** | -0.351 | -0.256 | \n| answers changed by reordering alone | 14.6% | 49.4% | **0.0%** | \n\nJev's makers call it a new class of model. Laya's author has said publicly that he published the same idea a year earlier. On the first claim the numbers are not kind to the marketing, and on the second they are not kind to Laya: the idea does look older than Jev, and Jev is still the better implementation of it by a wide margin.\n\nThe other five models are open ones I added so those two numbers would mean something. Most comparisons of these systems report one accuracy number at one candidate-set size, which hides most of what matters, because the ranking changes depending on how many options you offer.\n\nEverything ran on one frozen dataset, with the comparisons and the pass/fail\nrules written down before the first call ([PLAN.md](https://github.com/gazelle93/decision-models-under-pressure/blob/main/PLAN.md)). n is 200 items per\ndomain, which is enough to separate the large effects below and not enough for\nthe small ones. Every caveat is load-bearing.\n\nEvery model gets worse as the candidate list grows. The difference is how fast.\n\nJev starts highest and stays highest. At 128 candidates it answers 60% correctly, where Laya manages 39% and the best open model 41%. Chance at that list length is 0.8%, so everything here is doing real work; the question is how much of it survives a longer list.\n\nIts decline per doubling of the list is the shallowest of the seven, shallower even than the embedding scorers whose design is supposed to help them scale. Measured over the same range for every model (K=2 to 128, since Jev's API refuses more than 255 options):\n\n| model | accuracy lost per doubling of the candidate list | \n|---|---|\n| Jev | **-0.043** | \n| gte-large | -0.048 | \n| bge-large | -0.051 | \n| gliclass | -0.064 | \n| deberta-large | -0.070 | \n| deberta-base | -0.070 | \n| Laya | -0.073 | \n\nNotice how little a single-number benchmark would tell you. At two candidates Laya sits second of seven and trails Jev by two points. At 128 it has fallen to fourth and trails by twenty-two.\n\nNext the wrong answers had to get harder. Every item exists in two versions: one\nwhere the distractors come from unrelated domains, and one where they are the\nright answer's nearest neighbours, so `create alarm` competes against\n`delete alarm` and `snooze alarm` rather than against `musical work`. I matched\nthe surface shape of the words between the two versions so only meaning\nseparates them. The gap between them is how much a model was leaning on the\nwrong answers being obvious.\n\nAt 64 candidates Jev drops from 96.8% to 86.3%, losing about a tenth of what it had. Laya drops from 90.5% to 56.0% and gliclass from 87.0% to 51.2%, each losing something closer to four tenths. That is the widest spread in the whole experiment and the one I'd care most about in production, where candidate sets are full of near misses.\n\nThen I shuffled the options. Same question, same candidates, five different orderings, and I counted how often the answer changed.\n\nThis is where it helps to know how each model reads its candidate list, because that single design choice predicts the result almost perfectly:\n\n| model | how it reads the candidates | \n|---|---|\n| Jev | undisclosed, but its answers depend on the order, so not one at a time | \n| Laya | all candidates in one pass, as a set | \n| `gliclass-large-v3.0` | the same | \n| `deberta-v3-base-zeroshot-v2.0` | one pass per candidate, each scored against the text alone | \n| `deberta-v3-large-zeroshot-v2.0` | the same, larger | \n| `bge-large-en-v1.5` | embeds the text once, compares it to cached candidate vectors | \n| `thenlper/gte-large` | the same, different encoder | \n\nA model that scores each candidate on its own cannot notice that two candidates are similar, and the order you list them in cannot reach it. A model that reads them as a set gets the first ability and the second problem together.\n\nTwo of the four candidate-at-a-time models never flip: `bge-large` and\n`deberta-large` are exactly 0.0000 across 1,600 decisions each, which is what\nscoring each option in isolation should give you. It doubles as a check that the\nharness isn't shuffling something it shouldn't.\n\nThe other two are not quite zero. `deberta-base` flips on 0.2% of decisions and\n`gte-large` on 2.0%, and both turned out to be exact scoring ties broken by\nposition rather than a real order effect: all 32 of gte's flipping items at 64\ncandidates have two options tied to within 1e-9. Structural invariance holds. Ties\nare the crack in it, and if you are picking a model because order cannot reach\nit, 2% is still 2%.\n\nAgainst that baseline, Jev's 14.6% at 64 candidates is a real cost. Roughly one decision in seven is settled by list position rather than by content. It is also about a third of Laya's rate, which changes its answer on half of its decisions at the same list length.\n\nThe flips are not spread evenly, and the spread matters more than the headline:\n\n| where | Jev flips, unrelated distractors | Jev flips, near-neighbour distractors | \n|---|---|---|\n| request routing (clinc) | 1.5% | 9.0% | \n| request routing (mtop) | 3.0% | 12.5% | \n| financial topics | 5.0% | 22.0% | \n| emotional tone | 30.5% | 33.0% | \n\nIntent routing with easy distractors is the best case and it is genuinely stable. Intent routing with plausible competing options, which is what a real router faces, runs four to six times worse. Anything subjective is worse again, and barely improves when the distractors get easy.\n\nTwo fixes are available to anyone deploying these today. Present the options in a fixed canonical order, which at least makes the instability deterministic. Or ask the same question under several orderings and average, which buys exact stability at the price of several calls per decision.\n\nOne more thing fell out of the logs after the fact. Confidence and correctness come apart badly as the list grows, and not for everyone:\n\n| model | calibration error at K=2 | at K=256 | \n|---|---|---|\n| deberta-large | 0.064 | 0.070 | \n| deberta-base | 0.066 | 0.075 | \n| bge-large | 0.099 | 0.197 | \n| gte-large | 0.048 | 0.299 | \n| gliclass | 0.150 | 0.327 | \n| Laya | 0.032 | **0.573** | \n\nLaya is the best-calibrated model in the set at two candidates and by far the worst at 256. If you are gating on a confidence threshold, that is the number that decides whether the gate works. Jev is missing from this table because its API rounds probabilities to two decimals, which is too coarse to measure calibration; 98% of its responses contain at least one option at exactly 0.00.\n\nJev's training data is not disclosed. The items come from public datasets, so \"trained better\" and \"has seen these before\" cannot be told apart here. The one hint available points both ways: Jev is steadiest on exactly the intent-style domains a decision product would plausibly be trained on, and shakiest on the domain furthest from that. Nothing here settles it, and no outside test can settle it while the training data stays private.\n\nThe Jev comparison was also not part of the original plan. I wrote the\npre-registration around the three open architecture families and listed Jev as a\nconditional extra if I got API access. I did, it went in, and it reversed the\nconclusion I had been heading toward, which was that reading candidates as a set\ncarries costs and buys nothing. Jev reads candidates as a set and carries much\nsmaller costs. So the honest version is that the costs belong to those two open\ncheckpoints rather than to the architecture. I've left the original reasoning in\n[EXPERIMENTS.md](https://github.com/gazelle93/decision-models-under-pressure/blob/main/EXPERIMENTS.md) rather than quietly rewriting it.\n\nThree smaller limits. The near-distractor comparison rests on two domains, both\nof which every model here has plausibly seen. Jev's API caps candidate lists at\n255 options, so the head-to-head stops at 128; the open models run to 256 and\nthose numbers are in `results/published/`. And n is 200 items per domain, with\nbootstrap intervals on everything in EXPERIMENTS.md, which is enough to separate\nthe large effects here and not enough for anything subtle.\n\n```\npython -m venv .venv\n.venv/bin/pip install torch transformers datasets sentence-transformers scikit-learn\n\n.venv/bin/python -m harness.run.local --rq rq3    # run one question\n.venv/bin/python -m harness.analyze.rq3\n```\n\nThe dataset ships complete, so there is no rebuild step. To rebuild it from the upstream sources anyway:\n\n```\n.venv/bin/python -m harness.build --n 200         # build and run the checks\n.venv/bin/python -m harness.build.export\n```\n\nThe Jev arm cost $1.04 for the full grid of 29,600 calls:\n\n```\nexport OPENROUTER_API_KEY=sk-or-...\n.venv/bin/python -m harness.run.jev --rq rq3 --cap 2.00\n```\n\n`dataset/v3/` holds 1,000 items across five domains, with the item text, the\ngold label, both distractor pools and every derived field. It is published under\n**CC BY-SA 4.0**, because MTOP is CC BY-SA 4.0 and ShareAlike carries over. If\nyou reuse it, your version inherits that too. The code is MIT.\n\n| domain | source | licence | \n|---|---|---|\n| clinc | [CLINC-150](https://huggingface.co/datasets/clinc/clinc_oos) | CC BY 3.0 | \n| goemotions | [GoEmotions](https://huggingface.co/datasets/google-research-datasets/go_emotions) | Apache 2.0 | \n| mtop | [MTOP](https://huggingface.co/datasets/WillHeld/mtop) | CC BY-SA 4.0 | \n| dbpedia | [DBpedia Classes](https://huggingface.co/datasets/DeveloperOats/DBPedia_Classes) | CC0 1.0 | \n| fintopic | [Twitter Financial News](https://huggingface.co/datasets/zeroshot/twitter-financial-news-topic) | MIT | \n\nFull attribution, the upstream papers, what I changed, and the GoEmotions\ncontent warning are in [`dataset/v3/README.md`](https://github.com/gazelle93/decision-models-under-pressure/blob/main/dataset/v3/README.md). Texts are\nverbatim; every item carries a SHA-256 of its text so you can check it against\nthe original source. If you use this data, cite the upstream papers, not this\nrepo.\n\n| path | contents | \n|---|---|\n| `dataset/v3/` | 1,000 items, the 902-option label universe, per-question splits, the checks, datacard | \n| `harness/build/` | how the dataset is made: label universe, items, distractor tiers, the eight checks | \n| `harness/models/` | model adapters, one per family, plus the hosted Jev client | \n| `harness/run/` | the two runners, local and hosted | \n| `harness/analyze/` | one analysis per question, plus the figures | \n| `results/published/` | the aggregates every table and figure is built from | \n| `PLAN.md` | what I decided to measure, and the pass/fail rules, before anything ran | \n| `EXPERIMENTS.md` | every run, including the ones that went against what I expected | \n\nAn earlier version of this produced confident results that did not survive review. Auditing it turned up three things that mattered: a classifier with no access to the item text could identify the right answer from label formatting alone, an option's position was a function of its label rather than the item, and one model's apparent context ceiling was an artifact of how my harness passed options to it.\n\nI rebuilt the dataset around those failures, and the checks that caught them now run as assertions before any model does. Eight of them: no option string may appear in the prompt's own wording, surface features must not separate near distractors from far ones, a text-blind classifier must stay under twice chance, gold position must depend on the item, the near tier must be measurably nearer, every item must be usable in both tiers at every list size, no adjudicated ambiguous pair may sit together, and each option set must be well formed.\n\nSeven of the eight pass. The one that does not is the text-blind classifier,\nwhich still beats chance on some cells: 0.260 on DBpedia against a chance rate of\n0.0625 and a 0.125 gate. That means some accuracy on those cells comes from label\nformatting rather than from reading the text, and it inflates the level of the\ncurves without changing their shape. Per-cell numbers are in\n`dataset/v3/gates.json` rather than described away. Two fixes were tried and\nrejected, with their numbers recorded: coarser matching reopens the formatting\ncheck, and dropping the affected items shrinks the sample while making the\nresidual worse.\n\nTwo rules I'd written in advance fired against me during the run. One voided a family-level comparison when the two models inside a family disagreed more than the families did. The other required the order-invariant models to score exactly zero, which two of four did, with the other two traced to scoring ties.\n\nThe dataset in `dataset/v3/` is CC BY-SA 4.0, inherited from MTOP. Per-source\nlicences, credits and what I changed are in\n[`dataset/v3/README.md`](https://github.com/gazelle93/decision-models-under-pressure/blob/main/dataset/v3/README.md).\n\nThe code carries no licence, which under default copyright means you may read it but not reuse it. That is deliberate rather than an oversight: the point of this repo is the method and the numbers, both of which you are welcome to take. If you want to reuse the code itself, ask and I'll add a licence.", "url": "https://wpnews.pro/news/benchmarking-jev-laya-and-five-open-models-under-three-stresses", "canonical_source": "https://github.com/gazelle93/decision-models-under-pressure", "published_at": "2026-09-25 13:49:30+00:00", "updated_at": "2026-09-25 14:02:35.375325+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "large-language-models"], "entities": ["TypeSafe", "Jev", "Laya", "gte-large", "bge-large", "gliclass", "deberta-large", "deberta-base"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/benchmarking-jev-laya-and-five-open-models-under-three-stresses", "markdown": "https://wpnews.pro/news/benchmarking-jev-laya-and-five-open-models-under-three-stresses.md", "text": "https://wpnews.pro/news/benchmarking-jev-laya-and-five-open-models-under-three-stresses.txt", "jsonld": "https://wpnews.pro/news/benchmarking-jev-laya-and-five-open-models-under-three-stresses.jsonld"}}