{"slug": "our-own-jev-typed-decisions-in-one-forward-pass-from-an-in-house-judge-to-open", "title": "Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models", "summary": "An arXiv paper (2610.29052v1) reports that an in-house \"System 1\" judge built on a frozen backbone with a rank-16 LoRA adapter and a pointer head triaged about 148K agentic coding session records in one forward pass, judging 1,000 sessions in roughly three minutes on one GPU at about $0.06 versus roughly $1,000 per pass for the frontier LLM judge it replaced — about 1% of that cost. The authors report the frontier judge's stored labels matched a fresh two-judge consensus on only 26.9% of sessions where the two judges agreed. Applying the public Eikos recipe and a corpus of about 25K decisions to 1B, 2B and 35B backbones, the team's single-seed 35B model scored 79.3% on the 111 public hard items of JevBench, not significantly different from the commercial reference's 73.0% (unadjusted p=0.12), while a retrain on a later corpus release moved the 2B's hard-tier accuracy by 11.7 points.", "body_md": "arXiv:2610.29052v1 Announce Type: new \nAbstract: A typed decision maps evidence, a question and a fixed set of options to a probability for every option, computed in one forward pass, by reading a language model's logits or a small head over its hidden states, instead of generating and parsing text. The commercial system Jev popularised this System~1 interface, and open reproductions have followed. We report two uses of the pattern. \\emph{Part I} describes an in-house System~1 judge that triages about 148K agentic coding session records for a training corpus. It replaces a frontier LLM judge that cost on the order of \\$1,000 per pass and whose stored labels matched a fresh two-judge consensus on only 26.9\\% of the sessions where the two judges agreed; a self-hosted frozen backbone with a rank-16 LoRA adapter and a pointer head answers three typed questions in one forward pass, judges 1,000 sessions in about three minutes of one GPU, about \\$0.06 at a reference rate and roughly 1\\% of the frontier judge's cost, and keeps every session inside the organisation. \\emph{Part II} asks whether a readout that adds no parameters can be competitive and whether training one helps. Applying the public Eikos recipe and corpus of about 25K decisions to three further backbones (1B, 2B and 35B), we read the frozen output layer of a LoRA-adapted decoder. On the 111 public hard items of JevBench our single-seed 35B, developed with those items in view, scores 79.3\\%, not significantly different from the commercial reference (73.0\\%, unadjusted $p=0.12$) and ahead of five open systems after Holm--Bonferroni correction; our 2B is ahead of all four kev checkpoints on the leaderboard. A single-seed ablation finds no detectable benefit from training the 26 letter rows of the output layer, and a retrain on a later corpus release moves the 2B's hard-tier accuracy by 11.7 points, which shows how little small evaluations resolve.", "url": "https://wpnews.pro/news/our-own-jev-typed-decisions-in-one-forward-pass-from-an-in-house-judge-to-open", "canonical_source": "https://arxiv.org/abs/2610.29052", "published_at": "2026-10-22 04:00:00+00:00", "updated_at": "2026-10-22 04:12:25.203020+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents", "mlops"], "entities": ["Jev", "JevBench", "Eikos", "LoRA", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/our-own-jev-typed-decisions-in-one-forward-pass-from-an-in-house-judge-to-open", "markdown": "https://wpnews.pro/news/our-own-jev-typed-decisions-in-one-forward-pass-from-an-in-house-judge-to-open.md", "text": "https://wpnews.pro/news/our-own-jev-typed-decisions-in-one-forward-pass-from-an-in-house-judge-to-open.txt", "jsonld": "https://wpnews.pro/news/our-own-jev-typed-decisions-in-one-forward-pass-from-an-in-house-judge-to-open.jsonld"}}