I Built 30 Classifiers That Can't Write a Sentence. They Judge Every Call My Agent Makes.

Typesafe.ai's Jev is a zero-shot structured-output classifier that runs a single parallel forward pass with no autoregressive decoder, returning one of three typed primitives — Noul (Bernoulli yes/no), Choice (a pick from a fixed category list), or Score (a float) — rather than generated text. Typesafe.ai charges $0.042 per million input tokens for Jev, which the author says is cheap enough to stop tracking per-call costs. The author reports running 34 Jev-powered Python scripts totaling 21,606 lines with 1,609 passing tests in the vexjoy-agent repository, with eight hooks firing Jev at five lifecycle events and a meta-compositor choosing among 30 registered tools.

A Noul comes back with a confidence of 0.12. That’s not noise. I read it as the whole point of the model. Typesafe.ai shipped something called Jev recently, and most of the discussion around it got stuck on whether it counts as a “frontier model.” I want to set that argument aside. Jev is a narrow, specific thing: a zero-shot, structured-output classifier with no text generation in it at all. The part that’s missing the-part-thats-missing Jev doesn’t have an autoregressive decoder. It can’t write a sentence, can’t generate code, can’t hold a conversation. That’s the whole design. I keep seeing the standard workaround for this kind of decision: ask a big language model to generate a string, then write code to parse that string for a keyword. Standard LLMs predict one token, feed it back in, predict the next one, and repeat until they hit a stop condition. That loop costs time. Jev skips the loop entirely. It runs a single parallel forward pass over the input, plain text or structured JSON, and computes a decision boundary directly. I don’t have to wait for tokens I’m going to throw away anyway. No loop also means no per-token cost on the output side. Typesafe.ai charges $0.042 per million input tokens — I checked twice, because that’s cheap enough to stop tracking per-call. There’s nothing left to decode one piece at a time, so the response just lands. I’ve stopped budgeting for it. Three shapes, not one string three-shapes-not-one-string When I ask a normal LLM to classify something, I get a string back, then I write code to match that string against expected values and handle whatever shows up that I didn’t expect. Production bugs live in that parsing step. Jev skips the string. It returns one of three typed primitives: - Noul, a Bernoulli yes/no - Choice, a pick from a fixed category list - Score, a float on a sliding scale Each one maps directly onto a decision a caller already has to make. I don’t want a router that gets back a string that might say “yes” or “Yes.” or “yes, definitely.” I want a boolean. Forced answers versus calibrated ones forced-answers-versus-calibrated-ones Here’s the distinction. OpenAI’s structured outputs, and most constrained-decoding setups like it, work by masking invalid tokens. If the model wants to output something outside the schema, the system zeroes out that probability and forces the next best valid token instead. The model is still confused underneath. It just isn’t allowed to show it. Jev is trained differently. The training objective targets accurate probability directly, not just schema validity, the way constrained decoding does. So when Jev returns that Noul at 0.12, I take the number at face value. A confidence that low isn’t noise to filter out. A caller can act on it directly: fail over to a human, fall back to a default route, whatever the pipeline needs. That’s actionable. A forced answer and a calibrated admission of uncertainty can look identical on the wire. Same JSON blob, same boolean field. I only notice the difference when I’m deciding whether to trust the number. What I built around it what-i-built-around-it That was the observation. What follows is what happened when I stopped treating Jev as a single-call classifier and started treating it as a judgment layer that could run at every decision point in an agent toolkit. The toolkit I run, documented across the vexjoy-agent repository, now has 34 Jev-powered Python scripts totaling 21,606 lines, backed by 1,609 passing tests. Eight hooks fire Jev at five different lifecycle events. A meta-compositor decides which of 30 registered tools to run for each request. The whole thing dogfoods itself — Jev reviews the code that Jev tools produce. Source: scripts/jev- .py 34 files , hooks/jev- .py 8 files , scripts/tests/test jev .py test suite run: 1,609 passed . The /d router: four phases, one Jev call each the-d-router-four-phases-one-jev-call-each The /d command is the entry point. A user types /d review this PR for security issues and the system has to decide which agent, which skill, and which pipeline to invoke. Before Jev, this required loading a full routing manifest into the LLM’s context and asking it to pick. The /d router replaces that with a four-phase pipeline defined in skills/meta/d/SKILL.md . Phase 1 CLASSIFY runs jev-route.py as a single subprocess call. Jev gets the request text, a list of all available agents and skills from the manifest, and returns a structured result: agent, skill, pipeline, complexity, confidence, and reasoning. One HTTP round trip to Typesafe, no autoregressive generation. Phase 2 DECIDE reads that result and applies it. The agent, skill, and pipeline names are already validated against the live manifest inside the script — Jev picked from a constrained set, and the script verified the pick is real. Phase 3 ENHANCE maps boolean signals from the classification into stack entries. If tests requested came back true a Noul above 0.6 , the router adds test-driven-development and verification-before-completion to the skill stack. If research needed fired, it fans out a research coordinator agent in parallel. Phase 3.5 COMPOSE is where it gets dense. This phase calls jev-compose.py , a meta-compositor that uses one Jev call to decide which of 27 registered tools should run for this specific request. Source: skills/meta/d/SKILL.md , scripts/jev-route.py , scripts/jev-compose.py . The compose layer: 27 tools, one Jev call the-compose-layer-27-tools-one-jev-call jev-compose.py holds an AVAILABLE TOOLS registry of 27 Jev tools, organized into six categories: review 6 tools , prose 2 , classification 4 , security and risk 4 , request analysis 4 , execution planning 3 , and post-agent validation 4 . Three more scripts — jev-compose.py itself, jev-route.py , and jev-eval.py — sit outside the compose registry, bringing the total to 30. The compositor’s architecture is one Jev call with parallel questions: one Noul per candidate tool “should this tool run for this request?” plus one Choice for pipeline shape pre-filter, post-check, continuous, standalone, or none . That’s 28 parallel questions in a single HTTP request. Jev evaluates all of them in one forward pass and returns scores for each. Tools above the 0.5 threshold get selected. The compositor sorts them into deterministic phase ordering — pre-filter tools first, then standalone, then post-check — and returns an ordered pipeline spec. The whole decision costs one Jev call at approximately the same latency as a single Noul. When a user types /d review this PR , the compositor might select jev-diff-triage order 1 , jev-cascade-review order 4 , jev-pr-slop-scan order 5 , jev-comment-quality order 6 , and jev-review-pipeline order 8 — the full review chain. For /d write a blog post , it might select only jev-anti-slop and jev-voice-check as post-check tools. For /d TypeError: unhashable type list , it selects jev-error-classify as a standalone tool that handles the request without dispatching an LLM agent at all. Source: scripts/jev-compose.py AVAILABLE TOOLS list, lines 51-601; build payload , lines 686-737 . Four hooks across the agent lifecycle four-hooks-across-the-agent-lifecycle The four hooks fire Jev at every stage of an agent turn. Each hook is a Python script registered to a Claude Code lifecycle event. UserPromptSubmit jev-route-injector-userprompt.py . Detects a /d invocation via regex before the model generates its first token. Runs jev-route.py , then chains to jev-compose.py , then executes the composed tools — all three steps complete before the LLM sees the prompt. The results are injected as additionalContext so the model reads them instead of re-running the scripts. This exists because the model previously skipped the route script on a meta-question and reasoned about routing itself; a hook that runs before generation starts removes that failure mode. PreToolUse jev-pre-edit-guard-pretooluse.py . Fires before every Write, Edit, or Bash call. For Write and Edit, it runs jev-secret-scan.py on the content being written. A high-severity real secret severity 4 or above blocks the tool call entirely via permissionDecision:deny . Lower-severity findings inject a warning. For Bash, it runs jev-rollback-risk.py on the command and injects a warning for irreversible operations but never blocks. SubagentStop jev-post-agent-validator-subagentstop.py . Fires after every subagent completes. Runs three Jev tools in parallel via ThreadPoolExecutor : jev-completion-validator did the output address the request? , jev-agent-confidence does output quality match routing confidence? , and jev-scope-creep did the agent touch files outside the requested scope? . Results are written to a per-session state file for downstream hooks. Stop jev-turn-quality-gate-stop.py . Fires after the model finishes responding. Computes a working-tree diff, and if the turn produced code changes, runs jev-commit-readiness.py on the diff. When the verdict is not “ready,” it surfaces an advisory rewake. This hook is advisory only — it never blocks. All four hooks fail open. Network timeout, malformed response, missing script — every error path exits 0 and lets the turn proceed. The one exception is the PreToolUse secret scan at severity 4+, which blocks because a leaked secret costs more than a false block. Source: hooks/jev-route-injector-userprompt.py , hooks/jev-pre-edit-guard-pretooluse.py , hooks/jev-post-agent-validator-subagentstop.py , hooks/jev-turn-quality-gate-stop.py . The structured interview pattern the-structured-interview-pattern Jev’s accuracy depends on question quality. The project’s design philosophy docs/PHILOSOPHY.md , “Write Jev questions that cut” documents a pattern shift from v1 to v2 questions. v1 questions were broad. A single Noul asking “did the agent expand scope?” A vague question returns a vague probability. v2 questions are structured interviews: 10-15 Nouls per tool, each isolating one dimension. For scope creep, that means separate Nouls for “touched files outside the import chain,” “reformatted code it only read,” “added features not requested,” “changed error handling in unrelated paths,” and so on. Each question has a clear true/false boundary. A single Jev call carries up to 30 parallel Nouls at negligible marginal cost. The criteria blocks — what , examples , not for per option — encode the hard distinctions. The jev-compose.py source shows this: each tool in AVAILABLE TOOLS carries when to run , examples , when not to run , and negative examples . These criteria exist to resolve the ambiguous cases. “Agent added a test file for code it edited” is expected practice, not scope expansion. “Agent rewrote an unrelated README” is obvious scope creep and doesn’t need criteria. The criteria matter for the gray zone between those two. Source: docs/PHILOSOPHY.md “Ask many specific questions, not few broad ones”; “Criteria encode the hard distinctions” , scripts/jev-compose.py AVAILABLE TOOLS per-tool criteria . Dogfooding: Jev reviews Jev dogfooding-jev-reviews-jev The toolkit uses Jev to review its own output. When I ask the /d router to review a PR that includes changes to Jev scripts, the compose layer selects the review chain jev-diff-triage , jev-cascade-review , jev-pr-slop-scan , jev-comment-quality , and those tools evaluate the diff — including diffs to themselves. The jev-route-injector hook runs the full route-compose-execute pipeline before the model generates a single token. The model receives Jev’s findings as prior results . The SKILL.md makes the separation explicit: “Jev handles every judgment call, the LLM agent handles only generative/creative work.” On a review request, the agent explains, prioritizes, and suggests fixes. It does not re-judge what Jev already decided. This creates a concrete feedback loop. When a Jev tool produces a false positive, the fix goes into the tool’s criteria block, and the next invocation evaluates whether the fix worked — using Jev. Source: skills/meta/d/SKILL.md Phase 4, “When JEV TOOL RESULTS is non-empty” section . What’s known and what isn’t whats-known-and-what-isnt Three limitations are concrete enough to name. Question quality is still the bottleneck. The cost per Jev call is near-zero. The constraint is thinking clearly about what to ask. The v2 structured-interview pattern produces sharper results than v1’s broad questions, but writing good criteria for each of 27 tools is slow, manual work. jev-test-strategy.py was recently rewritten from v1 to v2 question format; the remaining scripts need the same treatment. Calibration outside known domains is untested. I haven’t run a calibration stress test on adversarial or out-of-distribution input. Jev’s RLCD training targets accurate probabilities, and the scores I see on code review, secret scanning, and scope checking are consistent with what I’d expect. But I’m operating inside the domains I built questions for. I don’t know how calibration holds on content types or code patterns my criteria don’t cover. Context window costs remain. Each Jev call is cheap, but the state field — the evidence Jev evaluates — comes from somewhere. On a large diff, the pre-edit guard truncates to 4,000 characters before passing to Jev. The compose layer truncates similarly. That truncation loses information. KV cache optimization for Jev’s input side, and smarter truncation strategies, are on the list but not built. The question I started with was whether a zero-shot classifier with calibrated probabilities could serve as the judgment layer for an entire agent toolkit. Thirty scripts, four lifecycle hooks, and 17,183 lines of Python later, the answer is qualified: it works, when the questions are sharp and the criteria encode the actual hard cases. The model that can’t write a sentence turns out to be good at deciding things — as long as I’m specific about what I’m asking it to decide.