{"slug": "jev-rs-a-rust-crate-to-turn-a-llm-to-serve-jev", "title": "Jev-rs: a Rust crate to turn a LLM to serve jev", "summary": "Developer Yijun Yu released jev-rs, a Rust crate that turns any GGUF-served LLM into a probability-scoring engine for typed questions — yes/no (noul), choice, and score — returning probabilities instead of generated text. The tool is wire-compatible with TypeSafe's Jev POST /v1/systemone API and exposes a single MCP tool named judge over stdio for coding agents including Claude Code, Codex, Grok Build, and OpenCode. It installs a prebuilt jev binary into ~/.local/bin for macOS arm64/x86_64 and Linux x86_64/arm64, or builds from source via cargo install jev-rs, and supports OpenAI-compatible backends such as the DeepSeek API, vLLM, and SGLang through chat/completions with logprobs (max_tokens: 1, top_logprobs: 20).", "body_md": "**System One judgments from any LLM, in one prefill.** A Rust engine that\nanswers typed questions about a piece of state — `noul` (yes/no), `choice`\n(one of N), `score` (ordered scale) — with probabilities, never generated\ntext. Wire-compatible with TypeSafe's Jev\n[`POST /v1/systemone`](https://docs.typesafe.ai/api), and exposed to coding\nagents as an MCP tool.\n\n```\nClaude Code / Codex / Grok Build / OpenCode ──MCP stdio──▶ jev ──▶ llama-server (any GGUF)\nTypeSafe SDKs (TYPESAFE_BASE_URL) ──────────POST /v1/systemone──▶ jev serve ──▶ llama-server\ncurl -fsSL https://raw.githubusercontent.com/yijunyu/jev-rs/main/install.sh | sh\n```\n\nInstalls a prebuilt `jev` into `~/.local/bin` (macOS arm64/x86_64, Linux\nx86_64/arm64) or builds from source with cargo if no binary matches.\nWith a Rust toolchain, `cargo install jev-rs` works too. Then\nstart any GGUF model behind `llama-server` (macOS: `brew install llama.cpp`):\n\n```\nllama-server -hf Qwen/Qwen3-4B-GGUF --port 8089 -np 2 -c 8192\n```\n\nTry it:\n\n```\njev --backend http://127.0.0.1:8089 ask \\\n  --state \"We were billed twice for March. Refund the duplicate today or we cancel.\" \\\n  --choice \"dept=Which team handles this?|billing:refunds and invoices,technical:bugs,sales:pricing\" \\\n  --score  \"urgency=How urgent?|not urgent,soon,blocking\" \\\n  --noul   \"churn=Does the customer threaten to leave?\"\n{\n  \"dept\":    {\"type\":\"choice\",\"choice\":\"billing\",\"probabilities\":{\"billing\":1.0,\"technical\":0.0,\"sales\":0.0},\"confidence\":1.0},\n  \"urgency\": {\"type\":\"score\",\"score\":1.98,\"legend\":{\"0\":\"not urgent\",\"1\":\"soon\",\"2\":\"blocking\"},\"probabilities\":{\"0\":0.0,\"1\":0.02,\"2\":0.98},\"confidence\":0.98},\n  \"churn\":   {\"type\":\"noul\",\"noul\":1.0}\n}\n```\n\nSet `JEV_BACKEND_URL` once to drop the `--backend` flag. Use\n`--template gemma|llama3|raw` for non-ChatML models.\n\n**Hosted or OpenAI-compatible backends.** `--backend-kind openai` scores\nthrough `chat/completions` with `logprobs` (`max_tokens: 1`,\n`top_logprobs: 20`), so the DeepSeek API, vLLM, SGLang or llama-server's\nown `/v1` endpoint work without raw prompt access:\n\n```\nexport JEV_API_KEY=$DEEPSEEK_API_KEY\njev --backend https://api.deepseek.com/v1 --backend-kind openai --model deepseek-flash \\\n    eval examples/dev_tasks.jsonl\n```\n\nThe server applies its own chat template, so answers depend on the model\nemitting the option letter as its first token; the raw `llamacpp` path is\nexact and preferred when you control the server.\n\n`jev mcp` is an MCP server over stdio with one tool, `judge`. The agent\nsends a state and typed questions and gets probabilities back instead of\nprompting a big model to classify and parsing its prose. Typical uses inside\nan agent session: routing a task to a skill, triaging tool output, gating a\nrisky command, ranking candidates, yes/no checks on a diff.\n\n**Claude Code**\n\n```\nclaude mcp add jev -e JEV_BACKEND_URL=http://127.0.0.1:8089 -- jev mcp\n```\n\nor in `.mcp.json` at the project root:\n\n```\n{\"mcpServers\": {\"jev\": {\"command\": \"jev\", \"args\": [\"mcp\"], \"env\": {\"JEV_BACKEND_URL\": \"http://127.0.0.1:8089\"}}}}\n```\n\n**Codex** (`~/.codex/config.toml`)\n\n```\n[mcp_servers.jev]\ncommand = \"jev\"\nargs = [\"mcp\"]\nenv = { JEV_BACKEND_URL = \"http://127.0.0.1:8089\" }\n```\n\n**Grok Build** (`.grok/mcp.json` in the project, or `~/.grok/mcp.json`)\n\n```\n{\"servers\": {\"jev\": {\"command\": \"jev mcp\", \"env\": {\"JEV_BACKEND_URL\": \"http://127.0.0.1:8089\"}}}}\n```\n\nor `grok mcp add jev --command \"jev mcp\" --env JEV_BACKEND_URL=http://127.0.0.1:8089`.\n\n**OpenCode** (`opencode.json`)\n\n```\n{\"mcp\": {\"jev\": {\"type\": \"local\", \"command\": [\"jev\", \"mcp\"], \"environment\": {\"JEV_BACKEND_URL\": \"http://127.0.0.1:8089\"}}}}\n```\n\nThen tell the agent, in its instructions file, when to reach for it:\n\nUse the `judge` tool for any classification, routing, triage or yes/no\ndecision about text or tool output. Put the facts in `state` and ask one\njudgment per question. Trust answers with confidence ≥ 0.8; otherwise\ndecide yourself.\n\n**TypeSafe SDK users.** `jev serve` speaks the same wire format as\n`api.typesafe.ai`, so the official Python/JS SDKs, the\n[`jev`](https://crates.io/crates/jev) crate and any Jev integration run\nagainst a local model unchanged:\n\n```\njev --backend http://127.0.0.1:8089 serve --bind 127.0.0.1:8090\nexport TYPESAFE_BASE_URL=http://127.0.0.1:8090 TYPESAFE_API_KEY=local\n```\n\n| command | does | \n|---|---|\n| `jev mcp` | MCP server over stdio, tool `judge` | \n| `jev serve` | HTTP server: `/v1/systemone` ,`/v1/models` ,`/health` ;`--api-keys` for bearer auth | \n| `jev ask` | one request from flags, `--file` , or stdin;`--compare` also hits the hosted API | \n| `jev eval cases.jsonl` | accuracy, Brier, top-label ECE, coverage at ≤5 % error, latency, token cost | \n| `jev calibrate cases.jsonl` | fit per-bucket temperatures, write `calibration.json` (load with`--calibration` ) | \n\nCase-file line: `{\"state\": ..., \"questions\": {...}, \"gold\": {\"id\": \"key-or-level\"}}`.\n\n1. The state is rendered once into a shared prefix; each question is a\nsuffix ending in `Answer:` with options labelled`A` ,`B` ,`C` … so the\nbackend's prompt cache prefills the state once per request.\n2. The backend returns raw next-token log-probabilities for the labels.\nNothing is decoded; `usage.output_tokens` is always 0.\n3. Restricted softmax over the labels, divided by a fitted temperature per\n`(question type, option count)` bucket.`confidence` is TypeSafe's\ndocumented formula;`score` is the probability-weighted level.\n4. `--permutations k` averages`k` rotations of the option order to control\nposition bias.\n\nApple M1 Ultra 128 GB, `llama-server` via Homebrew, zero-shot, same\nprompts. `examples/dev_tasks.jsonl`: 25 shell commands × 3 questions\n(command class 8-way, safe to re-run, output volume 3-level), hand-labelled,\n75 decisions, 0 failed requests for either model.\n\n| metric | Qwen3-4B Q4_K_M | Qwen3.8-Flash-Next Q2_K_XL (73 GB) | DeepSeek-V4-Flash MXFP4 (156 GB, SSD-streamed) | \n|---|---|---|---|\n| backend | llama-server, raw logprobs | llama-server, raw logprobs | ds4-rs-metal, chat logprobs, thinking off | \n| accuracy overall | 0.71 | **0.84** | 0.76 | \n| accuracy: choice / noul / score | 0.92 / 0.64 / 0.56 | 0.92 / 0.76 / **0.84** | 0.88 / **0.92** / 0.48 | \n| ECE raw | 0.244 | 0.108 | 0.112 | \n| Brier raw | 0.512 | 0.276 | 0.371 | \n| coverage at ≤5 % error | 0.31 | **0.59** | 0.56 | \n| latency p50 per question (warm) | **75 ms** | 700 ms | 41 s | \n\nWhat changed with the larger models: the 8-way classification was already\nsaturated at 4B; the gains are on the yes/no and ordinal questions and on\nprobability quality. Both large models are close to calibrated out of the\nbox (fitted temperatures near 1), so their confidence can gate about twice\nas many decisions at a 5 % error budget. The 4B model is far faster and\nits fitted temperatures of 3–6 say its raw probabilities should not be\ntrusted without `jev calibrate`.\n\nDeepSeek V4 Flash gives the best yes/no judgment of the three (0.92 on\n\"safe to re-run\") and the worst ordinal one: of its 13 wrong `score`\nanswers, 12 guessed low and 11 of those by exactly one level, a systematic\nbias a per-model calibration or a coarser scale would absorb. Its latency is an artefact of the run,\nnot the model: 156 GB of tensors on a 128 GB machine means experts stream\nfrom SSD on every request. With the model resident (a 192 GB or 256 GB\nMac) the same engine prefills at hundreds of tokens per second.\n\nPrompt caching depends on the architecture: Qwen3-4B re-evaluates only the\nquestion suffix after the first question (33–62 tokens), while the hybrid\nQwen3.8-Next re-evaluates the full prompt for every question in\n`llama-server`, so its four-question example costs 3.0 s against 0.4 s.\n\nTwo further experiments against real Claude Code session logs, predicting\noutput floods before a command runs and triaging tool output after it, are\nin [`docs/EXPERIMENTS.md`](https://github.com/yijunyu/jev-rs/blob/main/docs/EXPERIMENTS.md); one is a clear win for the\njudge, the other a clear loss to a blind rule.\n\nHosted Jev has been independently measured at 236–276 ms p50 per request\n([jev-benchmarks](https://github.com/AbdelStark/jev-benchmarks),\n[decision-model-benchmark](https://github.com/nibzard/decision-model-benchmark)).\n\nOpen Jev replacements appeared within a week of the launch\n([Laya](https://huggingface.co/convaiinnovations/laya),\n[jeff](https://github.com/lodos3/jeff),\n[jev-bridge](https://github.com/TOSUKUi/jev-bridge)). jev-rs is built to\nbe the judgment engine inside two Rust systems —\n[PRECC](https://github.com/peri-a-i/precc-cc), a Claude Code hook that\nsaves tokens, and [ds4-rs-metal](https://github.com/yijunyu/ds4-rs-metal) /\n[Local Mind](https://yijunyu.github.io/local-mind/), an on-device\nDeepSeek-V4 engine — where an in-process,\nKV-forking scorer is the point.\n\n- At most 26 options per question (single-letter labels); Jev accepts 255.\n- One backend, `llama-server` . In-process llama.cpp and ds4-rs backends are next.\n- Zero-shot only; no RLCD-style training.\n\nMIT or Apache-2.0, at your option.", "url": "https://wpnews.pro/news/jev-rs-a-rust-crate-to-turn-a-llm-to-serve-jev", "canonical_source": "https://github.com/yijunyu/jev-rs", "published_at": "2026-09-23 12:02:22+00:00", "updated_at": "2026-09-23 12:30:34.203604+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "agent-protocols", "developer-tools"], "entities": ["jev-rs", "Yijun Yu", "TypeSafe", "Claude Code", "Codex", "Grok Build", "OpenCode", "DeepSeek"], "alternates": {"html": "https://wpnews.pro/news/jev-rs-a-rust-crate-to-turn-a-llm-to-serve-jev", "markdown": "https://wpnews.pro/news/jev-rs-a-rust-crate-to-turn-a-llm-to-serve-jev.md", "text": "https://wpnews.pro/news/jev-rs-a-rust-crate-to-turn-a-llm-to-serve-jev.txt", "jsonld": "https://wpnews.pro/news/jev-rs-a-rust-crate-to-turn-a-llm-to-serve-jev.jsonld"}}