{"slug": "a-local-jev-backed-by-diffusiongemma", "title": "A local Jev backed by DiffusionGemma", "summary": "LocalJev, a TypeScript server for Bun 1.2+, implements a Jev-compatible POST /v1/systemone API backed by the DiffusionGemma model diffusiongemma-26B-A4B-it-4bit through an OpenAI-compatible Chat Completions endpoint. Because the normal oMLX API does not expose the vLLM request extensions OpenJev relies on (diffusion_seed_canvas, diffusion_read_only, and requested token logprobs), LocalJev translates Jev state and typed questions into classification prompts, asks DiffusionGemma for a JSON probability scalar or vector, validates and retries malformed output, then normalizes vectors to return Jev-compatible choices, expected scores, and entropy-based confidence. The project states the result is wire-compatible but not mathematically equivalent to OpenJev's logit read, since probabilities are self-reported by the model rather than read from its logits, and advises evaluating calibration before consequential decisions.", "body_md": "A local, Jev-compatible `POST /v1/systemone` API written in TypeScript for\n[Bun](https://bun.sh/), backed by DiffusionGemma through an OpenAI-compatible\nChat Completions endpoint.\n\nThe defaults target:\n\n- inference server: `http://127.0.0.1:8000`\n- model: `diffusiongemma-26B-A4B-it-4bit`\n- LocalJev API: `http://127.0.0.1:8080`\n\n[Jev](https://typesafe.ai/) uses a typed decision API rather than an OpenAI chat API.\n[OpenJev](https://github.com/razorback16/openjev) implements the Jev wire protocol and\nobtains probabilities with a special one-step DiffusionGemma **structured read**. Its\nbackend depends on unmerged vLLM request extensions such as\n`diffusion_seed_canvas`, `diffusion_read_only`, and requested token logprobs.\n\nThe normal oMLX API does not expose those primitives. LocalJev therefore takes the portable approach:\n\n1. translate `state` and typed Jev questions into a classification prompt;\n2. ask DiffusionGemma for a JSON probability scalar/vector;\n3. validate the complete result and retry malformed output;\n4. normalize vectors and calculate Jev-compatible choices, expected scores, and entropy-based confidence;\n5. return the normal Jev response shape.\n\nThis is wire-compatible, but not mathematically equivalent to OpenJev's logit read. The probabilities are generated/self-reported by the model rather than read directly from its logits. Evaluate their calibration on your own workload before relying on them for consequential decisions.\n\nRequires Bun 1.2+ and a running oMLX server.\n\n```\nbun install\ncp .env.example .env\n$EDITOR .env # replace the upstream API-key placeholder\nbun run start\n```\n\nBun loads `.env` automatically. Alternatively, set the key in your shell before\nstarting the server:\n\n```\n# fish\nset -gx LOCALJEV_UPSTREAM_API_KEY 'your-local-omlx-key'\n# bash/zsh\nexport LOCALJEV_UPSTREAM_API_KEY='your-local-omlx-key'\n```\n\nLocalJev listens on `http://127.0.0.1:8080`. Check that the configured model is\navailable:\n\n```\ncurl http://127.0.0.1:8080/ready\n```\n\nMake a decision:\n\n```\ncurl http://127.0.0.1:8080/v1/systemone \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"jev-latest\",\n    \"state\": \"Hi, I have been trying to connect Stripe but keep getting a 403 error.\",\n    \"questions\": {\n      \"department\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which team should handle this?\",\n        \"criteria\": {\n          \"billing\": \"Payment or subscription issues\",\n          \"technical\": \"Bugs or integration problems\",\n          \"sales\": \"Pricing or account questions\"\n        }\n      },\n      \"frustration\": {\n        \"type\": \"score\",\n        \"instructions\": \"How frustrated does the customer appear?\",\n        \"criteria\": [\"Calm\", \"Frustrated but civil\", \"Very angry\"]\n      },\n      \"urgent\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does this require an immediate response?\"\n      }\n    }\n  }'\n```\n\nThe SDK requires an API-key value. LocalJev accepts any value unless\n`LOCALJEV_API_KEY` is configured. Set the SDK environment for your shell:\n\n```\n# fish\nset -gx TYPESAFE_BASE_URL http://127.0.0.1:8080\nset -gx TYPESAFE_API_KEY local\n# bash/zsh\nexport TYPESAFE_BASE_URL=http://127.0.0.1:8080\nexport TYPESAFE_API_KEY=local\npython\nfrom typesafe_sdk import TypeSafeClient\n\nclient = TypeSafeClient()\nresponse = client.system_one(\n    \"I was charged twice this month.\",\n    {\n        \"billing\": {\n            \"type\": \"noul\",\n            \"instructions\": \"Is this a billing issue?\",\n        }\n    },\n)\nprint(response.nouls[\"billing\"].noul)\n```\n\n`jev-latest` and `jev-preview` are accepted aliases so SDK defaults work unchanged.\n\n| Variable | Default | Purpose | \n|---|---|---|\n| `LOCALJEV_UPSTREAM` | `http://127.0.0.1:8000` | OpenAI-compatible base URL, with or without `/v1` | \n| `LOCALJEV_UPSTREAM_API_KEY` | empty | Bearer key sent to the inference server | \n| `LOCALJEV_UPSTREAM_MODEL` | `diffusiongemma-26B-A4B-it-4bit` | Upstream model identifier | \n| `LOCALJEV_API_KEY` | empty | Optional Bearer key required from LocalJev clients | \n| `LOCALJEV_HOST` | `127.0.0.1` | Listen address | \n| `LOCALJEV_PORT` | `8080` | Listen port | \n| `LOCALJEV_TIMEOUT` | `180` | Upstream timeout in seconds | \n| `LOCALJEV_MAX_INFLIGHT` | `2` | Concurrent calls admitted upstream | \n| `LOCALJEV_MAX_QUEUE` | `64` | Waiting decisions before HTTP 529 | \n| `LOCALJEV_MALFORMED_RETRIES` | `2` | Corrective retries for invalid model JSON | \n| `LOCALJEV_MAX_OUTPUT_TOKENS` | `2048` | Per-completion output ceiling | \n| `LOCALJEV_QUESTIONS_PER_CALL` | `16` | Chunking limit per model call | \n| `LOCALJEV_OUTCOMES_PER_CALL` | `128` | Choice/score outcomes per model call | \n\nBun automatically loads `.env`, so you can also copy `.env.example`, replace its\nplaceholder, and run the server.\n\n```\nbun install\nbun test\nbun run typecheck\nbun run smoke       # live call to the configured inference server\n```\n\nThe repeatable bake-off uses public gold labels for news categorization (AG News), yes/no reading comprehension (BoolQ), and five-level sentiment (SST-5). It runs the same LocalJev engine against five installed models, comparing quality, calibration, retries, and full-decision latency at two actual input lengths.\n\n```\n# Quick integration check (30 requests, not a meaningful quality sample)\nbun run eval --out eval/runs/pilot --limit 3\n\n# 5 models × 120 labeled examples × 2 input lengths = 1,200 requests\nbun run eval --out eval/runs/my-bakeoff\n\n# Regenerate a completed or partial report without running inference\nbun run eval:report eval/runs/my-bakeoff\n```\n\nRequires oMLX and the upstream key in `.env`; no running LocalJev HTTP server or\nPython is needed. See [the evaluation guide](https://github.com/githubnext/localjev/blob/main/docs/evaluation.md) for pinned data\nsources, methodology, configuration, resuming runs, and limitations.\n\nThe [first completed bake-off](https://github.com/githubnext/localjev/blob/main/docs/evaluation-results-2026-09-18.md) includes\n1,200 requests on an M5 Max. Gemma 4 26B-A4B and Qwen3.6 were the strongest overall\ncandidates in this small screening sample; the report includes per-task results,\nlatency, context effects, and caveats rather than claiming a definitive winner.\n\nNot currently for this model. As of September 18, 2026, DiffusionGemma support is\nstill tracked as open in both\n[`lmstudio-ai/mlx-engine#336`](https://github.com/lmstudio-ai/mlx-engine/issues/336)\nand\n[`lmstudio-ai/lmstudio-bug-tracker#2037`](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2037).\nThe reported MLX backend fails to load `diffusion_gemma`, while the normal llama.cpp\nbackend reports an unknown architecture. oMLX already loads and serves your exact\ncheckpoint successfully, so it is the better runner for this Mac today.\n\nEven after LM Studio adds ordinary generation support, changing runners alone will\nnot make the result OpenJev-equivalent. The runner must expose seeded diffusion\ncanvases, read-only denoising, and selected-token logits/logprobs. If LM Studio only\nprovides standard Chat Completions, LocalJev can use it by changing\n`LOCALJEV_UPSTREAM`, but the probability path remains prompted/self-reported.\n\nFor direct model probabilities, the best paths are:\n\n1. add the structured-read primitives to oMLX's DiffusionGemma lane and consume them here; or\n2. run OpenJev's patched vLLM backend on a supported NVIDIA machine.", "url": "https://wpnews.pro/news/a-local-jev-backed-by-diffusiongemma", "canonical_source": "https://github.com/githubnext/localjev", "published_at": "2026-09-19 13:02:40+00:00", "updated_at": "2026-09-19 13:25:04.182905+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-products", "large-language-models"], "entities": ["LocalJev", "DiffusionGemma", "Jev", "OpenJev", "Bun", "oMLX", "vLLM", "TypeSafeClient"], "alternates": {"html": "https://wpnews.pro/news/a-local-jev-backed-by-diffusiongemma", "markdown": "https://wpnews.pro/news/a-local-jev-backed-by-diffusiongemma.md", "text": "https://wpnews.pro/news/a-local-jev-backed-by-diffusiongemma.txt", "jsonld": "https://wpnews.pro/news/a-local-jev-backed-by-diffusiongemma.jsonld"}}