Typed-lm: a Rust jev open source alternative Neurono ML released typed-lm, an Apache-2.0 Rust inference engine that turns dense decoder models including Llama, Qwen2, Qwen3, Mistral, Gemma, Gemma2 and Gemma3 into a typed semantic-routing API returning booleans, choices and scores instead of generated text. On a single RTX 3070 with F16 weights, a full request with a 64-token prefix completes in 14 ms prefill plus 36 ms for five batched suffixes, and the recommended CPU mode, a GGUF Q4_K_M checkpoint with the mkl feature, cuts 64-token prefill from a 3.13 s baseline to 0.52 s. The project is drop-in compatible with the Jev (TypeSafe AI) contract and supports LoRA, QLoRA and full training against the same decision-position loss read at inference. Deterministic inference · Adapter training · Apache-2.0 typed-lm turns dense decoder models — Llama, Qwen2, Qwen3, Mistral, Gemma, Gemma2 and Gemma3 — into a typed semantic-routing API. Send a state and typed questions ; receive booleans, choices and scores your code can branch on. No text generation, no parsing. | 7 | 3 | 4 | 1 | |---|---|---|---| | dense model families | question primitives | training methods | forward pass per request | One forward pass means milliseconds, not seconds . On a single RTX 3070 with F16 weights, a full request — the shared prefill plus five batched question suffixes — is answered in tens to hundreds of milliseconds. GPU release, Qwen2.5-1.5B, F16 , RTX 3070 : | Prefix | prefill | 5 batched suffixes | single next token | |---|---|---|---| | 64 | 14 ms | 36 ms | 52 ms | | 256 | 31 ms | 81 ms | 65 ms | | 1024 | 154 ms | 379 ms | 64 ms | Adding a question adds a suffix to the same batched pass, not a new request, so latency grows with the prefix length — not with the number of questions. CPU numbers release, dense F32 | Prefix | Stage | Baseline | + CPU flash | + MKL | |---|---|---|---|---| | 64 | prefill | 3.13 s | 2.34 s | 0.52 s | | 256 | prefill | 8.65 s | 4.97 s | 1.69 s | | 1024 | prefill | 28.23 s | 20.66 s | 13.13 s | | 64 | 5 batched suffixes | 1.44 s | 1.33 s | 0.25 s | | 256 | 5 batched suffixes | 2.47 s | 1.98 s | 0.35 s | | 1024 | 5 batched suffixes | 4.74 s | 4.59 s | 2.57 s | The recommended CPU mode is a GGUF Q4 K M checkpoint with the mkl feature. The session prefix cache skips the prefill entirely for repeated states. More in benchmarks https://neurono-ml.github.io/typed-lm/engineering/benchmarks.html . A large language model answers by generating text token by token. When your software needs a judgment it can branch on, that creates a mismatch: you prompt, you parse, you validate — and you still get a string. typed-lm removes the mismatch. It runs the model once , reads the logits at a single decision position , and returns a typed value with a calibrated distribution. flowchart LR client "Client" :::neutral request "state + questions" :::primary subgraph model "typed-lm-serve" direction TB prefill "shared prefill" :::accent batch "batched decision positions" :::accent end answers "typed answers