{"slug": "how-do-you-run-system-one-decision-models-locally", "title": "How do you run System One decision models locally?", "summary": "Ollaya reached release 0.7.3 on 27 September 2026, four days after its repository appeared on 23 September, and now serves ten model families behind an API it calls wire-identical to TypeSafe's Jev, according to the project's README. The runtime loads ONNX, GGUF and MLX models on CPU, NVIDIA CUDA 12/13 and Apple Metal/MLX, exposes /v1/systemone and /api/decide, and the official TypeSafe Python SDK 0.7.1 reportedly works unchanged against it. On Apple silicon, laya-mlx 0.2.0 loads only the three Laya checkpoints and its author reports 7.4 to 13.4 ms per short question on an M3 Max, while laya-mlx clamps temperatures some Laya checkpoints ship outside its supported range and warns that confidence from affected buckets should be treated as uncalibrated.", "body_md": "# How do you run System One decision models locally? Ollaya, laya-mlx and the runtime slot\n\nTo run a System One decision model locally, you install a runtime, pull an open decision model into it and point the TypeSafe SDK at `localhost`. As of 27 September 2026 the shortest path is [Ollaya](https://stackness.dev/tools/ollaya), which went from a new repository on 23 September to release 0.7.3 in four days and serves ten model families behind an API it calls wire-identical to [Jev](https://stackness.dev/tools/typesafe-jev). On a Mac, [laya-mlx](https://stackness.dev/tools/laya-mlx) is the other finished route. Its author reports 7.4 to 13.4 ms for one short question on an M3 Max.\n\nThe [open-source alternatives post](https://stackness.dev/blog/open-source-alternatives-to-jev-and-what-each-one-actually-replaces) covered the models. This one covers the layer that loads and serves them: the runtime.\n\n## What is a decision model runtime, and how is it different from the model?\n\nThe model is the weights: an encoder or small LLM plus a decision head that turns a state and typed questions into answers with probabilities. The runtime is what loads those weights on your hardware and answers requests. It picks the format (ONNX, GGUF, MLX, Core ML), the device, the API and how batching works, the job Ollama does for chat models.\n\nThe split matters because the runtime can change what the model says. laya-mlx clamps temperatures that some Laya checkpoints ship outside its supported range and warns you to \"treat confidence from the affected buckets as uncalibrated\". [llamacpp-jev](https://stackness.dev/tools/llamacpp-jev) returns \"raw label softmax, not calibrated\". The same weights can give two confidence numbers through two runtimes, so if you threshold on confidence, pin the runtime version as well as the model.\n\nThe [System One explainer](https://stackness.dev/blog/what-is-a-system-one-model-and-where-does-it-go-in-your-stack) put the model slot beside your LLM. The runtime is the slot under it, in the same place [Ollama](https://stackness.dev/tools/ollama) or [vLLM](https://stackness.dev/tools/vllm) sits under a chat model.\n\n## How do you run a System One model locally, end to end?\n\nWith Ollaya it takes three commands, all from its [README](https://github.com/ollaya-dev/ollaya):\n\n```\ncurl -fsSL https://ollaya.dev/install.sh | sh\nollaya run winnow:e4b --preset triage \"Third time this year you've double-charged me...\"\nexport TYPESAFE_BASE_URL=http://localhost:11435\n```\n\nThe daemon listens on `127.0.0.1:11435` and serves `/v1/systemone` plus its own `/api/decide`, which adds timings. Ollaya says the official TypeSafe Python SDK 0.7.1 \"works unchanged\" against it. There is a CUDA Docker image and a Windows installer, and weights come from each model's own Hugging Face repository.\n\nOn Apple silicon, laya-mlx is a Python library, not a server:\n\n```\npip install laya-mlx\npython\nimport laya_mlx as laya\nagent = laya.load(\"aac6fef/laya-mlx\")\nagent.predict(\"I was billed twice.\", {\"department\": {\"type\": \"choice\",\n  \"instructions\": \"Who should handle this?\", \"criteria\": [\"billing\", \"technical\", \"sales\"]}})\n```\n\nIt needs Python 3.11, macOS 14 and an M-series chip. Its README calls it an \"independent MLX port, not an official Convai Innovations release\". [Kev](https://stackness.dev/tools/kev) ships its own server (`python -m kev.serve --run jaredpalmer/kev-4b`), which its author says the TypeSafe SDK works against unchanged.\n\n## Ollaya, laya-mlx or plain llama.cpp: which one loads which models?\n\nOllaya loads the most. laya-mlx loads only Laya. Plain [llama.cpp](https://stackness.dev/tools/llama-cpp) loads any GGUF decoder but needs a shim for the decision API and cannot load Laya at all, because Laya is a ModernBERT encoder with a custom head and has no GGUF build.\n\n| Runtime | Models it loads | Formats | Hardware | API | \n|---|---|---|---|---|\n| Ollaya 0.7.3 | [winnow](https://stackness.dev/tools/winnow-12b) , laya, decider, kev, nli, gliclass, qwen3guard, decision, von, jevk5 | ONNX, GGUF, MLX | CPU, NVIDIA CUDA 12/13, Apple Metal and MLX | `/v1/systemone` ,`/api/decide` , MCP server | \n| laya-mlx 0.2.0 | The three [Laya](https://stackness.dev/tools/laya) checkpoints | [MLX](https://stackness.dev/tools/mlx) | Apple silicon only | Python library | \n| [laya-coreml](https://stackness.dev/tools/laya-coreml) | Laya checkpoints | [Core ML](https://stackness.dev/tools/core-ml) , Neural Engine | Apple silicon | Library | \n| Kev server | Kev 0.8B to 27B | safetensors LoRA | CUDA, ROCm, MLX | `/v1/systemone` | \n| llama.cpp with llamacpp-jev | Any GGUF decoder, for example Qwen3.5-2B | GGUF | Anything llama.cpp runs on | `/v1/systemone` , uncalibrated | \n\nThree projects from the same fortnight bring their own runtime. [SemIf](https://stackness.dev/tools/semif) reads logits on CUDA, llama.cpp or MLX. [NanoJev](https://stackness.dev/tools/nanojev) serves on CUDA only and targets game and agent tasks. cua-s1-forms-coreml is a 706,048-parameter form-action classifier with a 224-byte input limit, built for use inside an app rather than as a general decision model.\n\n## What latency do the local runtimes claim, and who measured it?\n\nAlmost every local number is self-reported by the runtime or model author, and no two use the same request shape. The one independent laya-mlx study measured slower per-question times than the port's README, on a heavier workload, and found throughput dropping under sustained load. Read launch-page numbers as best cases for speed.\n\n| Claim | Number | Measured by | Setup | \n|---|---|---|---|\n| Ollaya, laya:en | 8 to 10 ms | Ollaya | RTX 4090, 5 questions over HTTP | \n| Ollaya, winnow:e4b | 89 ms median | Ollaya | RTX 4090, 5 questions over HTTP | \n| Hosted Jev, as cited by Ollaya | 236 to 276 ms | Two third-party benchmarks | Over the network, client in France or unstated | \n| laya-mlx | 13.42 ms English, 7.39 ms multilingual (P50) | Port author | M3 Max, one short question, model load excluded | \n| laya-mlx, independent | 6.6 to 19.6 ms per question at 50-token states | [Sperix Labs](https://github.com/jayluxferro/laya-mlx-benchmarks) | M4 Max, 16 questions per call, 100 repeats | \n| laya-coreml | 4.98 ms P50 | Port author | M3 Max Neural Engine, one 91-token question | \n| Laya | 32.8 ms | Convai Innovations | Tesla T4, one question | \n| Kev-4B | 18.1 ms model time | Jared Palmer | H100, 6 questions | \n\nOllaya's homepage puts winnow:e4b at 89 ms and 0.722 accuracy beside Jev at 236 to 276 ms and 0.738. Its fastest model, laya:en at 8 to 10 ms, scores 0.361 on the same test. The headline speed and the headline accuracy come from different models. Ollaya does say to read the latencies \"as an order-of-magnitude comparison\", and its developer wrote on [Hacker News](https://news.ycombinator.com/item?id=49848269) that \"the small ones I support today are well below Jev on harder queries\". Its Jev accuracy figure is taken from a benchmark report by Winnow, one of the models it ships.\n\nRead the independent study by Justice Owusu Agyemang before deploying laya-mlx. Over six minutes of sustained load the English checkpoint fell from 23.5 to 5.3 questions a second, with one call taking 15.7 s. The author discloses it was prepared with LLM assistance. The [benchmarking post](https://stackness.dev/blog/how-do-you-benchmark-a-system-one-model-what-jevbench-scores-and-what-calibration-says-it-cannot) covers why accuracy tables without calibration say little.\n\n## Where does the runtime sit next to an LLM gateway in the same stack?\n\nBeside it, on its own endpoint. No runtime I found turns decisions into OpenAI chat completions. Every one speaks the Jev wire format, `POST /v1/systemone`. Your gateway either passes that route through, or your code calls the runtime directly beside the chat traffic.\n\nThe gateways have started to pass it through. [LiteLLM](https://stackness.dev/tools/litellm) forwards `/typesafe/v1/systemone` with cost tracking, and its docs say `TYPESAFE_API_BASE` can point at a custom instance. Pointing that at Ollaya or Kev should work, but nobody has published a test. [OpenRouter](https://stackness.dev/tools/openrouter) hosts Jev at its own `/api/v1/systemone`, outside the chat API. GoModel forwards to hosted Jev or a self-hosted Kev server. [laya-ultrafast](https://stackness.dev/tools/laya-ultrafast) shows the split cleanly: Laya decides in-process on the Mac, and an OpenRouter model writes the text.\n\nIn a stack with both, the chat model sits behind the gateway you already run, and the decision model sits behind a runtime on localhost or a GPU box, reached through that gateway's pass-through or directly. Support in vLLM, LocalAI and NVIDIA Dynamo is still at the pull-request stage, and I could not confirm any of it has merged. The runtime is one more entry among the [AI tools developers keep beside their models](https://stackness.dev/categories/ai-tools), and for now the least settled one.\n\n## When is the hosted API still the right slot?\n\nWhen the decisions are hard, the context is long or the language is not English. Jev takes 64k tokens per request against Laya's 512 to 1,024 and Kev's 384-token training states. In an [independent stress test](https://github.com/gazelle93/decision-models-under-pressure) with 128 candidates, Jev picked correctly 60 percent of the time against 39 percent for Laya. And the input costs $0.042 per million tokens, with output free.\n\nLocal wins on privacy, on volume and on the network round trip. Ollaya logs no states and listens only on 127.0.0.1. It is not automatically cheaper once you count a GPU and the hours spent running it. The pattern people describe on Hacker News is to prototype on Jev, collect a labelled set, then fine-tune a small open model. Kev reports taking Kev-4B from 67.7 to 73.6 percent on one workload for about $1 of H100 time. Jev is still behind a waitlist with no published SLA, which argues for having a local runtime ready either way.\n\nOn Stackness, as of 27 September 2026, Ollama is in one human's stack and Jev is in one, the founder's. llama.cpp, Laya, laya-mlx, MLX and Core ML have none ([data sources](https://stackness.dev/about/data-sources)). Those numbers are too small to carry signal. The [large language models and model tools on Stackness](https://stackness.dev/categories/llms) will show when that changes.\n\n## Key numbers\n\n- **8 to 10 ms** for laya:en on an RTX 4090, 5 questions, measured by Ollaya, which also scores that model at**0.361** accuracy ([Ollaya](https://github.com/ollaya-dev/ollaya) ,**27 September 2026** ).\n- **89 ms** and**0.722** for winnow:e4b against**236 to 276 ms** and**0.738** for hosted Jev, the Jev figures from third parties ([ollaya.dev](https://ollaya.dev/) ).\n- **13.42 ms** P50 for one short question on laya-mlx on an M3 Max, measured by its author ([laya-mlx](https://github.com/mizorewww/laya-mlx) ,**22 September 2026** ).\n- **23.5 to 5.3** questions per second for laya-mlx across six minutes of sustained load on an M4 Max, measured independently ([laya-mlx-benchmarks](https://github.com/jayluxferro/laya-mlx-benchmarks) ,**22 September 2026** ).\n- **4 days** from Ollaya's first commit on**23 September 2026** to release 0.7.3.\n- **64k** tokens per request for hosted Jev, against**512 to 1,024** for Laya ([TypeSafe docs](https://docs.typesafe.ai/) ).\n\n## Quick answers\n\n**How do you run System One decision models locally?** Install a runtime such as Ollaya, pull an open decision model like winnow or kev, and set `TYPESAFE_BASE_URL` to `http://localhost:11435`. The TypeSafe SDK then talks to your machine instead of the hosted API.\n\n**Is Ollaya the same as Ollama?** No. It borrows Ollama's pull, run and serve design for decision models and states it is not affiliated with Ollama or TypeSafe.\n\n**Can I run Jev itself locally?** No. TypeSafe has not released Jev's weights. Local runtimes serve open models trained to answer the same kind of questions.\n\n**What is the fastest local option on a Mac?** laya-coreml reports 4.98 ms and laya-mlx 7.4 to 13.4 ms for one short question on an M3 Max. Both are the port author's own numbers.\n\n**Does llama.cpp run Laya?** No. Laya is an encoder with a custom head and has no GGUF build. llama.cpp can serve decoder-based decision models through llamacpp-jev, with uncalibrated probabilities.\n\n**Are the local latency numbers comparable with Jev's?** Not directly. Local figures exclude the network and Jev's include it, and question counts range from one to sixteen per call.", "url": "https://wpnews.pro/news/how-do-you-run-system-one-decision-models-locally", "canonical_source": "https://stackness.dev/blog/how-do-you-run-system-one-decision-models-locally-ollaya-laya-mlx-and-the-runtime-slot", "published_at": "2026-09-27 10:41:41+00:00", "updated_at": "2026-09-27 11:01:24.845993+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "large-language-models", "developer-tools", "ai-agents"], "entities": ["Ollaya", "laya-mlx", "TypeSafe SDK", "Jev", "Laya", "llama.cpp", "Ollama", "vLLM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-do-you-run-system-one-decision-models-locally", "markdown": "https://wpnews.pro/news/how-do-you-run-system-one-decision-models-locally.md", "text": "https://wpnews.pro/news/how-do-you-run-system-one-decision-models-locally.txt", "jsonld": "https://wpnews.pro/news/how-do-you-run-system-one-decision-models-locally.jsonld"}}