Does an open-weight decision model beat a hosted one? Jev vs. Laya

A developer chose the open-weight Laya typed-decision model over TypeSafe's hosted Jev for routine agent decisions because Laya runs locally on a Mac Studio with no per-token bill, while Jev is an early-access hosted service priced at $0.042 per million input tokens with no output-token charge. The Laya checkpoint, convaiinnovations/laya-typed-decisions, has about 421 million parameters, a ModernBERT-large encoder, and a 1,024-token limit, and was trained on four synthetic workflows: agent observability, customer service, invoice processing, and security incidents. The developer said privacy and control were the decisive reasons and did not calculate a break-even, noting the workstation, electricity, storage, and time still cost money.

You ask a chat model a yes-or-no question and get back three paragraphs. Then your code has to parse the paragraphs, guess what it meant, and hope the JSON is valid this time. I got tired of that for the small, bounded choices my agents make all day. Here's the model I picked, the one I didn't, and the replay numbers behind the decision. This is my account of one deployment. Whether Laya beats Jev in general is a different question, and I didn't test it. The setup A coding agent makes dozens of small decisions that aren't really language tasks. Which specialist should take this job? Does this alert need a human right now? Which local model can handle this request without blowing the memory budget? Prose is the wrong output type for a bounded question. TypeSafe's Jev https://docs.typesafe.ai/introduction is built for exactly this. You hand it a state and a set of predefined questions, and it returns typed answers. A choice picks among named options. A score rates on an ordered scale. A noul returns a probability for a yes-or-no statement. No paragraph to interpret. I liked it immediately. But I had a requirement Jev couldn't meet. I wanted routine decisions running on my Mac Studio, with local data staying local. I also wanted to control the exact model, the permitted options, and what happens when the model is unsure or down. Jev is an early-access hosted service. Laya https://huggingface.co/convaiinnovations/laya-typed-decisions is an open-weight model with the same kind of typed-decision interface, published with downloadable weights and an Apache 2.0 runtime. For this system, that settled it. What's actually going on Both models solve the same narrow problem: turn a compact state and a few questions into structured decisions. Neither writes code or reasons through a long investigation. It's a specialist. Your main model keeps its job. Jev's managed API is a good deal on paper. TypeSafe lists $0.042 per million input tokens with no output-token charge https://typesafe.ai/blog/introducing-system-one-models-and-jev and says Jev handles choices with up to 255 options. Hosted also means nobody babysits model files, accelerator compatibility, startup time, or process supervision. That's real work I chose to own. Laya's trade-offs run the other way. I downloaded a pinned checkpoint once and serve it over a loopback-only API. Routine decision requests never leave the machine. There's no per-token bill, though the workstation, electricity, storage, and my time still cost money. I didn't calculate a break-even. Privacy and control were the decisive reasons. Cost was a side effect. I have no savings number to give you. The checkpoint, convaiinnovations/laya-typed-decisions , has about 421 million parameters, a ModernBERT-large encoder, and a 1,024-token limit. Its publisher trained it on four synthetic workflows: agent observability, customer service, invoice processing, and security incidents. Model routing isn't one of those, which is a reason to test rather than assume. Here's what makes it different from asking a chat model for JSON. The Laya runtime renders the question, each option, and the state into one token sequence, with a marker before each option. A bidirectional encoder reads the whole thing, a small decision head scores the marker positions, and a softmax turns those scores into a distribution over options. Several questions against one state batch into a single call. There's no autoregressive text output, so there's nothing to parse. The published runtime https://github.com/NandhaKishorM/laya shows the sequence construction and the head. For a choice , my code gets an option key and a probability for every allowed key. For a score , a distribution over ordered levels plus the expected level. For a noul , the probability of true . The runtime also reports a confidence derived from the distribution's normalized entropy. That confidence is not the probability the downstream task will succeed. I picked thresholds on a separate development set, and I'd want fresh evidence before reusing them on a different model roster. Laya's authors trained with rewards based on proper scoring rules, meant to encourage honest probabilities. The model card still warns it's overconfident and needs domain-specific calibration. A typed answer can be structurally perfect and factually wrong. I treat Laya as a recommender inside a program, never as a source of permission or truth. The fix My host is a Mac Studio with an M3 Ultra and 256 GB of unified memory. That hardware is part of the result. A smaller machine needs its own memory and latency tests. The original implementation brief got two things wrong. It assumed plain transformers.AutoModel could load Laya's decision model, and that the typed checkpoint lived only in a subfolder of the base model. Neither held. The typed checkpoint is its own repository and AutoModel loads only the encoder shape. The published laya runtime reconstructs the custom decision head, so the loader uses that instead of mistaking a bare encoder for a working decision engine. Around the checkpoint sits a FastAPI service that accepts the TypeSafe wire shape, so a client written for Jev can point at my box: { "model": "laya-local", "state": "The production API is down and customers are blocked.", "questions": { "route": { "type": "choice", "instructions": "Which team should handle this?", "criteria": { "billing": "Invoices and refunds", "technical": "Bugs, outages, and API errors" } } } } My code defines the labels before Laya ever sees them. The response identifies itself as laya-local-v1 . Accepting a Jev-style client alias doesn't pretend Jev supplied the answer. The service listens on 127.0.0.1:8017 , because Docker already had port 8000 on this machine. The service fails closed. It loads from an explicit local directory and refuses to download during inference. In production, it requires Apple's Metal accelerator and warms the model before reporting ready. It counts the exact rendered token sequence and rejects an over-budget request instead of letting the runtime quietly shorten the state to fit. It caps Choice at 20 options by default and serializes inference so concurrent callers can't pile uncontrolled load onto the accelerator. The more important change sits outside the model. A gateway first drops any model that isn't eligible: wrong capability, wrong data class, too little context, unhealthy, past deadline, or over the memory budget. Laya gets at most five survivors plus an explicit abstain option. It never sees an ineligible model and can't grant one permission. The gateway re-validates the returned option, confidence, margin, model identity, and eligibility before reserving capacity. If Laya abstains or times out, a permitted local fallback runs. If nothing is eligible, the gateway refuses the request. Inside ClaudeClaw, Laya now handles bounded specialist dispatch and how noncritical alerts get presented. A user who names a specialist still wins. Model advice can't suppress a critical alert. I also tried Laya for post-failure recovery choices. That profile failed acceptance, so it stays disabled, and the deterministic recovery rules stay in charge. What the replay showed I wanted a measured result for my own routing decision, not a latency figure copied from someone else's machine. The service's API, engine, SDK compatibility, and routing-contract suites recorded 59 passing tests, with one optional real-checkpoint test skipped. Separate HTTP runs exercised the real checkpoint on Metal. The held-out replay used previously recorded outcomes for two models qualified for this setup, Qwen3 Coder 30B and GPT-OSS 120B. The replay simulated model health and resources. The Laya inference was real. Nobody reran the coding tasks. I chose the profile on 20 development decisions, froze it, then ran 40 held-out decisions. Each task appears with more than one latency preference, so those 40 are correlated, not 40 independent wins. On the held-out set, deterministic quality-and-latency ordering produced 33 acceptable decisions out of 40, or 82.5%. Laya, with the same eligibility rules and local fallback, produced 37 of 40, or 92.5%. Four more acceptable decisions, a 10-percentage-point gain on this one replay. Laya directly selected a model 15 times, the gateway fell back 17 times, and the gateway blocked eight cases where no candidate was available. It never selected an ineligible model. The three misses matter. One near-threshold case fell back to a model that had failed its source task. Two cases had no acceptable candidate because both underlying models had failed. Laya can't repair a task; neither candidate can do. Latency is where it got humbling. The first held-out run had a routing p95 of 143.54 ms. A later supervised replay on a busy host kept the identical 37-of-40 outcome but pushed p95 to 466.69 ms, past my 100 ms aspiration and past the gateway's 300 ms decision timeout. That run is why the timeout and fallback exist. They fired for real. Twenty near-threshold decisions repeated three times kept their selected-or-abstained status every time, but repetition doesn't add independent cases. And the busy-host replay didn't prove a live gateway enforcing 300 ms would keep all 37 acceptable choices under that load. Live checks went a bit further. The supervised Laya service and gateway reported ready. Gateway acceptance covered a direct Laya choice, local fallback, and duplicate-request rejection. ClaudeClaw's compiled dispatch checks picked the expected specialist three times. Those are bounded integration checks. They say nothing about the next agent task I haven't written yet. Why this matters I did not run Jev and Laya side by side on these 40 cases. The 10-point gain is against my deterministic router, not against Jev. Laya's publisher reports 76.6% top-choice accuracy on its 2,000-decision test split against a published Jev figure of 72.7%. The same card says Jev matches the reference probability distributions better and has better raw calibration. Those numbers are leads. I didn't run a head-to-head under one protocol. The card also warns about overconfidence, English-only training, and accuracy falling as the option list grows, while TypeSafe says Jev supports far larger choices. I'd test Jev directly before claiming Laya is more accurate, faster, or cheaper for the same production decisions. What I can say is narrower. On one frozen replay, the local system made more acceptable routing decisions than my previous ordering, and it kept the decision input, checkpoint, thresholds, and fallback policy under my control. Jev may well be the better product if you want a managed API, have large choice sets, or don't want to run local inference. I chose Laya because local processing was a requirement, and I was willing to own the testing and operations that came with it. Quick reference - Typed decisions: choice returns an option plus a probability per option, score a distribution over ordered levels, noul a probability of true. Nothing to parse. - Laya checkpoint: convaiinnovations/laya-typed-decisions , about 421M parameters, ModernBERT-large, 1,024-token limit, Apache 2.0. - Load it with the published laya runtime, not bare transformers.AutoModel , which gives you an encoder with no decision head. - Deterministic code filters eligibility first. The model only ranks survivors, plus an explicit abstain option. - Re-validate everything the model returns before acting. A decision receipt records; it doesn’t authorize. - Confidence is normalized entropy, not task success probability. Calibrate thresholds on a development set and freeze them before held-out. - Budget for timeouts. My p95 went from 143 ms to 467 ms on a busy host with identical decisions. Terms worth knowing Some of this vocabulary comes from a small corner of the AI world. Short definitions, in the order they show up. - Typed decision. An answer with a fixed shape your code can use directly, such as one option from a list or a number between 0 and 1. The opposite of a paragraph you have to parse. - State. The text or JSON you hand the model describes the situation it's deciding about. An alert, a task description, a customer message. - Choice, score, noul. Jev's three question types, which Laya copies. A choice picks one option from a list you define. A score picks a level on an ordered scale you define, like low, medium, critical. A noul is a yes-or-no statement and returns the probability that it's true. The term is TypeSafe's. - Open-weight model. A model whose trained weights you can download and run yourself, as opposed to one you can only reach through a vendor's API. - Checkpoint. A saved set of model weights. "Pinned checkpoint" means I locked to one exact version and verify it hasn't changed. - Loopback-only. The service listens on 127.0.0.1, so only programs on the same machine can reach it. Nothing on the network can. - Token. The unit a model reads. Roughly a short word or word fragment. Laya's 1,024-token limit is the total length of the state, question, and options after they're rendered together. - Encoder, bidirectional. A model that reads the entire input at once, in both directions, and produces a representation of it. It doesn't generate text. ModernBERT is a 2024-era encoder family. Chat models are decoders, which is a different design. - Autoregressive. Generating output one token at a time, each token depending on the ones before. That's how chat models write. Laya doesn't do it, which is why there's nothing to parse and why it's fast. - Decision head. A small neural network bolted onto the encoder that turns its output into scores, one per option. The base encoder alone can't make decisions. This is the piece transformers.AutoModel doesn't load. - Softmax. A math step that turns a list of raw scores into probabilities that add up to 1. - Normalized entropy. A measure of how spread out a probability distribution is, scaled from 0 to 1. Laya's confidence is 1 minus that. All options equally likely gives confidence 0. One option at 100% gives confidence 1. It measures how sure the model is, not how right it is. - Calibration, overconfidence. A calibrated model that says "80%" is right about 80% of the time. An overconfident one says 80% and is right less often. Laya's own model card says it's overconfident. - Proper scoring rule. A training reward designed so the model scores best only when it reports what it believes. It's meant to discourage bluffing. It doesn't guarantee calibration on your data. - Development set, held-out set. Two separate batches of test cases. You tune thresholds on the development set, freeze them, and then measure on the held-out set the model has never influenced. Tuning on the held-out set makes the result meaningless. - Correlated cases. Test cases that share an underlying task, so they tend to succeed or fail together. Forty correlated cases carry less evidence than forty independent ones. - p95. The latency that 95% of requests came in under. A better measure of "how slow does it get" than the average. - Metal, MPS. Apple's GPU framework and PyTorch's backend for it. Requiring MPS means the service refuses to start if the model would silently run on the CPU. - Wire shape. The exact JSON layout of a request and response. Matching Jev's wire shape means a client written for Jev works against my service without changes. - Fails closed. When something is missing or wrong, the service stops instead of guessing. No checkpoint, no start. Request too long, rejected, not trimmed. - Abstain. An explicit "none of these" option the gateway always adds. It lets Laya decline instead of forcing a pick. - Eligibility gateway. The deterministic code that filters out models Laya isn't allowed to choose before Laya sees the list, then re-checks the answer afterward. Found this useful? I share practical lessons from my systems engineering journey at As The Geek Learns https://astgl.substack.com .