Laya: Multilingual, non-autoregressive System 1 decision engine

Laya released a multilingual, non-autoregressive "System 1" decision engine that returns typed decisions across 100+ languages in a single forward pass in 33 ms, installable via `python -m pip install laya` on Python 3.10 or newer. The library's `laya-multilingual` checkpoint reads up to 8,192 tokens with `max_len=8192`, and a router selects the appropriate checkpoint per request, sending non-English text to the multilingual model. Optional extras include `laya[serve]`, `laya[mcp]`, `laya[langchain]`, `laya[llamaindex]`, `laya[crewai]`, `laya[onnx]`, and `laya[fast]`, with a TypeScript/Node.js/browser port published to npm as `laya-ts`.

Multilingual, non-autoregressive System 1 decision engine. Typed decisions over 100+ languages in a single forward pass — 33 ms — trained with reinforcement learning against strictly proper scoring rules RLCD , with a router that picks the right checkpoint per request. python -m pip install laya With uv https://docs.astral.sh/uv/ , run uv add laya in a uv project or uv pip install laya in a virtual environment. Python 3.10 or newer. Optional extras: laya serve HTTP server , laya mcp MCP server , laya langchain LangChain and LangGraph , laya llamaindex LlamaIndex selectors , laya crewai CrewAI routing , laya onnx ONNX Runtime , laya fast TileLang GPU fast path . Step-by-step setup for each platform, CPU-only or GPU PyTorch builds, and troubleshooting are in Installation details installation-details . For TypeScript / Node.js / browser, see laya-ts/ https://github.com/NandhaKishorM/laya/blob/main/laya-ts . npm releases npm install laya-ts are published from this repository's laya-ts-v release tags. Long documents. laya-multilingual reads up to 8,192 tokens with max len=8192 . Measured accuracy and time by document length, reproducible with research/scripts/bench long context.py https://github.com/NandhaKishorM/laya/blob/main/research/scripts/bench long context.py : Long documents: laya-multilingual reads up to 8,192 tokens. It ships with a 1,024-token limit that cuts long documents off, so pass max len=8192 for them: result = router.predict long document, questions, model="multilingual", max len=8192 In the table above, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary 8 to 17 of 20 , so check long-document accuracy on your own data. Short inputs give identical answers with max len=8192 , and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU. Name the checkpoint with model="multilingual" , since long mostly-English text would otherwise route to the English checkpoint. python from laya import Router router = Router downloads a checkpoint on first use; Router preload=True loads all three up front state = "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan." questions = { "department": {"type": "choice", "instructions": "Which department should handle this?", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else"}}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": "not urgent", "soon", "blocking" }, "churn risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"}, } result = router.predict state, questions print result "answers" "department" "choice" billing print result "answers" "churn risk" "noul" probability the answer is yes print result "routing" "model" english The same call works in any of 100+ languages. The Router detects the script and language and sends non-English text to laya-multilingual : for text in "मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।", "La aplicación se cierra cada vez que abro la configuración." : r = router.predict text, {"department": questions "department" } print r "routing" "model" , r "answers" "department" "choice" multilingual billing multilingual technical From the command line, laya "My payment failed twice" --preset triage answers a ready-made question set. More in the full quickstart quickstart-route-mode-recommended and the docs https://nandhakishorm.github.io/laya/ . The shipped checkpoints work zero-shot, but fine-tuning on decisions from your own domain is where accuracy jumps. On the typed-decisions benchmark 2,000 decisions across four workflows , the fine-tuned laya-typed-decisions checkpoint scores 0.766 accuracy, against 0.362 for the base English checkpoint on the same decisions. Fine-tuning notebook https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya finetune typed decisions 2xT4 kaggle.ipynb : runs the whole loop on Kaggle's free 2x T4 GPUs build the dataset, train, fit calibration temperatures, evaluate, and push the result to the Hub . Details in Fine-Tuning fine-tuning . nandhakishorm.github.io/laya https://nandhakishorm.github.io/laya/ : guides for prediction hooks https://nandhakishorm.github.io/laya/hooks/ , schema-driven decisions https://nandhakishorm.github.io/laya/structured/ , Docker https://nandhakishorm.github.io/laya/docker/ and LangChain and LangGraph https://nandhakishorm.github.io/laya/langchain/ , plus a full API reference https://nandhakishorm.github.io/laya/reference/ . - ONNX catches up with PyTorch. ONNXAgent gains predict batch with sort by length , predict long and decide batch , scripts/export onnx.py --quantize writes a per-channel INT8 copy for CPU, and laya-evals run --onnx scores an export with the same gates as the torch path. - Opt-in abstention. min confidence= on predict , predict batch , decide and decide batch flags answers below a threshold on answer confidence with low confidence: True , and decide returns None for them. - Batch everywhere. decide batch , Router.predict long , laya --batch FILE , the MCP laya predict batch / laya route batch / laya decide tools, and LangChain batch / abatch all run on shared forward passes. New LayaDecision LangChain , LlamaIndex selectors laya llamaindex and CrewAI routing laya crewai . - Per-request token budget. max len / head max len now reach every surface: laya-serve capped by LAYA MAX TOKEN BUDGET , Router.predict batch requests, the CLI --questions , --max-len , --head-max-len , MCP tools and LangChain nodes. - Operations. LAYA MAX LOADED , LAYA REVISION and per-checkpoint SHA-256 maps; /health reports the device a checkpoint really runs on and its CPU-fallback count; the 503 busy answer carries Retry-After ; compile=True no longer recompiles for every request shape. - Stricter inputs. A null or duplicate choice label, a short temperature list, a None state and non-dict questions are refused with a message that names them, and usage "options" says when the head budget left two options with the same tokens. Laya evaluates typed questions choice , score , noul over any state text, email, ticket or JSON document in a single forward pass — 33 ms for one question, 7.2 ms/question batched, measured on a T4. No text generation, so nothing to parse and nothing to hallucinate. Three checkpoints, and a Router that picks between them per request: | | encoder | params | context | use it for | |---|---|---|---|---| | laya https://huggingface.co/convaiinnovations/laya | ModernBERT-large | 421M | 512 | English | | laya-multilingual https://huggingface.co/convaiinnovations/laya-multilingual | mmBERT-base | 322M | 1024 up to 8,192 | 100+ languages, 2x faster | | laya-typed-decisions https://huggingface.co/convaiinnovations/laya-typed-decisions | ModernBERT-large | 421M | 1024 | the typed-decisions workflows | Python 3.10 or newer. The dependencies set that floor: huggingface hub 1.x, transformers 5.x and torch 2.14 all require 3.10. Optional PyTorch build selection: If you need a CPU-only or GPU-specific PyTorch build, follow PyTorch's installation guide https://pytorch.org/get-started/locally/ after creating your virtual environment and before installing Laya. Replace pip or pip3 in the selected command with the environment's Python executable followed by -m pip . If you already use a virtual environment, install the PyPI release with: python -m pip install laya For a new environment, choose the commands for your platform below. Run them from your project directory; the explicit Python paths keep installation and verification in the same environment. macOS / Linux with Python 3.10 or newer : On Debian/Ubuntu, the system Python may require sudo apt install python3-venv before creating a virtual environment. If venv reports that ensurepip is unavailable, install that package and retry. python3 -m venv .venv .venv/bin/python -m pip install laya .venv/bin/python -I -c "import laya; print laya. version " Windows PowerShell this example uses an installed Python 3.11 : py -3.11 -m venv .venv .\.venv\Scripts\python.exe -m pip install laya .\.venv\Scripts\python.exe -I -c "import laya; print laya. version " Both checks print the installed Laya version without loading a checkpoint. -I excludes the current directory from the import search path, so a local source copy cannot mask a missing installation. Keep using the same virtual environment's Python when running your application. Intel GPU XPU Install a supported Intel GPU driver first. For an XPU-enabled PyTorch build, install its wheel before Laya; the default PyPI wheel may be CPU-only. PyTorch's validated hardware and OS list is in the Intel GPU guide https://docs.pytorch.org/docs/2.14/notes/get start xpu.html . .\.venv\Scripts\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/xpu .\.venv\Scripts\python.exe -m pip install laya .\.venv\Scripts\python.exe -c "import torch; print torch.xpu.is available " For a source checkout, replace pip install laya with pip install -e . . Laya automatically selects an available XPU when no device is specified; you can also request one explicitly with device="xpu" in laya.load or Router device="xpu" . Install from GitHub To use the development version instead of the PyPI release, create the virtual environment above and replace its installation command with the appropriate command below. Git must be installed. macOS / Linux .venv/bin/python -m pip install "git+https://github.com/NandhaKishorM/laya.git" Windows PowerShell .\.venv\Scripts\python.exe -m pip install "git+https://github.com/NandhaKishorM/laya.git" Run the same version check afterward. The GitHub version follows the repository's default branch and may differ from the published release. Install with uv uv https://docs.astral.sh/uv/ creates the virtual environment, downloads a matching Python if none is installed, and installs into it. The commands are the same on macOS, Linux and Windows PowerShell: uv venv --python 3.12 uv pip install laya uv pip install targets the .venv in the current directory without activating it, so run the version check for your platform above afterward. Extras and the GitHub version install the same way: uv pip install "laya serve " , uv pip install "git+https://github.com/NandhaKishorM/laya.git" . For a CPU-only or GPU-specific PyTorch build, add --torch-backend=auto to pick the build that matches the machine's GPU driver, or name one such as --torch-backend=cpu ; uv's PyTorch guide https://docs.astral.sh/uv/guides/integration/pytorch/ has the list. If your application is a uv project, add Laya as a dependency instead: python uv add laya uv run python -I -c "import laya; print laya. version " --torch-backend applies to uv pip only; in a uv project, uv's PyTorch guide shows how to set the PyTorch index in pyproject.toml . Model setup and troubleshooting Continue with the Router quickstart quickstart-route-mode-recommended to run inference. Loading a Hub checkpoint requires access to Hugging Face on its first download; the quickstart's Router preload=True loads all three configured checkpoints at construction. - ModuleNotFoundError: No module named 'laya' : run both installation and your script with the same virtual environment's Python executable shown above. In an editor, select that interpreter as well. - Missing rl agent config.json : this file ships with a Laya checkpoint alongside model.safetensors ; it is not a configuration file you need to create in the source repository. For a local model, pass the directory containing those checkpoint files. Installing the package also installs a laya command for quick local testing, no script needed: laya "I was charged twice, please refund" routing decision only; works offline, no download laya "Refactor this service" --predict full answers downloads the checkpoint on first use laya "Mein Konto wurde zweimal belastet" --lang de force a language instead of detecting it laya "My payment failed twice" --model ml pin a checkpoint: names, aliases and casing all resolve as the SDK resolves them laya "My payment failed twice" --preset triage answer a ready-made preset triage, email, guard, moderation, router laya --batch tickets.txt --predict score a file of requests, one per line, in one batch cat tickets.txt | laya --batch - --predict --json stdin; one JSON line of answers per request laya "Where is my card" --questions intents.json answer your own questions, written in a JSON file laya interactive mode Routing alone never downloads a checkpoint, so it returns in milliseconds. --predict loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing. --batch with or without --predict sends the whole file through Router.predict batch in one process, so the requests share checkpoint loads and forward passes — measured 2.6x on 20 tickets vs looping predict one by one, with --batch-size N to bound the forward pass, --sort-by-length to group similarly sized requests inside it, and --json for JSONL output. Batch routing laya --batch FILE , no --predict likewise answers with route batch in one pass, still without loading anything. --sort-by-length reaches the length grouping Agent.predict batch has done since 294, which until now only the library call in Batch Mode batch-mode-score-many-states-in-one-forward-pass could ask for. A forward pass pads every state in it to the longest one, so a two-line request sharing a pass with a two-paragraph one spends most of its compute on padding; grouping by length first puts comparable sizes together. Measured on 128 real Yelp reviews of 69-2,293 characters, run as laya --batch FILE --predict --model laya --batch-size 8 with and without the flag, four interleaved rounds on an Apple-silicon Mac at the CLI's own device pick: 41.25 s median unsorted against 29.23 s sorted, a paired median of 1.42x 1.39-1.44x across the four pairs , with all 128 decisions identical and still in input order. research/ measures the same knob at 2.15x over 10,000 synthetic multilingual tickets. It needs --batch-size N with 1 < N < the number of requests — a single pass over the whole file has no group to reorder — and the CLI says so on stderr rather than printing a same-speed result and leaving you to notice. --questions takes the same question dict the SDK takes, as JSON: either the mapping itself, or {"state key": "body", "questions": {...}} when the question's instructions name a field other than request . It implies --predict , and a question set with many labels usually wants --head-max-len with it: on 58 MASSIVE-INTENT labels written as one choice question, the English checkpoint goes from 24/58 correct at its default 192-token option budget to 34/58 at --head-max-len 384 , for about 1.4x the per-request time on CPU. Widening it further costs the accuracy back, because max len then leaves fewer tokens for the request itself. Honest limits honest-limits describes the same budget ceiling for a 77-option question. examples/server.py is a self-contained FastAPI app for testing Laya without writing any code: a two-pane playground edit the request as a form or as JSON, run it with Ctrl+Enter, read each answer's full distribution and calibrated confidence, copy it as curl or Python , plus a plain JSON API /predict , /predict/batch for scripting against. pip install "laya serve " python examples/server.py http://127.0.0.1:8000 Open http://127.0.0.1:8000 in a browser for the playground, or hit it directly: curl -s localhost:8000/predict -H 'content-type: application/json' -d '{ "state": {"body": "We were billed twice for March. Please refund it today."}, "questions": { "department": {"type": "choice", "instructions": "Which department should handle this?", "criteria": {"billing": "invoices, payments, refunds", "other": "everything else"}}, "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": "not urgent", "soon", "critical" } } }' | python -m json.tool --no-preload loads checkpoints lazily instead of all three up front; --device cuda|cpu|mps pins the device. See python examples/server.py --help for the rest. A request body may carry model to pin a checkpoint instead of letting the router choose, and it takes exactly the spellings laya --model takes: names, aliases and casing all resolve through laya.router . GET /models lists the checkpoints and the aliases alongside them. /predict/batch accepts two optional body fields that control the shape of the forward passes without changing any answer: batch size states per pass; omit it and the whole batch is one pass and sort by length group similarly sized states so each pass pads to a shorter maximum — it needs a batch size below the number of states, since one pass has nothing to reorder . Measured on the running server: 64 real reviews of 91–2,293 characters, one choice question, batch size: 8 , three interleaved rounds on MPS — the default one-pass shape 5.35s median against 3.04s sorted, a paired median of 1.77x min 1.76x , and 64/64 labels identical in input order. batch size: 8 on its own was 4.95s, so the length grouping is what earns the second factor: 1.66x over the sized batch. laya-client https://github.com/NandhaKishorM/laya/blob/main/sdk/typescript/README.md supports JavaScript and TypeScript applications over HTTP to a self-hosted laya-serve /v1/systemone server. It includes inferred answer types, all five question presets, ESM/CommonJS exports, and cancellation. From this checkout, install and start the Python inference service: pip install -e '. serve ' LAYA HOST=127.0.0.1 LAYA MODELS=english laya-serve In another terminal: cd sdk/typescript npm ci npm run build node examples/triage.mjs The npm package is named laya-client . Use it when a JavaScript or TypeScript application talks over HTTP to self-hosted Python laya-serve . Use laya-ts when inference must run directly inside JavaScript through its local ONNX runtime, without a Python server. See installation, examples, and publishing https://github.com/NandhaKishorM/laya/blob/main/sdk/typescript/README.md and the repository analysis https://github.com/NandhaKishorM/laya/blob/main/docs/typescript-sdk.md for architecture and scope. To try the Python SDK in a CPU container, see the Docker Compose quickstart https://github.com/NandhaKishorM/laya/blob/main/docs/docker.md . It runs a sample request and keeps downloaded models between runs. Laya ships three checkpoints. The built-in Router is the recommended entry point: it evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass. python from laya import Router Preload checkpoints into memory for instant sub-35ms routing router = Router preload=True 1. State in any language or schema state = { "from": "user@acme.com", "subject": "Duplicate charge on invoice 4411", "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan." } 2. Define your typed questions questions = { "department": { "type": "choice", "instructions": "Which department should handle this request?", "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "sales": "pricing, new contracts", "other": "everything else" } }, "urgency": { "type": "score", "instructions": "How urgent is this request?", "criteria": "not urgent", "soon", "critical deadline or blocking issue" }, "churn risk": { "type": "noul", "instructions": "Does the user threaten to cancel or leave?" }, "refund requested": { "type": "noul", "instructions": "Does the user explicitly request a refund?" } } 3. English state - automatically routed to laya ModernBERT-large, 39.5 ms res en = router.predict state, questions print "Department :", res en "answers" "department" "choice" - billing confidence: 0.94 print "Routing :", res en "routing" "model" - english 4. Hindi state - automatically routed to laya-multilingual mmBERT-base, 32.8 ms res hi = router.predict {"body": "मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें।"}, questions print "Department :", res hi "answers" "department" "choice" - billing confidence: 0.86 print "Routing :", res hi "routing" "model" - multilingual 5. Explicit override when you want a specific checkpoint res td = router.predict state, questions, model="typed-decisions" Every result carries full routing metadata explaining why the choice was made: res hi "routing" { 'model': 'multilingual', 'repo': 'convaiinnovations/laya/multilingual', 'reason': 'non-Latin script devanagari, 100% of letters ; the English checkpoint cannot read it' } Inspect a routing decision without running any forward pass: router.route {"body": "Der Kunde wurde zweimal belastet"}, questions .reason "Latin script but language looks like 'de', not English" Very short Latin-script text often carries nothing that identifies its language "Quero cancelar" , "Esqueci minha senha" . Such text goes to default , which is "english" unless you change it. If most of your traffic is not English, set: router = Router default="multilingual" router.route {"body": "Esqueci minha senha"} .model - multilingual router.route {"body": "Please refund the duplicate charge"} .model - english If a lazy router receives an interleaved workload whose requests route to different checkpoints, calling predict in a loop can still cause unnecessary checkpoint churn when the required checkpoints exceed the resident cache, for example with max loaded=1 or when typed-decisions is also used. Router.predict batch routes the full workload first, groups requests by checkpoint, then groups requests with the same question schema within each checkpoint. Each compatible group is dispatched to Agent.predict batch so states can share forward passes, and results are restored to the original request order. requests = {"state": "Please refund invoice 1", "questions": questions}, {"state": "تم خصم المبلغ مرتين", "questions": questions}, {"state": "Please refund invoice 2", "questions": questions}, results = Router max loaded=1 .predict batch requests results stay in input order while compatible requests are batched by checkpoint Each item can independently set model , task , lang , lang guess , or the token budget max len , head max len . Use route batch requests when you only want the ordered routing decisions without loading any checkpoint. predict many is an alias for predict batch . Requests are validated before model loading. Different requests may use different question schemas; requests sharing a checkpoint, question schema and token budget are passed together to Agent.predict batch . Prediction hooks prediction-hooks installed on the Router run once per request, as they do for predict , so a redaction hook rewrites every state before the model sees it. Requests that share a checkpoint run all their start hooks before their shared forward pass; see docs/hooks/lifecycle.md https://github.com/NandhaKishorM/laya/blob/main/docs/hooks/lifecycle.md routerpredict batch . You can also bound the Agent-level forward-pass batch size, and forward the length grouping knob to every group see Batch Mode batch-mode-score-many-states-in-one-forward-pass : results = router.predict batch requests, batch size=8, sort by length=True On a shared benchmark 17,416 questions, one T4 GPU, identical questions per model : | Benchmark / Task | English laya | Multilingual laya-multilingual | Router Routed | |---|---|---|---| | MASSIVE intent, English | 0.783 | 0.657 | 0.783 | | MASSIVE intent, 13 other languages | 0.306 | 0.451 | 0.451 | | XNLI, English | 0.860 | 0.843 | 0.860 | | XNLI, 14 other languages | 0.521 | 0.731 | 0.731 | | Languages usable 3x random | 23 / 51 | 48 / 51 | 48 / 51 | | Latency, 1 question T4 GPU | 39.5 ms | 32.8 ms | 32.8 ms | | Latency, 10 questions batched | 158.6 ms | 72.3 ms | 72.3 ms | The English checkpoint collapses on non-Latin scripts Khmer scores 0.000 accuracy at 0.952 confidence , the raw-temperature figure; 0.705 as served after the clamp . Because the model stays confident while being wrong, confidence gating cannot save you. Router detects the script in <0.5 ms pure Python before the forward pass. A cold checkpoint build costs seconds; language detection costs microseconds. The lazy default keeps two checkpoints resident — english and multilingual , the only two automatic routing chooses between — so a language flip costs detection only once each has been built. max loaded=1 rebuilds the checkpoint it just evicted on every switch measured at a 7.4 s median reload on CPU and 10.3 s on T4 , and traffic that only ever sees one language never builds the second, so the default costs a single-language deployment nothing. For a server or production app, preload: Every checkpoint resident in memory; language flips cost detection only <1 ms router = Router preload=True router = Router preload=True, device="cuda" Or preload only the specific checkpoints you serve: router.preload "english", "multilingual" If your app already built an agent, attach it to avoid duplicate VRAM: router.attach "english", existing agent Manage resident memory default keeps two hot: english + multilingual, LRU eviction router = Router max loaded=3 keep all three hot, e.g. with auto task detection router = Router max loaded=1 memory-constrained host, reloads on every switch router.unload free memory | Deployment Mode | Per-Request Latency | Model Reloads | |---|---|---| | Router lazy, max loaded=2 | detection only <1 ms on a switch, after each language's first load | 1 the first time a language appears | | Router max loaded=1 | 7 to 10 s on every language switch | 1 per switch | | Router preload=True | 32.8 ms GPU / 193–464 ms CPU | none | A rebuild still re-reads the checkpoint, but each checkpoint's tokenizer is parsed once per process and reused by every Agent — including one the Router rebuilds after eviction. The multilingual tokenizer.json alone is 34 MB / 256k vocab, several times the cost of applying its weights. Preloading is still the right answer for a server: it removes the rebuild rather than making it cheaper. Routing asks one question: can the English checkpoint read this state? The built-in detector answers it from the script and a function-word heuristic, and is deliberately dependency-free. That heuristic is best-effort on Latin-script languages it holds no word list for, so a short request can carry no usable signal: python from laya.lang import analyse analyse "Care este ora in Tokyo?" {'script': 'latin', 'language': 'en', 'is english': True} - the English checkpoint If you already run a language-identification model, hand routing the answer instead of relying on the heuristic. lang guess takes a language code or a callable receiving the state, and is checked after an explicit lang= and before detection: A code you already know router.predict state, questions, lang guess="ro" A callable, e.g. wrapping fastText, CLD3 or a transformer LID router.predict state, questions, lang guess=lambda s: my lid s Or install one for every request on a server router = Router preload=True, lang guess=my lid The hint only decides English or not : a code whose primary subtag is en , eng or english routes to the English checkpoint, and every other code that names a language routes to the multilingual one. "en US" and "en US.UTF-8" are read as English, so $LANG can be passed straight through. Returning None , or a code that names no language, makes it abstain and the built-in detector decides as before — so a LID model that is unsure does not force a checkpoint. C , POSIX and C.UTF-8 abstain, which matters because C.UTF-8 is the default $LANG in the official Python image: passing it through no longer pins every request to the multilingual checkpoint, which is what it used to do. The ISO 639-2 special codes und , zxx and mul abstain for the same reason. An explicit model= , task= or lang= still wins, and the default path is unchanged. laya.serve exposes the Router over HTTP on the same POST /v1/systemone wire protocol as TypeSafe's hosted Jev API. Laya's answer payload is already schema-identical to what Jev returns choice / score / noul answers and a {input tokens, output tokens} usage block , so an existing Jev client — e.g. the hs-jev https://github.com/getmissionctrl/hs-jev Haskell client — just needs its baseUrl repointed; nothing else changes. pip install "laya serve " adds fastapi + uvicorn + python-multipart LAYA DEVICE=cuda LAYA PRELOAD=1 laya-serve binds 0.0.0.0:8000, preloads all 3 checkpoints curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ "state": {"body": "billed twice, refund please or we cancel"}, "questions": {"dept": {"type": "choice", "instructions": "which team?", "criteria": {"billing": "refunds", "tech": "bugs"}}} }' To evaluate multiple states in a single call, send a states array to POST /v1/systemone/batch capped at 64 states : curl -s localhost:8000/v1/systemone/batch -H 'content-type: application/json' -d '{ "states": {"body": "billed twice, refund please"}, {"body": "cannot login, getting 500 error"} , "questions": {"dept": {"type": "choice", "instructions": "which team?", "criteria": {"billing": "refunds", "tech": "bugs"}}} }' The endpoint shares forward passes via Router.predict batch and returns an array of results in matching order along with aggregated total usage . Configuration is by environment variable: LAYA HOST , LAYA PORT , LAYA DEVICE , LAYA PRELOAD , LAYA MODELS comma list to preload , LAYA THREADS cap torch intra-op threads for CPU inference — keep at or below physical cores , LAYA AUTO TASK , LAYA MAX LOADED checkpoints resident at once, 2 by default; raise it to 3 when LAYA AUTO TASK makes a third one reachable on demand, or the server rebuilds one every time routing switches , and LAYA API KEY when set, clients must send Authorization: Bearer <key . A client's model field is honoured when it names a Laya checkpoint english / multilingual / typed-decisions , otherwise the router auto-selects by script/language. The same body can also carry task , lang , lang guess , min confidence , max len and head max len — the controls Router.predict takes that a JSON body can state — while the five hook arguments hooks , on predict start , on predict end , hooks raise , hooks timeout are refused with 422, because a hook is a callable that has to live where the server runs. When publishing the server under a reverse-proxy prefix such as /laya , set LAYA ROOT PATH=/laya . The proxy should strip that prefix before forwarding; the setting controls FastAPI-generated URLs and leaves the internal routes unchanged. Three things differ from Jev when you port a client: - Options per question. A question's options share the checkpoint's option budget, head max len 192 tokens on laya , 256 on the other two , not Jev's cap of 255 options. In addition, the HTTP server laya.serve enforces an amplification guard of at most 100 choice options per question MAX CHOICE OPTIONS = 100 , rejected with 413 before inference . Once options overflow the token budget, around 20 options with a short description each, every option is trimmed to fit, so long or similar labels can reach the model reading the same Where Jev leads where-jev-leads . Once they no longer fit the window at all, the library rejects the request with 422. For more candidates, narrow them first with predict shortlist Honest limits honest-limits . - Score levels. Every level needs a description. A null level is rejected with 422 rather than scored and echoed back in legend . - confidence on choice and score answers is 1 minus normalised entropy, a measure of how concentrated the distribution is, not Jev's n·p max − 1 / n − 1 . A threshold carried over from Jev does not transfer. For one calibrated number on every question type, gate on answer confidence , the probability of the reported answer. Both values shift with the number of options, so fit and validate any cutoff on held-out data at the option counts your workload uses 394 https://github.com/NandhaKishorM/laya/issues/394 . This repo is a flake. On a machine with an NVIDIA GPU: nix run . laya-serve build prebuilt CUDA torch, no compile and serve nix develop dev shell: torch-bin, transformers, fastapi, pytest For a NixOS host, import the module and enable the service: flake inputs: laya.url = "github:<you /laya"; or path:/… on the same host { imports = laya.nixosModules.default ; services.laya-serve = { enable = true; host = "0.0.0.0"; or bind to the Tailscale/LAN address openFirewall = true; device = "cuda"; models = "english" "multilingual" "typed-decisions" ; apiKeyFile = config.age.secrets.laya-api-key.path; optional bearer auth }; } The module runs a hardened DynamicUser systemd unit with CUDA device access, caches weights under /var/lib/laya-serve , and reads the bearer token if any via LoadCredential so it never enters the store. If you only need a single checkpoint for a dedicated pipeline, you can load models directly: python import laya 1. Load a specific checkpoint directly from the hub agent = laya.load "convaiinnovations/laya" English root agent ml = laya.load "convaiinnovations/laya", subfolder="multilingual" 100+ languages agent td = laya.load "convaiinnovations/laya", subfolder="typed-decisions" 2. Run all questions in ONE single forward pass ~35 ms on GPU result = agent.predict state, questions answers = result "answers" print "Department :", answers "department" "choice" - billing confidence: 0.94 print "Urgency :", answers "urgency" "score" - 1.84 / 2.0 print "Churn Risk :", answers "churn risk" "noul" - 0.892 89.2% probability Passing an empty question dictionary to agent.predict state, {} or agent.system one state, {} returns the standard response with "answers": {} and "usage": {"input tokens": 0, "output tokens": 0} . The state is not tokenized and no model forward pass runs. predict handles one state per call, which leaves most of the GPU's batch dimension idle. When you have a list of items to score against the same questions — a backlog of tickets, a table of rows, a log slice — predict batch packs them into shared forward passes: states = {"body": t} for t in ticket texts a list of states results = agent.predict batch states, questions one forward pass for the whole list results i corresponds to states i , with the same output shape as predict Bound peak memory when the list or the texts are large — chunk into passes of N: results = agent.predict batch states, questions, batch size=64 Reduce padding when input lengths vary; results still follow the original state order: results = agent.predict batch states, questions, batch size=64, sort by length=True sort by length=True groups states by their longest encoded question row, after truncation. It looks ahead at most eight batches and reuses the encoded rows for sorting. This uses more temporary CPU memory for tokenized inputs, and takes effect only when 1 < batch size < len states . Benchmark it on your workload and backend: uniform lengths offer little benefit, and changed batch shapes can cause small floating-point differences, including near decision thresholds. Hooks still see states and final results in input order. The option is available on Agent and ONNXAgent . Results are aligned with states by index and identical in shape to predict . Changing batch shapes can introduce floating-point differences on CPU and GPU; check decision thresholds on your workload, particularly with mixed precision. Batching is a GPU throughput win — on an RTX 5060 Ti, per-decision latency drops from ~10 ms one-by-one to ~1 ms batched measured ~9–10× . On CPU, increasing batch size alone may not speed up inference; length grouping can help by reducing the padded work in a mixed-length workload. See the CPU measurements and reproduction commands https://github.com/NandhaKishorM/laya/blob/main/research/README.md length-batching . ONNXAgent.predict batch states, questions, batch size=..., sort by length=... has the same contract, backed by one ONNX Runtime session run per chunk, so an ONNX deployment gets the same batch API, the same result shape, and the same length grouping. Measured on the English checkpoint flat fp32 export, Apple M1, 80 support tickets alternating short and ~8x-longer documents, batch size=4 , best of 3 : wall clock 53.7 s unsorted → 36.3 s sorted ~1.48x , with 0/80 decision changes and max probability drift 0.0 across queue/noul/score. predict / system one truncate a state that exceeds max len to a single window the first, or for a conversation list the last , silently dropping the rest. predict long scans the whole state in overlapping windows, scores them in shared forward passes via predict batch , and aggregates per question: result = agent.predict long state, questions windows the state, one result back result = agent.predict long state, questions, window=256 smaller window isolates a localized span result = agent.predict long state, questions, hooks= AuditLog the scan, instrumented - noul takes the strongest window the statement holds if any window supports it . - choice / score take the most-confident window — averaging over a long, mostly-neutral document lets the neutral majority out-vote the one window that saw the deciding span. - A state that already fits one window is passed straight to system one identical output, plus usage "windows" = 1 . The key is total: 1 single window, N scanned windows, 0 a hook answered the document before the model read any of it. - Hooks wrap the inference that answers the state, so on a scanned document on predict start fires once with ctx.states holding the decoded windows, not the state you passed in it was tokenized to produce them . A start hook may replace that list: the answers are aggregated over whatever reached inference, and usage "windows" counts those states. What a rewritten scan costs is the attribution — answer "window" names a span of your document, so it is only reported when the scan reached inference unchanged. A hook that means to answer the document calls ctx.skip ... instead: its result comes back with no answer "window" and usage "windows" at 0, because no window scored it. A smaller window isolates a short deciding span better it becomes a larger fraction of its window ; the default max len - head max len favors context and throughput. Output shape matches predict , with usage "windows" added. ONNXAgent.predict long state, questions, window=..., stride=..., batch size=... has the same contract and the same aggregation rules, with the windows scored through ONNXAgent.predict batch — one ONNX Runtime session run for all of them, or one per chunk when batch size bounds memory. The returned probability is the deciding window's, not a calibrated number for the whole document — a noul max drifts up with the window count even with no signal, and choice can land on a confidently-neutral window when nothing is decisive. Each answer carries answer "window" the deciding window's index , token start / token end , and count so you can check the span the answer actually came from: r = agent.predict long state, questions r "answers" "refund" "window" {'index': 13, 'token start': 4680, 'token end': 5432, 'count': 14} The same scan is reachable from the Router, which routes first and then windows the checkpoint it picked — the same model= / task= / lang= hints, hooks and routing key as predict : result = router.predict long state, questions, model="multilingual" pip install laya fast adds an optional forward built from fused TileLang https://github.com/tile-ai/tilelang kernels: GEMM + bias/activation epilogues, GEMM + GEGLU, residual + LayerNorm, in-place RoPE, and a sliding-window flash attention that reads the packed QKV buffer directly. Weights stay resident in bf16 and every batch, length bucket is captured as a CUDA graph, so a one-question call no longer pays ~200 kernel launches from Python. agent = laya.load "convaiinnovations/laya", fast=True or: agent.accelerate agent.predict state, questions same API, same answers Numerics: on a fixed set of 60 states the fast path stays within 0.046 of an fp32 forward and within 0.076 of the stock bf16 path max |Δp| ≤ 0.05 vs fp32 on both checkpoints, argmax agreement ≥ 47/48 per question type; every per-option probability is in benchmarks/results/parity .json — see benchmarks/parity fast.py and BENCHMARKS.md https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md gpu-fast-path . The fast path runs in the agent's autocast dtype at the time accelerate is called: bf16 by default, fp16 if agent.dtype is torch.float16 , where it stays within 0.009 of fp32 on the same set BENCHMARKS.md https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md fp16 . Falls back to the stock forward on CPU/MPS or when tilelang is not installed; agent.deaccelerate restores it. Kernels compile once per shape bucket on first use a few seconds, cached on disk . compile=True compiles on the first request that needs a graph, and the first single-question request needs a second one torch specialises a batch of 1 . Both stall a live request. agent.warmup runs the forward on a few synthetic shapes now and returns the seconds it took, so the compiles happen before traffic arrives: agent = laya.load "convaiinnovations/laya", compile=True agent.warmup ~46 s on an RTX 4070 Ti SUPER; every later request ~10-30 ms Measured with benchmarks/bench compile.py --device cuda --warmup English checkpoint, torch 2.11, ten requests of changing shape : without it the first request took 51 s and the first single-question request, the eighth, took another 41 s; after warmup no request took more than 30 ms. It works the same with fast=True kernels and CUDA graphs for those buckets and costs a few forward passes on the stock path. Inductor caches compiled graphs under TORCHINDUCTOR CACHE DIR by default in /tmp ; point it at a persistent directory to keep them across restarts. Because Laya's probabilities are trained with strictly proper scoring rules RLCD , confidence scores are statistically meaningful: dept = answers "department" "choice" conf = answers "department" "confidence" if conf = THRESHOLD: refit and validate THRESHOLD on your own held-out data route automatically dept above it: act, and sample the decisions you act on else: escalate to human agent dept, reason=f"Low confidence {conf:.2f} " A threshold is a policy you choose from measured accuracy at that coverage on your data, not a property of the model. Both checkpoints are over-confident as shipped and laya-multilingual has no fitted temperatures at all, so fit them before relying on these numbers — see Calibration calibration above, and the fine-tuning notebook https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya finetune typed decisions 2xT4 kaggle.ipynb for the fitting loop itself. Then pick the point where the errors you accept are ones you can live with. Confidence orders decisions; it does not establish that a decision is correct. A threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's amp dtype , which is bf16 for all three shipped checkpoints. On the fixed set from benchmarks/parity fast.py 60 states, 288 questions per checkpoint, RTX 2000 Ada bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. LAYA CUDA AMP=fp16 selects fp16 and LAYA CUDA AMP=bf16 selects bf16 LAYA CPU AMP=bf16 is the CPU counterpart . MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches mps amp min rows rows -- 5 by default, LAYA MPS AMP MIN ROWS to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, agent.dtype is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; agent.dtype for rows returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with. predict , predict batch , system one and decide — on Agent , Router and ONNXAgent — take an opt-in min confidence , off by default. It is a caller-side policy on top of the emitted confidence: every answer whose answer confidence falls below the threshold is flagged low confidence: True , with the raw answer, probabilities and confidence left intact for inspection. res = agent.predict state, questions, min confidence=0.85 ans = res "answers" "department" if ans.get "low confidence" : answer confidence < 0.85 escalate to human agent ans "choice" , reason=f"Low confidence {ans 'answer confidence' :.2f} " else: route automatically ans "choice" The threshold reads answer confidence max p — the calibrated quantity, invariant to the number of options — never the entropy confidence . With decide ..., min confidence=... a low-confidence field comes back as None in the schema output, while return details=True keeps the answer and its confidence. LangChain LayaRouter https://github.com/NandhaKishorM/laya/blob/main/docs/langchain.md 's confidence threshold reads the same value: answer confidence when the answer carries it, confidence otherwise. Left unset, min confidence changes nothing. Hooks observe or shape every decision without forking: audit logging, PII redaction before inference, caching, metrics, confidence gating, routing overrides, and forwarding to an external service. They are opt-in, and unset hooks are a no-op. python import laya def log ctx : print ctx.model, ctx.results 0 "answers" , ctx.elapsed ms agent = laya.load "convaiinnovations/laya", on predict end=log agent.system one "I was charged twice.", {"urgent": {"type": "noul", "instructions": "Urgent?"}} A hook is a plain callable, or an object implementing any of on predict start , on predict end , on route , on load , on evict , on error . A start hook can rewrite the state/questions or ctx.skip ... a cached answer; an end hook can rewrite the result. See docs/hooks/ https://github.com/NandhaKishorM/laya/blob/main/docs/hooks/index.md and examples/hooks/ https://github.com/NandhaKishorM/laya/blob/main/examples/hooks . Describe the shape you want with a JSON schema or a pydantic model, and Laya answers it in one forward pass, with typed values and calibrated confidence. python import laya schema = { "type": "object", "properties": { "department": {"type": "string", "enum": "billing", "support", "sales" , "description": "Which team should handle this?"}, "urgency": {"type": "integer", "minimum": 0, "maximum": 2}, "needs human": {"type": "boolean"}, }, } agent = laya.load "convaiinnovations/laya" agent.decide "I was charged twice, refund me.", schema=schema {"department": "billing", "urgency": 2, "needs human": True} decide also works on a Router , accepts a pydantic model install laya structured , and can return per-field confidence with return details=True . In an LCEL chain or LangGraph node the same call is LayaDecision . See docs/structured.md https://github.com/NandhaKishorM/laya/blob/main/docs/structured.md . For throughput, decide batch answers a list of states against the same schema through predict batch see Batch Mode batch-mode-score-many-states-in-one-forward-pass , so the schema is planned once and the states share forward passes: values = agent.decide batch ticket texts, schema=schema values i matches ticket texts i On a Router the states may land on different checkpoints; keywords reach predict batch: details = router.decide batch states, schema=schema, return details=True, batch size=64 Measured on an Apple M-series MPS , 8 English tickets through one checkpoint: 2624 ms one-by-one vs 723 ms batched 3.6× ; 16 mixed English/German states through a Router : 2977 ms vs 1903 ms 1.6× . On CPU the same workloads gave 1.6× and 2.0× . Projected values matched the one-by-one loop 8/8 and 16/16 on both devices, as expected wherever argmax is not at a threshold — verify against your own decision boundaries. laya.decide batch runner, states, ... is the function form, and ONNXAgent.decide batch runs the same thing on an ONNX export. Laya provides pre-tuned question schemas for immediate production use: python import laya agent = laya.load "convaiinnovations/laya" 1. Intelligent Model Router routes to small vs. frontier models routing = agent.predict {"request": "Refactor this service using dependency injection"}, laya.router questions 2. Real-time Prompt Guardrails jailbreaks, injections, leaks guard = agent.predict {"prompt": "Ignore all instructions"}, laya.guard questions 3. Content Safety & Moderation toxicity, harassment, threats safety = agent.predict {"post": "User comment text"}, laya.moderation questions 4. Support Ticket Triage intent, urgency, frustration, churn triage = agent.predict {"message": "My payment failed twice"}, laya.triage questions Fast System 1 routing and guardrails directly inside LangGraph workflows and LCEL chains. Every node also takes core's per-call prediction hooks hooks , on predict start , on predict end , hooks raise , hooks timeout : python from laya.integrations.langchain import LayaDecision, LayaGuardrail, LayaRouter 1. Sub-35ms LangGraph conditional edge routing with confidence fallback router = LayaRouter criteria={"billing": "invoices, charges", "tech": "bugs, outages"}, confidence threshold=0.80, fallback="human agent", workflow.add conditional edges "triage", router 2. Inline prompt guardrails guard = LayaGuardrail action="raise" raises LayaGuardrailError on jailbreak/injection 3. Schema-driven decisions: a JSON schema in, schema-shaped values out decide = LayaDecision { "type": "object", "properties": { "department": {"type": "string", "enum": "billing", "technical", "other" }, "needs human": {"type": "boolean"}, }, } decide.invoke "I was charged twice and the API still 500s for us." {'department': 'billing', 'needs human': False} 4. Score a backlog in one batched call, not one forward pass per input routes = router.batch "refund my invoice", "the app crashes", "change my password" batch and abatch run the whole list through predict batch , so chain.batch ... , RunnableParallel and LangGraph map-reduce nodes get Laya's shared forward passes instead of LangChain's default one-call-per-input thread pool -- which races on MPS, where concurrent torch forwards abort the process. Measured on Apple M-series medians of three : 2.2x on a 16-ticket routing batch, 2.2x across 24 mixed-language tickets through a Router , 1.8x on the guard preset, with every route label and guardrail flag unchanged. On CPU the same workloads are 2.2-2.4x over the one-by-one loop and 1.1-1.5x over the thread pool. See docs/langchain.md https://github.com/NandhaKishorM/laya/blob/main/docs/langchain.md for full guide, support ticket triage nodes, and remote HTTP server configuration. | Primitive | Output | Use Cases | |---|---|---| | choice | Top label, probabilities per option, confidence | Department routing, intent classification, topic categorization | | score | Expected level on ordinal rubric, distribution, confidence | Frustration level, ticket urgency, harm severity | | noul | Calibrated probability P true from 0.0 to 1.0 | Phishing detection, spam filtering, jailbreak detection, churn risk | noul always scores two semantic slots in false, true order and returns the probability of the second slot. For compatibility, those slots are shown to the model as false and true by default. The optional labels mapping overrides only that model-facing text without changing the returned meaning: question = { "type": "noul", "instructions": "Is this review positive?", "criteria": { "false": "the review is negative", "true": "the review is positive", }, "labels": { "false": "B", "true": "A", }, } The labels mapping is optional. It must contain exactly the string keys false and true , whose values must be distinct non-empty strings. Mapping order does not matter, and the returned noul value is still P true . Label sensitivity varies by checkpoint and state, so validate any override on your own data rather than treating A / B as a universal fix. Every question type also takes an optional option order : a permutation of the option indices saying which option goes in which slot. Slot s shows option option order s . Probabilities always come back keyed in your own option order, whatever order the model saw them in, so the key is presentation only — it never changes what a returned label means. question = { "type": "choice", "instructions": "Which team should handle this?", "criteria": {"billing": "...", "technical": "...", "sales": "..."}, "option order": 2, 0, 1 , slot 0 shows sales , slot 1 billing , slot 2 technical } It exists because where an option sits changes the answer. research/eval/presentation checks.py measures the English checkpoint on five options whose text is identical : the per-slot logits centre at +1.68, +1.47, +0.11, -1.35, -1.91 , a spread decided by position alone, and the multilingual checkpoint leans the other way on score see 131 https://github.com/NandhaKishorM/laya/issues/131 . The checkpoint-side fix is a position-balanced retrain; until then option order lets a caller average the effect out, by asking the same question under orders in which every option occupies every slot equally often: k = len question "criteria" the k rotations go in as k questions, so they share one forward pass rotations = { "q%d" % r: dict question, option order= i + r % k for i in range k for r in range k } answers = agent.predict state, rotations "answers" totals = {label: 0.0 for label in question "criteria" } for answer in answers.values : for label, value in answer "probabilities" .items : totals label += value / k decision = max totals, key=totals.get predict already scores every question in a request in one parallel forward pass, so this costs k rows rather than k forward passes — sending the rotations as separate predict calls would cost k of them. On 62 banking intents shortlisted to 20 over 49 states, averaging this way took the share of answers that change with presentation order from 16.3% to 6.1%. Omitting option order keeps the canonical order and the behaviour every existing caller already has. Laya can be exposed as an MCP https://modelcontextprotocol.io stdio server, so any MCP client OpenClaw, Claude Desktop, Cursor, ... can call typed decisions as tools laya predict , laya predict batch , laya route , laya route batch , laya decide , laya shortlist , laya preset , laya status without writing glue code. This is an optional extra : the core package has no mcp dependency. pip install "laya mcp " laya-mcp-server or: python -m laya.mcp.server Example MCP client configuration stdio transport : { "mcpServers": { "laya": { "command": "laya-mcp-server", "env": { "LAYA DEVICE": "cpu" } } } } The environment variables follow the contract documented at the top of laya/serve.py https://github.com/NandhaKishorM/laya/blob/main/laya/serve.py , so the same variable has one meaning across the package: | Variable | Default | Meaning | |---|---|---| | LAYA DEVICE | auto | Same as laya.serve : the device type is lower-cased CUDA → cuda , CUDA:0 → cuda:0 and passed to torch, whose device parser is case-sensitive | | LAYA PRELOAD | 1 | Same as laya.serve : build the checkpoints at startup, not lazily | | LAYA MODELS | english,multilingual | Comma list to preload serve contract . MCP difference: an empty value preloads english,multilingual so typed-decisions stays lazy; in laya.serve empty means every checkpoint | | LAYA THREADS | torch default | Same as laya.serve : cap torch intra-op threads for CPU inference; keep it at or below the physical core count | | LAYA AUTO TASK | 0 | Same as laya.serve : 1 lets a request whose question ids match a typed-decisions workflow route to that checkpoint, which is then loaded on demand; it never joins the preload list | The tools return structured JSON answers with probabilities, routing metadata, device, latency ms . laya predict batch and laya route batch are the MCP form of Router.predict batch / route batch batch-mode-score-many-states-in-one-forward-pass : one tool call takes an array of {state, questions, model?, task?, lang?, lang guess?, max len?, head max len?} requests, routes them first, groups them by checkpoint, and shares forward passes between requests with the same question schema, returning the answers in input order. max len / head max len are per request there, exactly as Router.predict batch reads them, so one wide question can raise its own budget without shrinking the rest of the batch to it. On 16 mixed-language tickets through the tool functions themselves, one batch call beat 16 laya predict calls by 2.1-2.3x on MPS 983-1082 ms - 467-477 ms and ~1.25x on CPU 1861-2471 ms - 1470-1911 ms , with 0/16 decision flips choice label, rounded score, noul sign against the loop. Prefer it whenever a client has more than a few requests: each saved round trip is also an MCP request/response. laya decide is the MCP form of laya.decide schema-driven-decisions : it takes a JSON schema enum choices, booleans, bounded integers instead of hand-written questions and returns the decided values projected onto that schema -- enum member, integer level, boolean -- beside per-field confidence and probabilities , so a client that already knows the answer shape never parses an answer map by hand. laya shortlist is the MCP form of predict shortlist honest-limits : it shortlists a many-option choice question to its k most likely labels by embedding similarity mean-pooled from the answering checkpoint's own encoder, so no extra model is downloaded , answers in one forward pass, and returns per-question shortlist metadata kept labels, cosine scores, k , option count . The guardrails shown on every decision tool point clients to laya shortlist for 20-option choices. As with the SDK, use it for structured decisions only; not for open Q&A or text generation. Tests: tests/test mcp.py CI, no weights and tests/test mcp local e2e.py local, real weights and a real stdio handshake . laya.evals scores a labelled dataset and gates a build on it, so a quality change is a reviewable diff instead of a hand-check. It is pure Python plus numpy, imports no torch, and needs no weights until you point it at a checkpoint. laya-evals validate research/evals/fixture.jsonl laya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language laya-evals run data.jsonl --baseline baseline.json --tolerance choice accuracy=0.02 --json report.json run reports overall and per-slice metrics choice accuracy , noul accuracy , score mae , ece , mean confidence , latency and exits non-zero on a threshold or baseline failure, so it drops into CI unchanged. A weight-free job runs the metric and API tests on every PR, and a scheduled workflow evaluates the English checkpoint against the committed baseline. See docs/evals.md https://github.com/NandhaKishorM/laya/blob/main/docs/evals.md for the dataset format and the gate. Community diagnostics: Chinese workplace decisions Feishu-style https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/feishu zh/README.md · 中文说明 https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/feishu zh/README.zh-CN.md . Includes frozen synthetic cases, archived paired Laya/Jev responses, and an offline audit; separate from the benchmark suites below. Also Chinese short-command routing https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/zh short commands/README.md · 中文说明 https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/zh short commands/README.zh-CN.md : 18 frozen commands and a seven-rung ablation of the documented prompt guidance, which locates the accuracy loss on the four-question path rather than the six-option one. Full report: BENCHMARKS.md https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md — every run consolidated, languages and themes, with per-language detail for all 51 languages. All Laya numbers below are measured. Every model answered byte-identical questions fixed seed in the same run. Reproduce with research/scripts/laya benchmark colab.ipynb https://github.com/NandhaKishorM/laya/blob/main/research/scripts/laya benchmark colab.ipynb on a T4. | questions per call | laya | laya-multilingual | |---|---|---| | 1 | 39.5 ms | 32.8 ms | | 5 | 84.5 ms | 40.1 ms | | 10 | 158.6 ms 15.9 ms/q | 72.3 ms 7.2 ms/q | | 50 | 771 ms | 337 ms 6.8 ms/q | Batched throughput reaches 103-332 questions/sec on a single T4. For reference, TypeSafe Jev has been independently measured at 236-276 ms p50 AbdelStark https://github.com/AbdelStark/jev-benchmarks , nibzard https://github.com/nibzard/decision-model-benchmark -- Laya answers a single question roughly 6-7x faster . Every Laya figure is what Router .predict ... actually returns — the checkpoint the router selects for that input, not a hand-picked best of three. Jev figures are third-party published, never measured here no TypeSafe API access , so sample sizes and prompts differ. The Laya column here comes from the Applications run research/scripts/bench apps.py , N=400 per task; research/results/app benchmark results.json , so AG News / DAIR Emotion read 0.950 / 0.595 here versus 0.947 / 0.573 in the committed T4 English suites N=600 below. | | Jev 1.13.0 | Laya routed | | |---|---|---|---| | typed-decisions, 2,000 decisions | 0.727 | 0.766 | +0.039 | | AG News, 4 labels | 0.910 | 0.950 | +0.040 | | DAIR Emotion, 6 labels | 0.480 | 0.595 | +0.115 | | Banking77 72 vs 77 labels | 0.870 | 0.425 | Jev leads on 20 options | | ECE lower better | 0.246 | 0.081 | 3× better post-temperature | | p50 latency, 1 question | 236–276 ms | 32.8 ms | 7.8× faster | | Languages usable | no published benchmark | 48 of 51 | — | | Weights | closed API | Apache 2.0 | — | | Cost | $0.042 / 1M tokens | $0 self-hosted | — | On DAIR Emotion, Jev assigned zero probability to the true label on 16% of examples — a hard failure for anything branching on confidence. - High-cardinality label spaces 20 options at default settings : On Banking77, Jev scores 0.870 on 72 labels while Laya scores 0.425 on 77 labels at default 256-token head budget . This is an architectural token-budget constraint: options share a fixed head max len budget 192 tokens on English, 256 on multilingual , so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. While laya-multilingual supports 1,024 context and up to 8,192 in the encoder and you can raise agent.cfg "head max len" = 512 at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning. predict shortlist see Honest limits honest-limits keeps the top k labels with a caller-supplied embedding, then runs one forward pass on that shortlist. - Soft distribution matching: On typed-decisions, while Laya achieves higher argmax accuracy 0.766 vs 0.727 , Jev achieves higher soft accuracy 0.580 vs 0.471 against the teacher's full probability distributions. - Out-of-the-box raw calibration: Before temperature scaling, the base checkpoint has higher raw ECE 0.213 vs 0.144 . Laya achieves its 0.081 ECE after domain temperature fitting. Full detail, including every workflow and all 51 languages: BENCHMARKS.md https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md . 400 cases, 2,000 decisions, four workflows. | model | accuracy | soft acc | Brier | ECE | score MAE | |---|---|---|---|---|---| | laya-typed-decisions | 0.766 | 0.471 | 0.062 | 0.213 | 0.242 | | laya | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 | | laya-multilingual | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 | | Jev 1.13.0 published | 0.727 | 0.580 | 0.148 | 0.144 | 0.391 | | teacher self-agreement ceiling | 0.735 | | | | | | per-question majority class | 0.461 | | | | | | random guess | 0.318 | | | | | The fine-tuned checkpoint beats Jev by 3.9 points and clears the teacher ceiling, with 2.4x better Brier and 1.6x better score MAE. It wins on all four workflows: invoice processing 0.804, security incidents 0.766, customer service 0.764, agent-trace observability 0.730. By primitive: noul 0.857, choice 0.733, score 0.723. Two places it still trails Jev: soft accuracy 0.471 vs 0.580 — its argmax is better but its distributions match the teacher less well and ECE 0.213 vs 0.144 , which temperature fitting addresses. The base checkpoints sit below the majority-class baseline 0.362 and 0.352 against 0.461 . All of the capability on this benchmark comes from fine-tuning. Base rows match the committed runs suites.typed decisions in research/results/t4 colab benchmark.json : 0.3620 / 0.3515; CPU sweep part b : 0.3615 for English . The laya-typed-decisions row and its four per-workflow scores are from the fine-tuning run and have no committed result file behind them yet. | | laya | laya-multilingual | |---|---|---| | English | 0.783 | 0.657 | | 13 other languages | 0.306 | 0.451 | | XNLI, English | 0.860 | 0.843 | | XNLI, 14 other languages | 0.521 | 0.731 | Across all 51 languages the English checkpoint macro-averages 0.227 with macro ECE 0.733 0.571 as served, after the temperature clamp , and only 23 of 51 languages clear 3x random. laya-multilingual reaches 0.401 on the same 5,100 cases, clearing 3x random in 48 of 51 — see BENCHMARKS.md https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md for why those multilingual numbers are a re-run rather than the committed file. Khmer scores 0.000 at 95.2% confidence raw, 70.5% as served, on the English checkpoint. This is why Router quickstart-route-mode-recommended exists: the model's own confidence gives no warning, so the routing decision has to be made before the forward pass. | task | laya | laya-multilingual | note | |---|---|---|---| | AG News | 0.947 | 0.937 | in training mix | | BoolQ | 0.830 | 0.787 | in training mix | | DAIR Emotion | 0.573 | 0.513 | held out | | prompt-injections | 0.698 | 0.578 | held out, n=116 | | SST-5 ordinal | 0.372 | 0.282 | held out | Both checkpoints are over-confident as shipped. Refitting one temperature per question type, option count on held-out data moves mean ECE 0.466 - 0.081 laya and 0.314 - 0.106 laya-multilingual . laya-multilingual ships with no fitted temperatures at all, so fit them before relying on its probabilities. At checkpoint load, numeric temperature entries are clamped to 0.5, 5.0 ; invalid or non-finite entries use the neutral fallback 1.0 . A runtime warning reports the affected entries and applied values. Bucket-specific temperatures still take precedence over per-type values, including when a bucket uses the fallback. Raw values remain available in agent.temperature raw and agent.temperature by options raw . A fallback prevents a loading failure; it does not establish calibrated confidence. - The base checkpoints are near chance on typed-decisions zero-shot -- 0.362 and 0.352 against a 0.318 random baseline and a 0.461 majority-class baseline. The 0.766 figure comes from the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base to specialise, not a zero-shot decision engine. - Avoid boolean-word labels in choice questions. Choice keys are rendered verbatim, and the current checkpoints can follow labels such as true / false or yes / no instead of the option descriptions. Use semantic labels or opaque labels such as A / B , and validate them on the checkpoint and states you serve. - Semantic choice labels do not make negation safe. In the five cancellation examples from 377 https://github.com/NandhaKishorM/laya/issues/377 , a CPU run on Laya 0.3.20 with no action / cancel account keys selected cancel account for all four negated requests on laya and two on laya-multilingual ; one multilingual answer assigned it probability 0.9998 . The positive control passed on both checkpoints. These are narrow cancellation examples, not evidence that every negated state fails. Validate the exact checkpoint and wording you serve; using semantic keys alone does not avoid this failure. - High-cardinality choice questions and token budgets: Sequences split into an option prompt budget head max len and the remaining document/state budget. head max len is a cap, not the head length. The renderer fills the option prompt budget only as far as the question and its option descriptions need, and build sequence then sizes the state from the head it actually built room = max len - head len - 1 . The real room therefore depends on the question, not on the cap: - laya English defaults to 512 context head max len = 192 . The shipped department question below renders a 48-token head, leaving 463 tokens for state , not 320. - laya-multilingual and laya-typed-decisions default to 1,024 context head max len = 256 . The shipped question renders a 45-token head, leaving 978 tokens for state , not 768 mmBERT-base encoder supports up to 8,192 with RoPE . max len - head max len is not a safe figure in either direction , so size a state to max len - head len - 1 for the question you are actually asking: - For a question that does not fill the cap it understates the room, as above — conservative, but it discards state the model would have read. - For a high-cardinality question it overstates it. per = max 4, head max len - 16 // k floors at 4 tokens per option and head ids keeps a floor of 8, so past roughly head max len / 4 options the head grows beyond the cap. A state sized to the cap-based figure is then truncated , not merely conservative: k optionshead real room cap-based figure laya , max len=512 77 319 192 320 laya , max len=512 100 411 100 320 laya-multilingual , max len=1024 100 411 612 768 The crossover is at k ≈ head max len / 4 and the gap widens from there; MAX CHOICE OPTIONS = 100 in laya/serve.py puts the overstating regime within reach over HTTP. At default settings, a 77-option question like Banking77 allocates only 256 - 16 // 77 ≈ 3–4 tokens per label, which causes accuracy to fall off sharply 0.425 vs Jev's 0.870 . If evaluating 50+ options in a single question: 1. Raise agent.cfg "head max len" = 512 and agent.cfg "max len" = 1024 or up to 2048 / 4096 / 8192 so every option has enough tokens to remain distinct. Both are also per-request: predict state, questions, head max len=512, max len=1024 widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments LangChain guide https://github.com/NandhaKishorM/laya/blob/main/docs/langchain.md . laya --questions takes the same two budgets as --max-len / --head-max-len . 2. Or shortlist with embeddings and run one forward pass on the top k labels predict shortlist , example below . predict and system one still score every criterion they are given. 3. Or split the label set yourself into a coarse question and a fine question. - python import laya questions = { "intent": { "type": "choice", "instructions": "Which banking intent is this?", "criteria": { "card arrival": "where is my card", "transfer fee": "fee charged on a transfer", ...the rest of a large label set }, } } result = laya.predict shortlist agent, {"text": "I was charged twice for a transfer"}, questions, embed fn=laya.embed fn from agent agent , or any callable: texts - n, dim k=20, result "shortlist" "intent" "labels" the top 20 labels sent to the model embed fn texts returns one vector per string. embed fn from agent mean-pools the encoder already loaded on the agent; the decision head runs in the following predict / system one call. Probabilities on a shortlisted choice are over those k labels. When k is at least the number of labels, the original question is passed through and embed fn is not called. Shortlisting the same option set on every request re-embeds option texts that do not change. Wrap the embedder once with laya.cached embed fn embed fn and repeat calls embed only the new query text: lookups are exact string matches into an LRU of at most 4,096 entries about maxsize dim 4 bytes, so ~12 MB at the default with a 768-dim encoder , and texts missing from the cache are still embedded in one batched call. The wrapper's cache info reports hits and misses; call cache clear if the model behind embed fn changes. Issue 102 https://github.com/NandhaKishorM/laya/issues/102 reports that a top-20 zero-shot shortlist moved a BANKING77 run from 54.3% to 60.8% on the reporter's setup. Those figures are the reporter's; this repository has not remeasured them. - Ordinal score questions are the weakest primitive SST-5 0.372 . - noul can follow its option labels instead of the state, most strongly on laya English . noul renders its two options as false: / true: by default, and on the English checkpoint that label pair can dominate the answer, returning a confident "no" for clearly positive input 156 . Until a retrained checkpoint lands, check noul answers on your own data. You can override the model-facing pair while keeping the noul result as P true : {"type": "noul", "instructions": "Is this review positive?", "criteria": {"true": "the review is positive", "false": "the review is negative"}, "labels": {"true": "A", "false": "B"}} A noul with no criteria renders one generic option pair for every state no, the statement does not hold / yes, the statement holds . On laya that pair carries the whole decision, so it answers "no" whatever the state — give a noul criteria if you need it to discriminate. On laya-multilingual the criteria-less form does read the state.Label sensitivity varies by checkpoint and state, so validate the override on your own data. A two-option choice with neutral keys remains another workaround: {"type": "choice", "instructions": "Is this review positive?", "criteria": {"A": "yes, the review is positive", "B": "no, the review is negative"}} criteria on a noul must be keyed true / false — those two keys are the option text the model reads, so any other key is rejected instead of being quietly replaced with the defaults. Before that check, criteria: {"yes": ..., "no": ...} was accepted, dropped, and answered against false: / true: anyway, which cost 2 of 3 clearly positive reviews on the English checkpoint 156 https://github.com/NandhaKishorM/laya/issues/156 . Use labels as above to change the wording without touching the option text. - laya-multilingual has a position bias on score questions 131 : it rarely picks the first-listed level, in any language. For English score questions, route to model="english" , and for other languages validate score outputs on your own data before relying on them. - action.act probability carries no usable signal yet 185 . It reads 1.0 for almost every input, and its raw logits run against correctness AUROC 0.30 on 396 labelled decisions . Gate on confidence instead, which reaches an AUROC of 0.77 on the same items. - laya collapses outside English; laya-multilingual is weaker on English. Route, or pick deliberately. - omp-laya-judge https://github.com/F0Rextasy/omp-laya-judge : an oh-my-pi https://github.com/can1357/oh-my-pi plugin with a local System-1 judge MCP server and skill choice / bool / score , 0 tokens, about 0.3 s on CPU , confidence-gated escalation, and reproducible quiz and Snake demos. - laya-adk-toolkit https://github.com/Ashfaqbs/laya-adk-toolkit : Google ADK https://google.github.io/adk-docs/ tools that let an agent call Laya's classify / score / detect typed decisions directly as tools, instead of asking an LLM to guess at structured output. - laya-Ascend https://github.com/zzhdbw/laya-Ascend : Laya on Huawei Ascend NPUs through torch-npu , with a CPU vs NPU benchmark 34x to 71x faster at batch size 1 , a setup guide, and Snake and Tetris demos. - laya-apple https://github.com/tc3oliver/laya-apple : a correctness-validated Laya runtime for Apple silicon that uses the MLX GPU and the Apple Neural Engine, with automatic routing and concurrent heterogeneous serving. - stuntd https://github.com/bladedevoff/stuntd : runs Laya locally behind the Jev API POST /v1/systemone , no key and trains a head per decision on the frozen encoder from your own labelled rows, with a calibrated confidence threshold a 12-label intent task: 89.5% zero-shot to 100% trained . - Hugging Face Model: convaiinnovations/laya https://huggingface.co/convaiinnovations/laya - Interactive Web Demo: convaiinnovations/laya-demo https://huggingface.co/spaces/convaiinnovations/laya-demo - Engineering Writeup: Read the full story on Dev.to https://dev.to/nandakishor m 6cc0adfde9f/i-built-non-autoregressive-decision-models-a-year-ago-then-a-frontier-lab-called-it-a-18me Fine-tune Laya on your own domain data. The notebook runs on Kaggle's free 2xT4 GPUs and does the whole loop: build the dataset, train with RLCD proper-scoring-rule rewards, GRPO-style policy gradient , fit calibration temperatures, evaluate, and push the result to the Hub. The notebook enables gradient checkpointing on both the encoder and the decision head. For custom training loops, model.head checkpointing = True enables activation checkpointing for the decision-head layers; enable the encoder's gradient checkpointing separately. During gradient-enabled training, this reduces stored intermediate activations by recomputing them during backward, trading extra computation for lower activation memory. The head flag defaults to False and is bypassed in evaluation and under torch.no grad . The notebook fits one temperature per type choice , score , noul and removes inherited temperature by options from the exported config. Otherwise those old bucket values take precedence at inference and silently mask the new fit. Existing checkpoints still honor intentional bucket-specific temperatures, falling back to the corresponding per-type value when a bucket is absent; the runtime's temperature clamp is unchanged. This fixes configuration persistence, not measured model accuracy or calibration quality. The notebook's calibration samples come from its training items; evaluate on separate held-out data before claiming an improvement. Already published checkpoints are not rewritten. Run the CPU-only regression checks with python tests/test calibration persistence.py synthetic configs and tiny local fixtures; no pretrained downloads or training . Fine-tuning is where most of the value is. On the typed-decisions benchmark the base checkpoints score near chance zero-shot 0.36 and 0.35 against a 0.318 random baseline , while the fine-tuned checkpoint reaches 0.766 on the same 2,000 decisions -- above TypeSafe Jev's published 0.727 and above the 0.735 teacher self-agreement ceiling. Treat Laya as a fast base to specialise, not as a zero-shot decision engine. Runtime on 2xT4 is roughly 4-5 hours for 4 epochs over ~30k questions. docs/finetune browser agent.md https://github.com/NandhaKishorM/laya/blob/main/docs/finetune browser agent.md records a complete specialisation on a single 16 GB GPU with no paid API: Laya as the operation/target decider for browser-use/jev-ultrafast https://github.com/browser-use/jev-ultrafast same request format as TypeSafe Jev . Element top-1 among ~45 candidates goes from 0.10 zero-shot to 0.66, real-task success from 0 % to 62 % at 17-23 ms per step; weights, pipeline code and per-run results are on the Hub at cklxx/laya-browser https://huggingface.co/cklxx/laya-browser . The write-up covers the data recipe reverse-generated goals, executed DONE states, Mind2Web, on-policy corrections , the input-format change that mattered most, and the things that did not work. If Laya helps your research or products, consider supporting independent research: Apache 2.0. Developed by Convai Innovations.