{"slug": "laya-multilingual-non-autoregressive-system-1-decision-engine", "title": "Laya: Multilingual, non-autoregressive System 1 decision engine", "summary": "Laya released a multilingual, non-autoregressive \"System 1\" decision engine that returns typed decisions across 100+ languages in a single forward pass in 33 ms, installable via `python -m pip install laya` on Python 3.10 or newer. The library's `laya-multilingual` checkpoint reads up to 8,192 tokens with `max_len=8192`, and a router selects the appropriate checkpoint per request, sending non-English text to the multilingual model. Optional extras include `laya[serve]`, `laya[mcp]`, `laya[langchain]`, `laya[llamaindex]`, `laya[crewai]`, `laya[onnx]`, and `laya[fast]`, with a TypeScript/Node.js/browser port published to npm as `laya-ts`.", "body_md": "**Multilingual, non-autoregressive System 1 decision engine.** Typed decisions over 100+ languages in a single forward pass — 33 ms — trained with reinforcement learning against strictly proper scoring rules (RLCD), with a router that picks the right checkpoint per request.\n\n```\npython -m pip install laya\n```\n\nWith [uv](https://docs.astral.sh/uv/), run `uv add laya` in a uv project or `uv pip install laya` in a virtual environment.\n\nPython 3.10 or newer. Optional extras: `laya[serve]` (HTTP server), `laya[mcp]` (MCP server), `laya[langchain]` (LangChain and LangGraph), `laya[llamaindex]` (LlamaIndex selectors), `laya[crewai]` (CrewAI routing), `laya[onnx]` (ONNX Runtime), `laya[fast]` (TileLang GPU fast path). Step-by-step setup for each platform, CPU-only or GPU PyTorch builds, and troubleshooting are in [Installation details](#installation-details).\n\nFor TypeScript / Node.js / browser, see [`laya-ts/`](https://github.com/NandhaKishorM/laya/blob/main/laya-ts). npm releases (` npm install laya-ts`) are published from this repository's `laya-ts-v*` release tags.\n\n**Long documents.** `laya-multilingual` reads up to 8,192 tokens with `max_len=8192`. Measured accuracy and time by document length, reproducible with [`research/scripts/bench_long_context.py`](https://github.com/NandhaKishorM/laya/blob/main/research/scripts/bench_long_context.py):\n\n**Long documents: `laya-multilingual` reads up to 8,192 tokens.** It ships with a 1,024-token limit that cuts long documents off, so pass `max_len=8192` for them:\n\n```\nresult = router.predict(long_document, questions, model=\"multilingual\", max_len=8192)\n```\n\nIn the table above, 16 to 18 of 20 requests were answered correctly with up to about 4,000 tokens of text before them; beyond that results vary (8 to 17 of 20), so check long-document accuracy on your own data. Short inputs give identical answers with `max_len=8192`, and speed follows the input's real length, not the limit: short inputs are unchanged, and a 4,000-token input takes about 1.7 s on an Apple GPU. Name the checkpoint with `model=\"multilingual\"`, since long mostly-English text would otherwise route to the English checkpoint.\n\n``` python\nfrom laya import Router\n\nrouter = Router()  # downloads a checkpoint on first use; Router(preload=True) loads all three up front\n\nstate = \"Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.\"\nquestions = {\n    \"department\": {\"type\": \"choice\", \"instructions\": \"Which department should handle this?\",\n                   \"criteria\": {\"billing\": \"invoices, payments, refunds\",\n                                \"technical\": \"bugs, outages, system errors\",\n                                \"other\": \"everything else\"}},\n    \"urgency\": {\"type\": \"score\", \"instructions\": \"How urgent is this?\",\n                \"criteria\": [\"not urgent\", \"soon\", \"blocking\"]},\n    \"churn_risk\": {\"type\": \"noul\", \"instructions\": \"Does the user threaten to cancel or leave?\"},\n}\n\nresult = router.predict(state, questions)\nprint(result[\"answers\"][\"department\"][\"choice\"])  # billing\nprint(result[\"answers\"][\"churn_risk\"][\"noul\"])    # probability the answer is yes\nprint(result[\"routing\"][\"model\"])                 # english\n```\n\nThe same call works in any of 100+ languages. The `Router` detects the script and language and sends non-English text to `laya-multilingual`:\n\n```\nfor text in [\"मुझसे मार्च में दो बार शुल्क लिया गया, कृपया डुप्लिकेट राशि वापस करें।\",\n             \"La aplicación se cierra cada vez que abro la configuración.\"]:\n    r = router.predict(text, {\"department\": questions[\"department\"]})\n    print(r[\"routing\"][\"model\"], r[\"answers\"][\"department\"][\"choice\"])\n# multilingual billing\n# multilingual technical\n```\n\nFrom the command line, `laya \"My payment failed twice\" --preset triage` answers a ready-made question set. More in the [full quickstart](#quickstart-route-mode-recommended) and the [docs](https://nandhakishorm.github.io/laya/).\n\nThe shipped checkpoints work zero-shot, but fine-tuning on decisions from your own domain is where accuracy jumps. On the typed-decisions benchmark (2,000 decisions across four workflows), the fine-tuned `laya-typed-decisions` checkpoint scores **0.766** accuracy, against **0.362** for the base English checkpoint on the same decisions.\n\n**[Fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb)**: runs the whole loop on Kaggle's free 2x T4 GPUs (build the dataset, train, fit calibration temperatures, evaluate, and push the result to the Hub). Details in [Fine-Tuning](#fine-tuning).\n\n**[nandhakishorm.github.io/laya](https://nandhakishorm.github.io/laya/)**: guides for [prediction hooks](https://nandhakishorm.github.io/laya/hooks/), [schema-driven decisions](https://nandhakishorm.github.io/laya/structured/), [Docker](https://nandhakishorm.github.io/laya/docker/) and [LangChain and LangGraph](https://nandhakishorm.github.io/laya/langchain/), plus a full [API reference](https://nandhakishorm.github.io/laya/reference/).\n\n- **ONNX catches up with PyTorch.**`ONNXAgent` gains`predict_batch` (with`sort_by_length` ),`predict_long` and`decide_batch` ,`scripts/export_onnx.py --quantize` writes a per-channel INT8 copy for CPU, and`laya-evals run --onnx` scores an export with the same gates as the torch path.\n- **Opt-in abstention.**`min_confidence=` on`predict` ,`predict_batch` ,`decide` and`decide_batch` flags answers below a threshold on`answer_confidence` with`low_confidence: True` , and`decide` returns`None` for them.\n- **Batch everywhere.**`decide_batch` ,`Router.predict_long` ,`laya --batch FILE` , the MCP`laya_predict_batch` /`laya_route_batch` /`laya_decide` tools, and LangChain`batch()` /`abatch()` all run on shared forward passes. New`LayaDecision` (LangChain), LlamaIndex selectors (`laya[llamaindex]` ) and CrewAI routing (`laya[crewai]` ).\n- **Per-request token budget.**`max_len` /`head_max_len` now reach every surface:`laya-serve` (capped by`LAYA_MAX_TOKEN_BUDGET` ),`Router.predict_batch` requests, the CLI (`--questions` ,`--max-len` ,`--head-max-len` ), MCP tools and LangChain nodes.\n- **Operations.**`LAYA_MAX_LOADED` ,`LAYA_REVISION` and per-checkpoint SHA-256 maps;`/health` reports the device a checkpoint really runs on and its CPU-fallback count; the 503 busy answer carries`Retry-After` ;`compile=True` no longer recompiles for every request shape.\n- **Stricter inputs.** A null or duplicate`choice` label, a short temperature list, a`None` state and non-dict questions are refused with a message that names them, and`usage[\"options\"]` says when the head budget left two options with the same tokens.\n\nLaya evaluates typed questions (`choice`, `score`, `noul`) over any state (text, email, ticket or JSON document) in **a single forward pass** — 33 ms for one question, 7.2 ms/question batched, measured on a T4. No text generation, so nothing to parse and nothing to hallucinate.\n\nThree checkpoints, and a `Router` that picks between them per request:\n\n|  | encoder | params | context | use it for | \n|---|---|---|---|---|\n| [`laya`](https://huggingface.co/convaiinnovations/laya) | ModernBERT-large | 421M | 512 | English | \n| [`laya-multilingual`](https://huggingface.co/convaiinnovations/laya-multilingual) | mmBERT-base | 322M | 1024 (up to 8,192) | 100+ languages, 2x faster | \n| [`laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions) | ModernBERT-large | 421M | 1024 | the typed-decisions workflows | \n\nPython 3.10 or newer. The dependencies set that floor: `huggingface_hub` 1.x, `transformers` 5.x and `torch` 2.14 all require 3.10.\n\n**Optional PyTorch build selection:** If you need a CPU-only or GPU-specific PyTorch build, follow [PyTorch's installation guide](https://pytorch.org/get-started/locally/) after creating your virtual environment and before installing Laya. Replace `pip` or `pip3` in the selected command with the environment's Python executable followed by `-m pip`.\n\nIf you already use a virtual environment, install the PyPI release with:\n\n```\npython -m pip install laya\n```\n\nFor a new environment, choose the commands for your platform below. Run them from your project directory; the explicit Python paths keep installation and verification in the same environment.\n\n**macOS / Linux** (with Python 3.10 or newer):\n\nOn Debian/Ubuntu, the system Python may require `sudo apt install python3-venv` before creating a virtual environment. If `venv` reports that `ensurepip` is unavailable, install that package and retry.\n\n```\npython3 -m venv .venv\n.venv/bin/python -m pip install laya\n.venv/bin/python -I -c \"import laya; print(laya.__version__)\"\n```\n\n**Windows PowerShell** (this example uses an installed Python 3.11):\n\n```\npy -3.11 -m venv .venv\n.\\.venv\\Scripts\\python.exe -m pip install laya\n.\\.venv\\Scripts\\python.exe -I -c \"import laya; print(laya.__version__)\"\n```\n\nBoth checks print the installed Laya version without loading a checkpoint. `-I` excludes the current directory from the import search path, so a local source copy cannot mask a missing installation. Keep using the same virtual environment's Python when running your application.\n\n**Intel GPU (XPU)**\n\nInstall a supported Intel GPU driver first. For an XPU-enabled PyTorch build, install its wheel before Laya; the default PyPI wheel may be CPU-only. PyTorch's validated hardware and OS list is in the [Intel GPU guide](https://docs.pytorch.org/docs/2.14/notes/get_start_xpu.html).\n\n```\n.\\.venv\\Scripts\\python.exe -m pip install torch --index-url https://download.pytorch.org/whl/xpu\n.\\.venv\\Scripts\\python.exe -m pip install laya\n.\\.venv\\Scripts\\python.exe -c \"import torch; print(torch.xpu.is_available())\"\n```\n\nFor a source checkout, replace `pip install laya` with `pip install -e .`. Laya automatically selects an available XPU when no device is specified; you can also request one explicitly with `device=\"xpu\"` in `laya.load()` or `Router(device=\"xpu\")`.\n\n**Install from GitHub**\n\nTo use the development version instead of the PyPI release, create the virtual environment above and replace its installation command with the appropriate command below. Git must be installed.\n\n```\n# macOS / Linux\n.venv/bin/python -m pip install \"git+https://github.com/NandhaKishorM/laya.git\"\n# Windows PowerShell\n.\\.venv\\Scripts\\python.exe -m pip install \"git+https://github.com/NandhaKishorM/laya.git\"\n```\n\nRun the same version check afterward. The GitHub version follows the repository's default branch and may differ from the published release.\n\n**Install with uv**\n\n[uv](https://docs.astral.sh/uv/) creates the virtual environment, downloads a matching Python if none is installed, and installs into it. The commands are the same on macOS, Linux and Windows PowerShell:\n\n```\nuv venv --python 3.12\nuv pip install laya\n```\n\n`uv pip install` targets the `.venv` in the current directory without activating it, so run the version check for your platform above afterward. Extras and the GitHub version install the same way: `uv pip install \"laya[serve]\"`, `uv pip install \"git+https://github.com/NandhaKishorM/laya.git\"`. For a CPU-only or GPU-specific PyTorch build, add `--torch-backend=auto` to pick the build that matches the machine's GPU driver, or name one such as `--torch-backend=cpu`; [uv's PyTorch guide](https://docs.astral.sh/uv/guides/integration/pytorch/) has the list.\n\nIf your application is a uv project, add Laya as a dependency instead:\n\n``` python\nuv add laya\nuv run python -I -c \"import laya; print(laya.__version__)\"\n```\n\n`--torch-backend` applies to `uv pip` only; in a uv project, uv's PyTorch guide shows how to set the PyTorch index in `pyproject.toml`.\n\n**Model setup and troubleshooting**\n\nContinue with the [Router quickstart](#quickstart-route-mode-recommended) to run inference. Loading a Hub checkpoint requires access to Hugging Face on its first download; the quickstart's `Router(preload=True)` loads all three configured checkpoints at construction.\n\n- **`ModuleNotFoundError: No module named 'laya'`:** run both installation and your script with the same virtual environment's Python executable shown above. In an editor, select that interpreter as well.\n- **Missing `rl_agent_config.json`:** this file ships with a Laya checkpoint alongside` model.safetensors` ; it is not a configuration file you need to create in the source repository. For a local model, pass the directory containing those checkpoint files.\n\nInstalling the package also installs a `laya` command for quick local testing, no script needed:\n\n```\nlaya \"I was charged twice, please refund\"            # routing decision only; works offline, no download\nlaya \"Refactor this service\" --predict               # full answers (downloads the checkpoint on first use)\nlaya \"Mein Konto wurde zweimal belastet\" --lang de   # force a language instead of detecting it\nlaya \"My payment failed twice\" --model ml            # pin a checkpoint: names, aliases and casing all resolve as the SDK resolves them\nlaya \"My payment failed twice\" --preset triage       # answer a ready-made preset (triage, email, guard, moderation, router)\nlaya --batch tickets.txt --predict                   # score a file of requests, one per line, in one batch\ncat tickets.txt | laya --batch - --predict --json    # stdin; one JSON line of answers per request\nlaya \"Where is my card\" --questions intents.json     # answer your own questions, written in a JSON file\nlaya                                                 # interactive mode\n```\n\nRouting alone never downloads a checkpoint, so it returns in milliseconds. `--predict` loads the routed checkpoint, which needs network access to the Hugging Face hub the first time; if a checkpoint cannot be downloaded, the CLI says so instead of crashing. `--batch` (with or without `--predict`) sends the whole file through `Router.predict_batch` in one process, so the requests share checkpoint loads and forward passes — measured 2.6x on 20 tickets vs looping `predict` one by one, with `--batch-size N` to bound the forward pass, `--sort-by-length` to group similarly sized requests inside it, and `--json` for JSONL output. Batch routing (`laya --batch FILE`, no `--predict`) likewise answers with `route_batch` in one pass, still without loading anything.\n\n`--sort-by-length` reaches the length grouping `Agent.predict_batch` has done since #294, which\nuntil now only the library call in [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass)\ncould ask for. A forward pass pads every state in it to the longest one, so a two-line request\nsharing a pass with a two-paragraph one spends most of its compute on padding; grouping by length\nfirst puts comparable sizes together. Measured on 128 real Yelp reviews of 69-2,293 characters,\nrun as `laya --batch FILE --predict --model laya --batch-size 8` with and without the flag, four\ninterleaved rounds on an Apple-silicon Mac at the CLI's own device pick: 41.25 s median unsorted\nagainst 29.23 s sorted, a **paired median of 1.42x** (1.39-1.44x across the four pairs), with all\n128 decisions identical and still in input order. `research/` measures the same knob at 2.15x over\n10,000 synthetic multilingual tickets. It needs `--batch-size N` with 1 < N < the number of\nrequests — a single pass over the whole file has no group to reorder — and the CLI says so on\nstderr rather than printing a same-speed result and leaving you to notice.\n\n`--questions` takes the same question dict the SDK takes, as JSON: either the mapping itself, or\n`{\"state_key\": \"body\", \"questions\": {...}}` when the question's instructions name a field other than\n`request`. It implies `--predict`, and a question set with many labels usually wants\n`--head-max-len` with it: on 58 MASSIVE-INTENT labels written as one choice question, the English\ncheckpoint goes from 24/58 correct at its default 192-token option budget to 34/58 at\n`--head-max-len 384`, for about 1.4x the per-request time on CPU. Widening it further costs the\naccuracy back, because `max_len` then leaves fewer tokens for the request itself. [Honest\nlimits](#honest-limits) describes the same budget ceiling for a 77-option question.\n\n`examples/server.py` is a self-contained FastAPI app for testing Laya without writing any code:\na two-pane playground (edit the request as a form or as JSON, run it with Ctrl+Enter, read each\nanswer's full distribution and calibrated confidence, copy it as curl or Python), plus a plain\nJSON API (`/predict`, `/predict/batch`) for scripting against.\n\n```\npip install \"laya[serve]\"\npython examples/server.py               # http://127.0.0.1:8000\n```\n\nOpen `http://127.0.0.1:8000` in a browser for the playground, or hit it directly:\n\n```\ncurl -s localhost:8000/predict -H 'content-type: application/json' -d '{\n  \"state\": {\"body\": \"We were billed twice for March. Please refund it today.\"},\n  \"questions\": {\n    \"department\": {\"type\": \"choice\",\n                   \"instructions\": \"Which department should handle this?\",\n                   \"criteria\": {\"billing\": \"invoices, payments, refunds\", \"other\": \"everything else\"}},\n    \"urgency\": {\"type\": \"score\",\n                \"instructions\": \"How urgent is this?\",\n                \"criteria\": [\"not urgent\", \"soon\", \"critical\"]}\n  }\n}' | python -m json.tool\n```\n\n`--no-preload` loads checkpoints lazily instead of all three up front; `--device cuda|cpu|mps`\npins the device. See `python examples/server.py --help` for the rest.\n\nA request body may carry `model` to pin a checkpoint instead of letting the router choose, and it\ntakes exactly the spellings `laya --model` takes: names, aliases and casing all resolve through\n`laya.router`. `GET /models` lists the checkpoints and the aliases alongside them.\n\n`/predict/batch` accepts two optional body fields that control the shape of the forward passes\nwithout changing any answer: `batch_size` (states per pass; omit it and the whole batch is one\npass) and `sort_by_length` (group similarly sized states so each pass pads to a shorter maximum —\nit needs a `batch_size` below the number of states, since one pass has nothing to reorder).\nMeasured on the running server: 64 real reviews of 91–2,293 characters, one choice question,\n`batch_size: 8`, three interleaved rounds on MPS — the default one-pass shape 5.35s median against\n3.04s sorted, a paired median of **1.77x** (min 1.76x), and 64/64 labels identical in input order.\n`batch_size: 8` on its own was 4.95s, so the length grouping is what earns the second factor:\n1.66x over the sized batch.\n\n[`laya-client`](https://github.com/NandhaKishorM/laya/blob/main/sdk/typescript/README.md) supports JavaScript and TypeScript\napplications over HTTP to a self-hosted `laya-serve` `/v1/systemone` server. It\nincludes inferred answer types, all five question presets, ESM/CommonJS exports,\nand cancellation.\n\n```\n# From this checkout, install and start the Python inference service:\npip install -e '.[serve]'\nLAYA_HOST=127.0.0.1 LAYA_MODELS=english laya-serve\n\n# In another terminal:\ncd sdk/typescript\nnpm ci\nnpm run build\nnode examples/triage.mjs\n```\n\nThe npm package is named `laya-client`. Use it when a JavaScript or TypeScript\napplication talks over HTTP to self-hosted Python `laya-serve`. Use `laya-ts`\nwhen inference must run directly inside JavaScript through its local ONNX\nruntime, without a Python server.\nSee [installation, examples, and publishing](https://github.com/NandhaKishorM/laya/blob/main/sdk/typescript/README.md) and the\n[repository analysis](https://github.com/NandhaKishorM/laya/blob/main/docs/typescript-sdk.md) for architecture and scope.\n\nTo try the Python SDK in a CPU container, see the\n[Docker Compose quickstart](https://github.com/NandhaKishorM/laya/blob/main/docs/docker.md). It runs a sample request and keeps\ndownloaded models between runs.\n\nLaya ships three checkpoints. The built-in **`Router`** is the recommended entry point: it evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass.\n\n``` python\nfrom laya import Router\n\n# Preload checkpoints into memory for instant sub-35ms routing\nrouter = Router(preload=True)\n\n# 1. State in any language or schema\nstate = {\n    \"from\": \"user@acme.com\",\n    \"subject\": \"Duplicate charge on invoice #4411\",\n    \"body\": \"Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.\"\n}\n\n# 2. Define your typed questions\nquestions = {\n    \"department\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which department should handle this request?\",\n        \"criteria\": {\n            \"billing\": \"invoices, payments, refunds\",\n            \"technical\": \"bugs, outages, system errors\",\n            \"sales\": \"pricing, new contracts\",\n            \"other\": \"everything else\"\n        }\n    },\n    \"urgency\": {\n        \"type\": \"score\",\n        \"instructions\": \"How urgent is this request?\",\n        \"criteria\": [\"not urgent\", \"soon\", \"critical deadline or blocking issue\"]\n    },\n    \"churn_risk\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does the user threaten to cancel or leave?\"\n    },\n    \"refund_requested\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does the user explicitly request a refund?\"\n    }\n}\n\n# 3. English state -> automatically routed to laya (ModernBERT-large, 39.5 ms)\nres_en = router.predict(state, questions)\nprint(\"Department :\", res_en[\"answers\"][\"department\"][\"choice\"])  # -> billing (confidence: 0.94)\nprint(\"Routing    :\", res_en[\"routing\"][\"model\"])                 # -> english\n\n# 4. Hindi state -> automatically routed to laya-multilingual (mmBERT-base, 32.8 ms)\nres_hi = router.predict({\"body\": \"मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें।\"}, questions)\nprint(\"Department :\", res_hi[\"answers\"][\"department\"][\"choice\"])  # -> billing (confidence: 0.86)\nprint(\"Routing    :\", res_hi[\"routing\"][\"model\"])                 # -> multilingual\n\n# 5. Explicit override when you want a specific checkpoint\nres_td = router.predict(state, questions, model=\"typed-decisions\")\n```\n\nEvery result carries full routing metadata explaining why the choice was made:\n\n```\nres_hi[\"routing\"]\n# {\n#   'model': 'multilingual',\n#   'repo': 'convaiinnovations/laya/multilingual',\n#   'reason': 'non-Latin script (devanagari, 100% of letters); the English checkpoint cannot read it'\n# }\n```\n\nInspect a routing decision without running any forward pass:\n\n```\nrouter.route({\"body\": \"Der Kunde wurde zweimal belastet\"}, questions).reason\n# \"Latin script but language looks like 'de', not English\"\n```\n\nVery short Latin-script text often carries nothing that identifies its language (`\"Quero cancelar\"`, `\"Esqueci minha senha\"`). Such text goes to `default`, which is `\"english\"` unless you change it. If most of your traffic is not English, set:\n\n```\nrouter = Router(default=\"multilingual\")\nrouter.route({\"body\": \"Esqueci minha senha\"}).model                 # -> multilingual\nrouter.route({\"body\": \"Please refund the duplicate charge\"}).model  # -> english\n```\n\nIf a lazy router receives an interleaved workload whose requests route to different checkpoints, calling `predict()` in a loop can still cause unnecessary checkpoint churn when the required checkpoints exceed the resident cache, for example with `max_loaded=1` or when `typed-decisions` is also used.\n\n`Router.predict_batch()` routes the full workload first, groups requests by checkpoint, then groups requests with the same question schema within each checkpoint. Each compatible group is dispatched to `Agent.predict_batch()` so states can share forward passes, and results are restored to the original request order.\n\n```\nrequests = [\n    {\"state\": \"Please refund invoice 1\", \"questions\": questions},\n    {\"state\": \"تم خصم المبلغ مرتين\", \"questions\": questions},\n    {\"state\": \"Please refund invoice 2\", \"questions\": questions},\n]\n\nresults = Router(max_loaded=1).predict_batch(requests)\n# results stay in input order while compatible requests are batched by checkpoint\n```\n\nEach item can independently set `model`, `task`, `lang`, `lang_guess`, or the token budget (` max_len`, `head_max_len`). Use `route_batch(requests)` when you only want the ordered routing decisions without loading any checkpoint. `predict_many` is an alias for `predict_batch`.\n\nRequests are validated before model loading. Different requests may use different question schemas; requests sharing a checkpoint, question schema and token budget are passed together to `Agent.predict_batch()`.\n\n[Prediction hooks](#prediction-hooks) installed on the `Router` run once per request, as they do for `predict()`, so a redaction hook rewrites every state before the model sees it. Requests that share a checkpoint run all their start hooks before their shared forward pass; see [`docs/hooks/lifecycle.md`](https://github.com/NandhaKishorM/laya/blob/main/docs/hooks/lifecycle.md#routerpredict_batch).\n\nYou can also bound the Agent-level forward-pass batch size, and forward the length grouping knob\nto every group (see [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass)):\n\n```\nresults = router.predict_batch(requests, batch_size=8, sort_by_length=True)\n```\n\nOn a shared benchmark (17,416 questions, one T4 GPU, identical questions per model):\n\n| Benchmark / Task | English ( `laya` ) | Multilingual ( `laya-multilingual` ) | `Router` (Routed) | \n|---|---|---|---|\n| MASSIVE intent, English | **0.783** | 0.657 | **0.783** | \n| MASSIVE intent, 13 other languages | 0.306 | **0.451** | **0.451** | \n| XNLI, English | **0.860** | 0.843 | **0.860** | \n| XNLI, 14 other languages | 0.521 | **0.731** | **0.731** | \n| Languages usable (>3x random) | 23 / 51 | 48 / 51 | **48 / 51** | \n| Latency, 1 question (T4 GPU) | 39.5 ms | **32.8 ms** | **32.8 ms** | \n| Latency, 10 questions batched | 158.6 ms | **72.3 ms** | **72.3 ms** | \n\nThe English checkpoint collapses on non-Latin scripts (Khmer scores **0.000 accuracy at 0.952 confidence**, the raw-temperature figure; **0.705** as served after the clamp). Because the model stays confident while being wrong, confidence gating cannot save you. `Router` detects the script in <0.5 ms pure Python before the forward pass.\n\nA cold checkpoint build costs seconds; language detection costs microseconds. The lazy default keeps **two** checkpoints resident — `english` and `multilingual`, the only two automatic routing chooses between — so a language flip costs detection only once each has been built. `max_loaded=1` rebuilds the checkpoint it just evicted on *every* switch (measured at a 7.4 s median reload on CPU and 10.3 s on T4), and traffic that only ever sees one language never builds the second, so the default costs a single-language deployment nothing.\n\nFor a server or production app, preload:\n\n```\n# Every checkpoint resident in memory; language flips cost detection only (<1 ms)\nrouter = Router(preload=True)\nrouter = Router(preload=True, device=\"cuda\")\n\n# Or preload only the specific checkpoints you serve:\nrouter.preload([\"english\", \"multilingual\"])\n\n# If your app already built an agent, attach it to avoid duplicate VRAM:\nrouter.attach(\"english\", existing_agent)\n\n# Manage resident memory (default keeps two hot: english + multilingual, LRU eviction)\nrouter = Router(max_loaded=3)       # keep all three hot, e.g. with auto_task_detection\nrouter = Router(max_loaded=1)       # memory-constrained host, reloads on every switch\nrouter.unload()                     # free memory\n```\n\n| Deployment Mode | Per-Request Latency | Model Reloads | \n|---|---|---|\n| `Router()` (lazy,`max_loaded=2` ) | detection only (<1 ms) on a switch, after each language's first load | 1 the first time a language appears | \n| `Router(max_loaded=1)` | 7 to 10 s on every language switch | 1 per switch | \n| `Router(preload=True)` | **32.8 ms (GPU) / 193–464 ms (CPU)** | **none** | \n\nA rebuild still re-reads the checkpoint, but each checkpoint's tokenizer is parsed once per process\nand reused by every `Agent` — including one the Router rebuilds after eviction. The multilingual\n`tokenizer.json` alone is 34 MB / 256k vocab, several times the cost of applying its weights.\nPreloading is still the right answer for a server: it removes the rebuild rather than making it\ncheaper.\n\nRouting asks one question: *can the English checkpoint read this state?* The built-in detector answers it from the script and a function-word heuristic, and is deliberately dependency-free. That heuristic is best-effort on Latin-script languages it holds no word list for, so a short request can carry no usable signal:\n\n``` python\nfrom laya.lang import analyse\nanalyse(\"Care este ora in Tokyo?\")\n# {'script': 'latin', 'language': 'en', 'is_english': True}   -> the English checkpoint\n```\n\nIf you already run a language-identification model, hand routing the answer instead of relying on the heuristic. `lang_guess` takes a language code or a callable receiving the state, and is checked after an explicit `lang=` and before detection:\n\n```\n# A code you already know\nrouter.predict(state, questions, lang_guess=\"ro\")\n\n# A callable, e.g. wrapping fastText, CLD3 or a transformer LID\nrouter.predict(state, questions, lang_guess=lambda s: my_lid(s))\n\n# Or install one for every request on a server\nrouter = Router(preload=True, lang_guess=my_lid)\n```\n\nThe hint only decides *English or not*: a code whose primary subtag is `en`, `eng` or `english` routes to the English checkpoint, and every other code that names a language routes to the multilingual one. `\"en_US\"` and `\"en_US.UTF-8\"` are read as English, so `$LANG` can be passed straight through. Returning `None`, or a code that names no language, makes it abstain and the built-in detector decides as before — so a LID model that is unsure does not force a checkpoint. `C`, `POSIX` and `C.UTF-8` abstain, which matters because `C.UTF-8` is the default `$LANG` in the official Python image: passing it through no longer pins every request to the multilingual checkpoint, which is what it used to do. The ISO 639-2 special codes `und`, `zxx` and `mul` abstain for the same reason. An explicit `model=`, `task=` or `lang=` still wins, and the default path is unchanged.\n\n`laya.serve` exposes the `Router` over HTTP on the same `POST /v1/systemone`\nwire protocol as TypeSafe's hosted Jev API. Laya's answer payload is already\nschema-identical to what Jev returns (`choice`/` score`/` noul` answers and a\n`{input_tokens, output_tokens}` usage block), so an existing Jev client — e.g.\nthe [`hs-jev`](https://github.com/getmissionctrl/hs-jev) Haskell client — just\nneeds its `baseUrl` repointed; nothing else changes.\n\n```\npip install \"laya[serve]\"          # adds fastapi + uvicorn + python-multipart\nLAYA_DEVICE=cuda LAYA_PRELOAD=1 laya-serve   # binds 0.0.0.0:8000, preloads all 3 checkpoints\ncurl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{\n  \"state\": {\"body\": \"billed twice, refund please or we cancel\"},\n  \"questions\": {\"dept\": {\"type\": \"choice\", \"instructions\": \"which team?\",\n                \"criteria\": {\"billing\": \"refunds\", \"tech\": \"bugs\"}}}\n}'\n```\n\nTo evaluate multiple states in a single call, send a `states` array to `POST /v1/systemone/batch` (capped at 64 states):\n\n```\ncurl -s localhost:8000/v1/systemone/batch -H 'content-type: application/json' -d '{\n  \"states\": [\n    {\"body\": \"billed twice, refund please\"},\n    {\"body\": \"cannot login, getting 500 error\"}\n  ],\n  \"questions\": {\"dept\": {\"type\": \"choice\", \"instructions\": \"which team?\",\n                \"criteria\": {\"billing\": \"refunds\", \"tech\": \"bugs\"}}}\n}'\n```\n\nThe endpoint shares forward passes via `Router.predict_batch` and returns an array of `results` in matching order along with aggregated `total_usage`.\n\nConfiguration is by environment variable: `LAYA_HOST`, `LAYA_PORT`,\n`LAYA_DEVICE`, `LAYA_PRELOAD`, `LAYA_MODELS` (comma list to preload),\n`LAYA_THREADS` (cap torch intra-op threads for CPU inference — keep at or below\nphysical cores), `LAYA_AUTO_TASK`, `LAYA_MAX_LOADED` (checkpoints resident at\nonce, 2 by default; raise it to 3 when `LAYA_AUTO_TASK` makes a third one\nreachable on demand, or the server rebuilds one every time routing switches),\nand `LAYA_API_KEY` (when set, clients must\nsend `Authorization: Bearer <key>`). A client's `model` field is honoured when it\nnames a Laya checkpoint (`english`/` multilingual`/` typed-decisions`), otherwise\nthe router auto-selects by script/language. The same body can also carry `task`,\n`lang`, `lang_guess`, `min_confidence`, `max_len` and `head_max_len` — the controls\n`Router.predict` takes that a JSON body can state — while the five hook arguments\n(`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`) are refused\nwith 422, because a hook is a callable that has to live where the server runs.\n\nWhen publishing the server under a reverse-proxy prefix such as `/laya`, set\n`LAYA_ROOT_PATH=/laya`. The proxy should strip that prefix before forwarding;\nthe setting controls FastAPI-generated URLs and leaves the internal routes unchanged.\n\nThree things differ from Jev when you port a client:\n\n- **Options per question.** A question's options share the checkpoint's option budget,`head_max_len` (192 tokens on`laya` , 256 on the other two), not Jev's cap of 255 options. In addition, the HTTP server (`laya.serve` ) enforces an amplification guard of at most 100 choice options per question (`MAX_CHOICE_OPTIONS = 100` , rejected with 413 before inference). Once options overflow the token budget, around 20 options with a short description each, every option is trimmed to fit, so long or similar labels can reach the model reading the same ([Where Jev leads](#where-jev-leads) ). Once they no longer fit the window at all, the library rejects the request with 422. For more candidates, narrow them first with`predict_shortlist` ([Honest limits](#honest-limits) ).\n- **Score levels.** Every level needs a description. A`null` level is rejected with 422 rather than scored and echoed back in`legend` .\n- **`confidence`** on` choice` and`score` answers is 1 minus normalised entropy, a measure of how concentrated the distribution is, not Jev's`(n·p_max − 1)/(n − 1)` . A threshold carried over from Jev does not transfer. For one calibrated number on every question type, gate on`answer_confidence` , the probability of the reported answer. Both values shift with the number of options, so fit and validate any cutoff on held-out data at the option counts your workload uses ([#394](https://github.com/NandhaKishorM/laya/issues/394) ).\n\nThis repo is a flake. On a machine with an NVIDIA GPU:\n\n```\nnix run .#laya-serve          # build (prebuilt CUDA torch, no compile) and serve\nnix develop                   # dev shell: torch-bin, transformers, fastapi, pytest\n```\n\nFor a NixOS host, import the module and enable the service:\n\n```\n# flake inputs:  laya.url = \"github:<you>/laya\";  # or path:/… on the same host\n{\n  imports = [ laya.nixosModules.default ];\n  services.laya-serve = {\n    enable = true;\n    host = \"0.0.0.0\";           # or bind to the Tailscale/LAN address\n    openFirewall = true;\n    device = \"cuda\";\n    models = [ \"english\" \"multilingual\" \"typed-decisions\" ];\n    # apiKeyFile = config.age.secrets.laya-api-key.path;  # optional bearer auth\n  };\n}\n```\n\nThe module runs a hardened `DynamicUser` systemd unit with CUDA device access,\ncaches weights under `/var/lib/laya-serve`, and reads the bearer token (if any)\nvia `LoadCredential` so it never enters the store.\n\nIf you only need a single checkpoint for a dedicated pipeline, you can load models directly:\n\n``` python\nimport laya\n\n# 1. Load a specific checkpoint directly from the hub\nagent = laya.load(\"convaiinnovations/laya\")                           # English root\nagent_ml = laya.load(\"convaiinnovations/laya\", subfolder=\"multilingual\") # 100+ languages\nagent_td = laya.load(\"convaiinnovations/laya\", subfolder=\"typed-decisions\")\n\n# 2. Run all questions in ONE single forward pass (~35 ms on GPU)\nresult = agent.predict(state, questions)\nanswers = result[\"answers\"]\n\nprint(\"Department :\", answers[\"department\"][\"choice\"])   # -> billing (confidence: 0.94)\nprint(\"Urgency    :\", answers[\"urgency\"][\"score\"])        # -> 1.84 / 2.0\nprint(\"Churn Risk :\", answers[\"churn_risk\"][\"noul\"])       # -> 0.892 (89.2% probability)\n```\n\nPassing an empty question dictionary to `agent.predict(state, {})` or\n`agent.system_one(state, {})` returns the standard response with `\"answers\": {}`\nand `\"usage\": {\"input_tokens\": 0, \"output_tokens\": 0}`. The state is not tokenized\nand no model forward pass runs.\n\n`predict` handles one state per call, which leaves most of the GPU's batch dimension idle. When you\nhave a list of items to score against the *same* questions — a backlog of tickets, a table of rows,\na log slice — `predict_batch` packs them into shared forward passes:\n\n```\nstates = [{\"body\": t} for t in ticket_texts]           # a list of states\n\nresults = agent.predict_batch(states, questions)       # one forward pass for the whole list\n# results[i] corresponds to states[i], with the same output shape as predict\n\n# Bound peak memory when the list (or the texts) are large — chunk into passes of N:\nresults = agent.predict_batch(states, questions, batch_size=64)\n\n# Reduce padding when input lengths vary; results still follow the original state order:\nresults = agent.predict_batch(states, questions, batch_size=64, sort_by_length=True)\n```\n\n`sort_by_length=True` groups states by their longest encoded question row, after truncation.\nIt looks ahead at most eight batches and reuses the encoded rows for sorting. This uses more temporary\nCPU memory for tokenized inputs, and takes effect only when `1 < batch_size < len(states)`.\nBenchmark it on your workload and backend: uniform lengths offer little benefit, and changed\nbatch shapes can cause small floating-point differences, including near decision thresholds.\nHooks still see states and final results in input order. The option is available on `Agent` and\n`ONNXAgent`.\n\nResults are aligned with `states` by index and identical in shape to `predict`. Changing batch\nshapes can introduce floating-point differences on CPU and GPU; check decision thresholds on\nyour workload, particularly with mixed precision. Batching is a **GPU\nthroughput win** — on an RTX 5060 Ti, per-decision latency drops from ~10 ms one-by-one to ~1 ms\nbatched (measured ~9–10×). On CPU, increasing batch size alone may not speed up inference;\nlength grouping can help by reducing the padded work in a mixed-length workload. See the\n[CPU measurements and reproduction commands](https://github.com/NandhaKishorM/laya/blob/main/research/README.md#length-batching).\n\n`ONNXAgent.predict_batch(states, questions, batch_size=..., sort_by_length=...)` has the same\ncontract, backed by one ONNX Runtime session run per chunk, so an ONNX deployment gets the same\nbatch API, the same result shape, and the same length grouping. Measured on the English checkpoint\n(flat fp32 export, Apple M1, 80 support tickets alternating short and ~8x-longer documents,\n`batch_size=4`, best of 3): wall clock 53.7 s unsorted → 36.3 s sorted (**~1.48x**), with 0/80\ndecision changes and max probability drift 0.0 across queue/noul/score.\n\n`predict`/` system_one` truncate a state that exceeds `max_len` to a single window (the first, or\nfor a conversation list the last), silently dropping the rest. `predict_long` scans the whole state\nin overlapping windows, scores them in shared forward passes (via `predict_batch`), and aggregates\nper question:\n\n```\nresult = agent.predict_long(state, questions)              # windows the state, one result back\nresult = agent.predict_long(state, questions, window=256)  # smaller window isolates a localized span\nresult = agent.predict_long(state, questions, hooks=[AuditLog()])   # the scan, instrumented\n```\n\n- `noul` takes the strongest window (the statement holds if any window supports it).\n- `choice` /`score` take the most-confident window — averaging over a long, mostly-neutral\ndocument lets the neutral majority out-vote the one window that saw the deciding span.\n- A state that already fits one window is passed straight to `system_one` (identical output, plus`usage[\"windows\"] = 1` ). The key is total:`1` single window,`N` scanned windows,`0` a hook\nanswered the document before the model read any of it.\n- Hooks wrap the inference that answers the state, so on a scanned document `on_predict_start` fires once with`ctx.states` holding the decoded windows, not the state you passed in (it was\ntokenized to produce them). A start hook may replace that list: the answers are aggregated over\nwhatever reached inference, and`usage[\"windows\"]` counts those states. What a rewritten scan\ncosts is the attribution —`answer[\"window\"]` names a span of*your* document, so it is only\nreported when the scan reached inference unchanged. A hook that means to answer the document\ncalls`ctx.skip(...)` instead: its result comes back with no`answer[\"window\"]` and`usage[\"windows\"]` at 0, because no window scored it.\n\nA smaller `window` isolates a short deciding span better (it becomes a larger fraction of its\nwindow); the default (`max_len - head_max_len`) favors context and throughput. Output shape matches\n`predict`, with `usage[\"windows\"]` added.\n\n`ONNXAgent.predict_long(state, questions, window=..., stride=..., batch_size=...)` has the same\ncontract and the same aggregation rules, with the windows scored through `ONNXAgent.predict_batch`\n— one ONNX Runtime session run for all of them, or one per chunk when `batch_size` bounds memory.\n\nThe returned probability is the deciding window's, **not a calibrated number for the whole\ndocument** — a `noul` max drifts up with the window count even with no signal, and `choice` can land\non a confidently-neutral window when nothing is decisive. Each answer carries `answer[\"window\"]`\n(the deciding window's `index`, `token_start`/` token_end`, and `count`) so you can check the span\nthe answer actually came from:\n\n```\nr = agent.predict_long(state, questions)\nr[\"answers\"][\"refund\"][\"window\"]   # {'index': 13, 'token_start': 4680, 'token_end': 5432, 'count': 14}\n```\n\nThe same scan is reachable from the Router, which routes first and then windows the checkpoint it\npicked — the same `model=`/` task=`/` lang=` hints, hooks and `routing` key as `predict`:\n\n```\nresult = router.predict_long(state, questions, model=\"multilingual\")\n```\n\n`pip install laya[fast]` adds an optional forward built from fused [TileLang](https://github.com/tile-ai/tilelang)\nkernels: GEMM + bias/activation epilogues, GEMM + GEGLU, residual + LayerNorm, in-place RoPE, and a\nsliding-window flash attention that reads the packed QKV buffer directly. Weights stay resident in bf16\nand every (batch, length) bucket is captured as a CUDA graph, so a one-question call no longer pays\n~200 kernel launches from Python.\n\n```\nagent = laya.load(\"convaiinnovations/laya\", fast=True)   # or: agent.accelerate()\nagent.predict(state, questions)                            # same API, same answers\n```\n\nNumerics: on a fixed set of 60 states the fast path stays within 0.046 of an fp32 forward and within 0.076 of the stock\nbf16 path (max |Δp| ≤ 0.05 vs fp32 on both checkpoints, argmax agreement ≥ 47/48 per question type; every per-option\nprobability is in `benchmarks/results/parity_*.json`) — see `benchmarks/parity_fast.py` and [BENCHMARKS.md](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md#gpu-fast-path).\nThe fast path runs in the agent's autocast dtype at the time `accelerate()` is called: bf16 by default, fp16 if\n`agent.dtype` is `torch.float16`, where it stays within 0.009 of fp32 on the same set ([BENCHMARKS.md](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md#fp16)).\nFalls back to the stock forward on CPU/MPS or when `tilelang` is not installed; `agent.deaccelerate()`\nrestores it. Kernels compile once per shape bucket on first use (a few seconds, cached on disk).\n\n`compile=True` compiles on the first request that needs a graph, and the first single-question request\nneeds a second one (torch specialises a batch of 1). Both stall a live request. `agent.warmup()` runs the\nforward on a few synthetic shapes now and returns the seconds it took, so the compiles happen before\ntraffic arrives:\n\n```\nagent = laya.load(\"convaiinnovations/laya\", compile=True)\nagent.warmup()                   # ~46 s on an RTX 4070 Ti SUPER; every later request ~10-30 ms\n```\n\nMeasured with `benchmarks/bench_compile.py --device cuda [--warmup]` (English checkpoint, torch 2.11, ten\nrequests of changing shape): without it the first request took 51 s and the first single-question request,\nthe eighth, took another 41 s; after `warmup()` no request took more than 30 ms. It works the same with\n`fast=True` (kernels and CUDA graphs for those buckets) and costs a few forward passes on the stock path.\nInductor caches compiled graphs under `TORCHINDUCTOR_CACHE_DIR` (by default in `/tmp`); point it at a\npersistent directory to keep them across restarts.\n\nBecause Laya's probabilities are trained with strictly proper scoring rules (RLCD), confidence scores are statistically meaningful:\n\n```\ndept = answers[\"department\"][\"choice\"]\nconf = answers[\"department\"][\"confidence\"]\n\nif conf >= THRESHOLD:               # refit and validate THRESHOLD on your own held-out data\n    route_automatically(dept)       # above it: act, and sample the decisions you act on\nelse:\n    escalate_to_human_agent(dept, reason=f\"Low confidence ({conf:.2f})\")\n```\n\nA threshold is a policy you choose from measured accuracy at that coverage on your data, not a property of the model. Both checkpoints are over-confident as shipped and `laya-multilingual` has no fitted temperatures at all, so fit them before relying on these numbers — see [Calibration](#calibration) above, and the [fine-tuning notebook](https://github.com/NandhaKishorM/laya/blob/main/notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) for the fitting loop itself. Then pick the point where the errors you accept are ones you can live with. Confidence orders decisions; it does not establish that a decision is correct.\n\nA threshold also depends on the autocast dtype. On CUDA at compute capability 8 or above the runtime uses the checkpoint's `amp_dtype`, which is bf16 for all three shipped checkpoints. On the fixed set from `benchmarks/parity_fast.py` (60 states, 288 questions per checkpoint, RTX 2000 Ada) bf16 moves a probability by up to 0.073 against the fp32 forward and flips 3 of 864 argmaxes across the three checkpoints; fp16 stays within 0.019 and flips none, at the same latency. `LAYA_CUDA_AMP=fp16` selects fp16 and `LAYA_CUDA_AMP=bf16` selects bf16 (`LAYA_CPU_AMP=bf16` is the CPU counterpart). MPS autocasts in fp16 too, but the overhead dominates on a single small row, so there it engages only once a call reaches `mps_amp_min_rows` rows -- 5 by default, `LAYA_MPS_AMP_MIN_ROWS` to move it; a value that does not parse falls back to 5 and anything below 1 is clamped to 1. On MPS, `agent.dtype` is the autocast target, so it says float16 even when every call stays below the gate and runs in fp32; `agent.dtype_for(rows)` returns the precision a call with that many rows runs in. Fit and measure a threshold in the dtype you serve with.\n\n`predict`, `predict_batch`, `system_one` and `decide` — on `Agent`, `Router` and `ONNXAgent` — take an opt-in `min_confidence`, off by default. It is a caller-side policy on top of the emitted confidence: every answer whose `answer_confidence` falls below the threshold is flagged `low_confidence: True`, with the raw answer, probabilities and confidence left intact for inspection.\n\n```\nres = agent.predict(state, questions, min_confidence=0.85)\nans = res[\"answers\"][\"department\"]\n\nif ans.get(\"low_confidence\"):        # answer_confidence < 0.85\n    escalate_to_human_agent(ans[\"choice\"], reason=f\"Low confidence ({ans['answer_confidence']:.2f})\")\nelse:\n    route_automatically(ans[\"choice\"])\n```\n\nThe threshold reads `answer_confidence` (`max(p)`) — the calibrated quantity, invariant to the number of options — never the entropy `confidence`. With `decide(..., min_confidence=...)` a low-confidence field comes back as `None` in the schema output, while `return_details=True` keeps the answer and its confidence. [LangChain `LayaRouter`](https://github.com/NandhaKishorM/laya/blob/main/docs/langchain.md)'s `confidence_threshold` reads the same value: `answer_confidence` when the answer carries it, `confidence` otherwise. Left unset, `min_confidence` changes nothing.\n\nHooks observe or shape every decision without forking: audit logging, PII redaction before inference, caching, metrics, confidence gating, routing overrides, and forwarding to an external service. They are opt-in, and unset hooks are a no-op.\n\n``` python\nimport laya\n\ndef log(ctx):\n    print(ctx.model, ctx.results[0][\"answers\"], ctx.elapsed_ms)\n\nagent = laya.load(\"convaiinnovations/laya\", on_predict_end=log)\nagent.system_one(\"I was charged twice.\", {\"urgent\": {\"type\": \"noul\", \"instructions\": \"Urgent?\"}})\n```\n\nA hook is a plain callable, or an object implementing any of `on_predict_start`,\n`on_predict_end`, `on_route`, `on_load`, `on_evict`, `on_error`. A start hook can rewrite the\nstate/questions or `ctx.skip(...)` a cached answer; an end hook can rewrite the result. See\n[**`docs/hooks/`**](https://github.com/NandhaKishorM/laya/blob/main/docs/hooks/index.md) and [` examples/hooks/`](https://github.com/NandhaKishorM/laya/blob/main/examples/hooks).\n\nDescribe the shape you want with a JSON schema or a pydantic model, and Laya answers it in one forward pass, with typed values and calibrated confidence.\n\n``` python\nimport laya\n\nschema = {\n    \"type\": \"object\",\n    \"properties\": {\n        \"department\": {\"type\": \"string\", \"enum\": [\"billing\", \"support\", \"sales\"],\n                       \"description\": \"Which team should handle this?\"},\n        \"urgency\": {\"type\": \"integer\", \"minimum\": 0, \"maximum\": 2},\n        \"needs_human\": {\"type\": \"boolean\"},\n    },\n}\n\nagent = laya.load(\"convaiinnovations/laya\")\nagent.decide(\"I was charged twice, refund me.\", schema=schema)\n# {\"department\": \"billing\", \"urgency\": 2, \"needs_human\": True}\n```\n\n`decide` also works on a `Router`, accepts a pydantic model (install `laya[structured]`), and can\nreturn per-field confidence with `return_details=True`. In an LCEL chain or LangGraph node the\nsame call is `LayaDecision`. See [` docs/structured.md`](https://github.com/NandhaKishorM/laya/blob/main/docs/structured.md).\n\nFor throughput, `decide_batch` answers a list of states against the *same* schema through\n`predict_batch` (see [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass)), so the schema\nis planned once and the states share forward passes:\n\n```\nvalues = agent.decide_batch(ticket_texts, schema=schema)   # values[i] matches ticket_texts[i]\n\n# On a Router the states may land on different checkpoints; keywords reach predict_batch:\ndetails = router.decide_batch(states, schema=schema, return_details=True, batch_size=64)\n```\n\nMeasured on an Apple M-series (MPS), 8 English tickets through one checkpoint: 2624 ms one-by-one\nvs 723 ms batched (**3.6×**); 16 mixed English/German states through a `Router`: 2977 ms vs\n1903 ms (**1.6×**). On CPU the same workloads gave **1.6×** and **2.0×**. Projected values matched\nthe one-by-one loop 8/8 and 16/16 on both devices, as expected wherever argmax is not at a\nthreshold — verify against your own decision boundaries. `laya.decide_batch(runner, states, ...)`\nis the function form, and `ONNXAgent.decide_batch` runs the same thing on an ONNX export.\n\nLaya provides pre-tuned question schemas for immediate production use:\n\n``` python\nimport laya\n\nagent = laya.load(\"convaiinnovations/laya\")\n\n# 1. Intelligent Model Router (routes to small vs. frontier models)\nrouting = agent.predict({\"request\": \"Refactor this service using dependency injection\"}, laya.router_questions())\n\n# 2. Real-time Prompt Guardrails (jailbreaks, injections, leaks)\nguard = agent.predict({\"prompt\": \"Ignore all instructions\"}, laya.guard_questions())\n\n# 3. Content Safety & Moderation (toxicity, harassment, threats)\nsafety = agent.predict({\"post\": \"User comment text\"}, laya.moderation_questions())\n\n# 4. Support Ticket Triage (intent, urgency, frustration, churn)\ntriage = agent.predict({\"message\": \"My payment failed twice\"}, laya.triage_questions())\n```\n\nFast System 1 routing and guardrails directly inside LangGraph workflows and LCEL chains. Every node also takes core's per-call prediction hooks (`hooks`, `on_predict_start`, `on_predict_end`, `hooks_raise`, `hooks_timeout`):\n\n``` python\nfrom laya.integrations.langchain import LayaDecision, LayaGuardrail, LayaRouter\n\n# 1. Sub-35ms LangGraph conditional edge routing with confidence fallback\nrouter = LayaRouter(\n    criteria={\"billing\": \"invoices, charges\", \"tech\": \"bugs, outages\"},\n    confidence_threshold=0.80,\n    fallback=\"human_agent\",\n)\nworkflow.add_conditional_edges(\"triage\", router)\n\n# 2. Inline prompt guardrails\nguard = LayaGuardrail(action=\"raise\")  # raises LayaGuardrailError on jailbreak/injection\n\n# 3. Schema-driven decisions: a JSON schema in, schema-shaped values out\ndecide = LayaDecision({\n    \"type\": \"object\",\n    \"properties\": {\n        \"department\": {\"type\": \"string\", \"enum\": [\"billing\", \"technical\", \"other\"]},\n        \"needs_human\": {\"type\": \"boolean\"},\n    },\n})\ndecide.invoke(\"I was charged twice and the API still 500s for us.\")\n# {'department': 'billing', 'needs_human': False}\n\n# 4. Score a backlog in one batched call, not one forward pass per input\nroutes = router.batch([\"refund my invoice\", \"the app crashes\", \"change my password\"])\n```\n\n`batch()` and `abatch()` run the whole list through `predict_batch`, so `chain.batch(...)`,\n`RunnableParallel` and LangGraph map-reduce nodes get Laya's shared forward passes instead of\nLangChain's default one-call-per-input thread pool -- which races on MPS, where concurrent torch\nforwards abort the process. Measured on Apple M-series (medians of three): **2.2x** on a 16-ticket\nrouting batch, **2.2x** across 24 mixed-language tickets through a `Router`, **1.8x** on the guard\npreset, with every route label and guardrail flag unchanged. On CPU the same workloads are\n2.2-2.4x over the one-by-one loop and 1.1-1.5x over the thread pool.\n\nSee [**`docs/langchain.md`**](https://github.com/NandhaKishorM/laya/blob/main/docs/langchain.md) for full guide, support ticket triage nodes, and remote HTTP server configuration.\n\n| Primitive | Output | Use Cases | \n|---|---|---|\n| **`choice`** | Top label, probabilities per option, confidence | Department routing, intent classification, topic categorization | \n| **`score`** | Expected level on ordinal rubric, distribution, confidence | Frustration level, ticket urgency, harm severity | \n| **`noul`** | Calibrated probability P(true) from 0.0 to 1.0 | Phishing detection, spam filtering, jailbreak detection, churn risk | \n\n`noul` always scores two semantic slots in `[false, true]` order and returns the probability of\nthe second slot. For compatibility, those slots are shown to the model as `false` and `true` by\ndefault. The optional `labels` mapping overrides only that model-facing text without changing the\nreturned meaning:\n\n```\nquestion = {\n    \"type\": \"noul\",\n    \"instructions\": \"Is this review positive?\",\n    \"criteria\": {\n        \"false\": \"the review is negative\",\n        \"true\": \"the review is positive\",\n    },\n    \"labels\": {\n        \"false\": \"B\",\n        \"true\": \"A\",\n    },\n}\n```\n\nThe `labels` mapping is optional. It must contain exactly the string keys `false` and `true`,\nwhose values must be distinct non-empty strings. Mapping order does not matter, and the returned\n`noul` value is still P(true). Label sensitivity varies by checkpoint and state, so validate any\noverride on your own data rather than treating `A`/` B` as a universal fix.\n\nEvery question type also takes an optional `option_order`: a permutation of the option indices\nsaying which option goes in which slot. Slot `s` shows option `option_order[s]`. Probabilities\nalways come back keyed in your own option order, whatever order the model saw them in, so the\nkey is presentation only — it never changes what a returned label means.\n\n```\nquestion = {\n    \"type\": \"choice\",\n    \"instructions\": \"Which team should handle this?\",\n    \"criteria\": {\"billing\": \"...\", \"technical\": \"...\", \"sales\": \"...\"},\n    \"option_order\": [2, 0, 1],     # slot 0 shows `sales`, slot 1 `billing`, slot 2 `technical`\n}\n```\n\nIt exists because where an option sits changes the answer. `research/eval/presentation_checks.py`\nmeasures the English checkpoint on five options whose text is *identical*: the per-slot logits\ncentre at `[+1.68, +1.47, +0.11, -1.35, -1.91]`, a spread decided by position alone, and the\nmultilingual checkpoint leans the other way on `score` (see [#131](https://github.com/NandhaKishorM/laya/issues/131)).\nThe checkpoint-side fix is a position-balanced retrain; until then `option_order` lets a caller\naverage the effect out, by asking the same question under orders in which every option occupies\nevery slot equally often:\n\n```\nk = len(question[\"criteria\"])\n\n# the k rotations go in as k questions, so they share one forward pass\nrotations = {\n    \"q%d\" % r: dict(question, option_order=[(i + r) % k for i in range(k)])\n    for r in range(k)\n}\nanswers = agent.predict(state, rotations)[\"answers\"]\n\ntotals = {label: 0.0 for label in question[\"criteria\"]}\nfor answer in answers.values():\n    for label, value in answer[\"probabilities\"].items():\n        totals[label] += value / k\n\ndecision = max(totals, key=totals.get)\n```\n\n`predict` already scores every question in a request in one parallel forward pass, so this costs\n`k` rows rather than `k` forward passes — sending the rotations as separate `predict` calls would\ncost `k` of them. On 62 banking intents shortlisted to 20 over 49 states, averaging this way took\nthe share of answers that change with presentation order from 16.3% to 6.1%. Omitting\n`option_order` keeps the canonical order and the behaviour every existing caller already has.\n\nLaya can be exposed as an [MCP](https://modelcontextprotocol.io) stdio server, so any MCP\nclient (OpenClaw, Claude Desktop, Cursor, ...) can call typed decisions as tools\n(`laya_predict`, `laya_predict_batch`, `laya_route`, `laya_route_batch`, `laya_decide`, `laya_shortlist`, `laya_preset`, `laya_status`) without writing glue code.\nThis is an **optional extra**: the core package has no `mcp` dependency.\n\n```\npip install \"laya[mcp]\"\nlaya-mcp-server          # or: python -m laya.mcp.server\n```\n\nExample MCP client configuration (stdio transport):\n\n```\n{\n  \"mcpServers\": {\n    \"laya\": {\n      \"command\": \"laya-mcp-server\",\n      \"env\": { \"LAYA_DEVICE\": \"cpu\" }\n    }\n  }\n}\n```\n\nThe environment variables follow the contract documented at the top of\n[`laya/serve.py`](https://github.com/NandhaKishorM/laya/blob/main/laya/serve.py), so the same variable has one meaning across the\npackage:\n\n| Variable | Default | Meaning | \n|---|---|---|\n| `LAYA_DEVICE` | (auto) | Same as `laya.serve` : the device type is lower-cased (`CUDA` →`cuda` ,`CUDA:0` →`cuda:0` ) and passed to torch, whose device parser is case-sensitive | \n| `LAYA_PRELOAD` | `1` | Same as `laya.serve` : build the checkpoints at startup, not lazily | \n| `LAYA_MODELS` | `english,multilingual` | Comma list to preload (serve contract). MCP difference: an empty value preloads `english,multilingual` so`typed-decisions` stays lazy; in`laya.serve` empty means every checkpoint | \n| `LAYA_THREADS` | (torch default) | Same as `laya.serve` : cap torch intra-op threads for CPU inference; keep it at or below the physical core count | \n| `LAYA_AUTO_TASK` | `0` | Same as `laya.serve` :`1` lets a request whose question ids match a typed-decisions workflow route to that checkpoint, which is then loaded on demand; it never joins the preload list | \n\nThe tools return structured JSON (answers with probabilities, routing metadata, device,\n`latency_ms`). `laya_predict_batch` and `laya_route_batch` are the MCP form of\n[`Router.predict_batch` / `route_batch`](#batch-mode-score-many-states-in-one-forward-pass):\none tool call takes an array of `{state, questions, model?, task?, lang?, lang_guess?, max_len?, head_max_len?}` requests, routes them\nfirst, groups them by checkpoint, and shares forward passes between requests with the same\nquestion schema, returning the answers in input order. `max_len` / `head_max_len` are per request\nthere, exactly as `Router.predict_batch` reads them, so one wide question can raise its own budget\nwithout shrinking the rest of the batch to it. On 16 mixed-language tickets through\nthe tool functions themselves, one batch call beat 16 `laya_predict` calls by **2.1-2.3x on\nMPS** (983-1082 ms -> 467-477 ms) and **~1.25x on CPU** (1861-2471 ms -> 1470-1911 ms), with\n**0/16 decision flips** (choice label, rounded score, noul sign) against the loop. Prefer it\nwhenever a client has more than a few requests: each saved round trip is also an MCP\nrequest/response. `laya_decide` is the MCP form of [`laya.decide`](#schema-driven-decisions): it\ntakes a JSON schema (enum choices, booleans, bounded integers) instead of hand-written\nquestions and returns the decided `values` projected onto that schema -- enum member, integer\nlevel, boolean -- beside per-field `confidence` and `probabilities`, so a client that already\nknows the answer shape never parses an answer map by hand. `laya_shortlist` is the MCP form of [`predict_shortlist`](#honest-limits):\nit shortlists a many-option choice question to its `k` most likely labels by embedding\nsimilarity (mean-pooled from the answering checkpoint's own encoder, so no extra model is\ndownloaded), answers in one forward pass, and returns per-question shortlist metadata\n(kept labels, cosine scores, `k`, option count). The guardrails shown on every decision\ntool point clients to `laya_shortlist` for >20-option choices. As with the SDK, use it for structured\ndecisions only; not for open Q&A or\ntext generation. Tests: `tests/test_mcp.py` (CI, no weights) and\n`tests/test_mcp_local_e2e.py` (local, real weights and a real stdio handshake).\n\n`laya.evals` scores a labelled dataset and gates a build on it, so a quality change is a\nreviewable diff instead of a hand-check. It is pure Python plus numpy, imports no torch, and\nneeds no weights until you point it at a checkpoint.\n\n```\nlaya-evals validate research/evals/fixture.jsonl\nlaya-evals run data.jsonl --model english --min-accuracy 0.8 --max-ece 0.05 --slice language\nlaya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 --json report.json\n```\n\n`run` reports overall and per-slice metrics (`choice_accuracy`, `noul_accuracy`, `score_mae`,\n`ece`, `mean_confidence`, latency) and exits non-zero on a threshold or baseline failure, so it\ndrops into CI unchanged. A weight-free job runs the metric and API tests on every PR, and a\nscheduled workflow evaluates the English checkpoint against the committed baseline. See\n[**`docs/evals.md`**](https://github.com/NandhaKishorM/laya/blob/main/docs/evals.md) for the dataset format and the gate.\n\nCommunity diagnostics: [Chinese workplace decisions (Feishu-style)](https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/feishu_zh/README.md) · [中文说明](https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/feishu_zh/README.zh-CN.md). Includes frozen synthetic cases, archived paired Laya/Jev responses, and an offline audit; separate from the benchmark suites below. Also [Chinese short-command routing](https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/zh_short_commands/README.md) · [中文说明](https://github.com/NandhaKishorM/laya/blob/main/research/benchmarks/zh_short_commands/README.zh-CN.md): 18 frozen commands and a seven-rung ablation of the documented prompt guidance, which locates the accuracy loss on the four-question path rather than the six-option one.\n\n**Full report: [`BENCHMARKS.md`](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md)** — every run consolidated, languages and themes, with per-language detail for all 51 languages.\n\nAll Laya numbers below are measured. Every model answered byte-identical questions\n(fixed seed) in the same run. Reproduce with\n[`research/scripts/laya_benchmark_colab.ipynb`](https://github.com/NandhaKishorM/laya/blob/main/research/scripts/laya_benchmark_colab.ipynb) on a T4.\n\n| questions per call | `laya` | `laya-multilingual` | \n|---|---|---|\n| 1 | 39.5 ms | **32.8 ms** | \n| 5 | 84.5 ms | **40.1 ms** | \n| 10 | 158.6 ms (15.9 ms/q) | **72.3 ms (7.2 ms/q)** | \n| 50 | 771 ms | **337 ms (6.8 ms/q)** | \n\nBatched throughput reaches 103-332 questions/sec on a single T4. For reference, TypeSafe Jev\nhas been independently measured at 236-276 ms p50\n([AbdelStark](https://github.com/AbdelStark/jev-benchmarks),\n[nibzard](https://github.com/nibzard/decision-model-benchmark)) -- Laya answers a single\nquestion roughly **6-7x faster**.\n\nEvery Laya figure is what `Router().predict(...)` actually returns — the checkpoint the router\nselects for that input, not a hand-picked best of three. Jev figures are **third-party\npublished, never measured here** (no TypeSafe API access), so sample sizes and prompts differ.\nThe Laya column here comes from the Applications run (`research/scripts/bench_apps.py`, N=400 per task; `research/results/app_benchmark_results.json`), so AG News / DAIR Emotion read 0.950 / 0.595 here versus 0.947 / 0.573 in the committed T4 English suites (N=600) below.\n\n|  | Jev 1.13.0 | Laya (routed) |  | \n|---|---|---|---|\n| typed-decisions, 2,000 decisions | 0.727 | **0.766** | +0.039 | \n| AG News, 4 labels | 0.910 | **0.950** | +0.040 | \n| DAIR Emotion, 6 labels | 0.480 | **0.595** | +0.115 | \n| Banking77 (72 vs 77 labels) | **0.870** | 0.425 | Jev leads on >20 options | \n| ECE *(lower better)* | 0.246 | **0.081** | 3× better (post-temperature) | \n| p50 latency, 1 question | 236–276 ms | **32.8 ms** | 7.8× faster | \n| Languages usable | *no published benchmark* | **48 of 51** | — | \n| Weights | closed API | **Apache 2.0** | — | \n| Cost | $0.042 / 1M tokens | **$0 self-hosted** | — | \n\nOn DAIR Emotion, Jev assigned **zero probability to the true label on 16% of examples** — a hard\nfailure for anything branching on confidence.\n\n- **High-cardinality label spaces (>20 options at default settings):** On Banking77, Jev scores 0.870 (on 72 labels) while Laya scores 0.425 (on 77 labels at default 256-token head budget). This is an architectural token-budget constraint: options share a fixed`head_max_len` budget (192 tokens on English, 256 on multilingual), so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. While`laya-multilingual` supports 1,024 context (and up to 8,192 in the encoder) and you can raise`agent.cfg[\"head_max_len\"] = 512` at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning.`predict_shortlist` (see[Honest limits](#honest-limits) ) keeps the top`k` labels with a caller-supplied embedding, then runs one forward pass on that shortlist.\n- **Soft distribution matching:** On typed-decisions, while Laya achieves higher argmax accuracy (0.766 vs 0.727), Jev achieves higher soft accuracy (0.580 vs 0.471) against the teacher's full probability distributions.\n- **Out-of-the-box raw calibration:** Before temperature scaling, the base checkpoint has higher raw ECE (0.213 vs 0.144). Laya achieves its 0.081 ECE after domain temperature fitting.\n\nFull detail, including every workflow and all 51 languages: **[`BENCHMARKS.md`](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md)**.\n\n400 cases, 2,000 decisions, four workflows.\n\n| model | accuracy | soft acc | Brier | ECE | score MAE | \n|---|---|---|---|---|---|\n| **`laya-typed-decisions`** | **0.766** | 0.471 | **0.062** | 0.213 | **0.242** | \n| `laya` | 0.362 | 0.332 | 0.316 | 0.175 | 0.694 | \n| `laya-multilingual` | 0.352 | 0.328 | 0.463 | 0.314 | 0.760 | \n| *Jev 1.13.0 (published)* | *0.727* | *0.580* | *0.148* | *0.144* | *0.391* | \n| *teacher self-agreement ceiling* | *0.735* |  |  |  |  | \n| *per-question majority class* | *0.461* |  |  |  |  | \n| *random guess* | *0.318* |  |  |  |  | \n\nThe fine-tuned checkpoint beats Jev by 3.9 points and clears the teacher ceiling, with 2.4x\nbetter Brier and 1.6x better score MAE. It wins on all four workflows: invoice processing\n0.804, security incidents 0.766, customer service 0.764, agent-trace observability 0.730.\nBy primitive: `noul` 0.857, `choice` 0.733, `score` 0.723.\n\nTwo places it still trails Jev: **soft accuracy** (0.471 vs 0.580 — its argmax is better but\nits distributions match the teacher less well) and **ECE** (0.213 vs 0.144), which temperature\nfitting addresses.\n\n**The base checkpoints sit below the majority-class baseline** (0.362 and 0.352 against 0.461).\nAll of the capability on this benchmark comes from fine-tuning. Base rows match the committed runs (`suites.typed_decisions` in `research/results/t4_colab_benchmark.json`: 0.3620 / 0.3515; CPU sweep `part_b`: 0.3615 for English). The `laya-typed-decisions` row and its four per-workflow scores are from the fine-tuning run and have no committed result file behind them yet.\n\n|  | `laya` | `laya-multilingual` | \n|---|---|---|\n| English | **0.783** | 0.657 | \n| 13 other languages | 0.306 | **0.451** | \n| XNLI, English | **0.860** | 0.843 | \n| XNLI, 14 other languages | 0.521 | **0.731** | \n\nAcross all 51 languages the English checkpoint macro-averages **0.227** with macro ECE\n**0.733** *(0.571 as served, after the temperature clamp)*, and only 23 of 51 languages clear\n3x random. `laya-multilingual` reaches **0.401** on the same 5,100 cases, clearing 3x random in\n48 of 51 — see [BENCHMARKS.md](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md) for why those multilingual numbers are a re-run\nrather than the committed file. Khmer scores **0.000 at 95.2% confidence** raw, **70.5%** as\nserved, on the English checkpoint. This is why\n[`Router`](#quickstart-route-mode-recommended) exists: the model's own\nconfidence gives no warning, so the routing decision has to be made before the forward pass.\n\n| task | `laya` | `laya-multilingual` | note | \n|---|---|---|---|\n| AG News | **0.947** | 0.937 | in training mix | \n| BoolQ | **0.830** | 0.787 | in training mix | \n| DAIR Emotion | **0.573** | 0.513 | held out | \n| prompt-injections | **0.698** | 0.578 | held out, n=116 | \n| SST-5 (ordinal) | 0.372 | 0.282 | held out | \n\nBoth checkpoints are over-confident as shipped. Refitting one temperature per (question type,\noption count) on held-out data moves mean ECE **0.466 -> 0.081** (`laya`) and\n**0.314 -> 0.106** (`laya-multilingual`). `laya-multilingual` ships with no fitted\ntemperatures at all, so fit them before relying on its probabilities.\n\nAt checkpoint load, numeric temperature entries are clamped to `[0.5, 5.0]`; invalid or\nnon-finite entries use the neutral fallback `1.0`. A runtime warning reports the affected\nentries and applied values. Bucket-specific temperatures still take precedence over per-type\nvalues, including when a bucket uses the fallback. Raw values remain available in\n`agent.temperature_raw` and `agent.temperature_by_options_raw`. A fallback prevents a loading\nfailure; it does not establish calibrated confidence.\n\n- \n**The base checkpoints are near chance on typed-decisions zero-shot** -- 0.362 and 0.352\nagainst a 0.318 random baseline and a 0.461 majority-class baseline. The 0.766 figure comes\nfrom the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base to\nspecialise, not a zero-shot decision engine.\n- \n**Avoid boolean-word labels in `choice` questions.** Choice keys are rendered verbatim, and the\ncurrent checkpoints can follow labels such as`true` /`false` or`yes` /`no` instead of the option\ndescriptions. Use semantic labels or opaque labels such as`A` /`B` , and validate them on the\ncheckpoint and states you serve.\n- \n**Semantic `choice` labels do not make negation safe.** In the five cancellation examples from[#377](https://github.com/NandhaKishorM/laya/issues/377) , a CPU run on Laya 0.3.20 with`no_action` /`cancel_account` keys selected`cancel_account` for all four negated requests on`laya` and two on`laya-multilingual` ; one multilingual answer assigned it probability`0.9998` .\nThe positive control passed on both checkpoints. These are narrow cancellation examples, not\nevidence that every negated state fails. Validate the exact checkpoint and wording you serve;\nusing semantic keys alone does not avoid this failure.\n- \n**High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len` ) and the remaining document/state budget.**`head_max_len` is a cap, not the head length.** The renderer fills the option prompt budget only as far as the question and its option descriptions need, and`build_sequence` then sizes the state from the head it*actually built* (`room = max_len - head_len - 1` ). The real room therefore depends on the question, not on the cap:\n  - \n`laya` (English) defaults to 512 context (`head_max_len = 192` ). The shipped department question below renders a**48-token** head, leaving**463 tokens for state** , not 320.\n  - \n`laya-multilingual` and`laya-typed-decisions` default to 1,024 context (`head_max_len = 256` ). The shipped question renders a**45-token** head, leaving**978 tokens for state** , not 768 (mmBERT-base encoder supports up to 8,192 with RoPE).`max_len - head_max_len` is**not a safe figure in either direction** , so size a state to`max_len - head_len - 1` for the question you are actually asking:\n  - \nFor a question that does not fill the cap it **understates** the room, as above — conservative, but it discards state the model would have read.\n  - \nFor a high-cardinality question it **overstates** it.`per = max(4, (head_max_len - 16) // k)` floors at 4 tokens per option and`head_ids` keeps a floor of 8, so past roughly`head_max_len / 4` options the head grows*beyond* the cap. A state sized to the cap-based figure is then**truncated** , not merely conservative:`k` optionshead real room cap-based figure `laya` ,`max_len=512` 77 319 **192** 320 `laya` ,`max_len=512` 100 411 **100** 320 `laya-multilingual` ,`max_len=1024` 100 411 **612** 768 The crossover is at `k ≈ head_max_len / 4` and the gap widens from there;`MAX_CHOICE_OPTIONS = 100` in`laya/serve.py` puts the overstating regime within reach over HTTP.\n At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:\n  1. Raise `agent.cfg[\"head_max_len\"] = 512` and`agent.cfg[\"max_len\"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request:`predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](https://github.com/NandhaKishorM/laya/blob/main/docs/langchain.md) ).`laya --questions` takes the same two budgets as`--max-len` /`--head-max-len` .\n  2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist` , example below).`predict` and`system_one` still score every criterion they are given.\n  3. Or split the label set yourself into a coarse question and a fine question.\n- \n\n``` python\nimport laya\n\nquestions = {\n    \"intent\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which banking intent is this?\",\n        \"criteria\": {\n            \"card_arrival\": \"where is my card\",\n            \"transfer_fee\": \"fee charged on a transfer\",\n            # ...the rest of a large label set\n        },\n    }\n}\nresult = laya.predict_shortlist(\n    agent,\n    {\"text\": \"I was charged twice for a transfer\"},\n    questions,\n    embed_fn=laya.embed_fn_from_agent(agent),  # or any callable: texts -> (n, dim)\n    k=20,\n)\nresult[\"shortlist\"][\"intent\"][\"labels\"]  # the top 20 labels sent to the model\n```\n\n`embed_fn(texts)` returns one vector per string. `embed_fn_from_agent` mean-pools the encoder already loaded on the agent; the decision head runs in the following `predict` / `system_one` call. Probabilities on a shortlisted choice are over those `k` labels. When `k` is at least the number of labels, the original question is passed through and `embed_fn` is not called.\n\nShortlisting the same option set on every request re-embeds option texts that do not change. Wrap the embedder once with `laya.cached_embed_fn(embed_fn)` and repeat calls embed only the new query text: lookups are exact string matches into an LRU of at most 4,096 entries (about `maxsize * dim * 4` bytes, so ~12 MB at the default with a 768-dim encoder), and texts missing from the cache are still embedded in one batched call. The wrapper's `cache_info()` reports hits and misses; call `cache_clear()` if the model behind `embed_fn` changes.\n\n[Issue #102](https://github.com/NandhaKishorM/laya/issues/102) reports that a top-20 zero-shot shortlist moved a BANKING77 run from 54.3% to 60.8% on the reporter's setup. Those figures are the reporter's; this repository has not remeasured them.\n\n- \nOrdinal `score` questions are the weakest primitive (SST-5 0.372).\n- \n**`noul` can follow its option labels instead of the state, most strongly on `laya` (English).**`noul` renders its two options as`false:` /`true:` by default, and on the English checkpoint that label pair can dominate the answer, returning a confident \"no\" for clearly positive input (#156). Until a retrained checkpoint lands, check`noul` answers on your own data. You can override the model-facing pair while keeping the`noul` result as P(true):\n\n```\n{\"type\": \"noul\", \"instructions\": \"Is this review positive?\",\n \"criteria\": {\"true\": \"the review is positive\", \"false\": \"the review is negative\"},\n \"labels\": {\"true\": \"A\", \"false\": \"B\"}}\n```\n\n A `noul` with no`criteria` renders one generic option pair for every state (`no, the statement does not hold` /`yes, the statement holds` ). On`laya` that pair carries the whole decision, so\nit answers \"no\" whatever the state — give a`noul` criteria if you need it to discriminate. On`laya-multilingual` the criteria-less form does read the state.Label sensitivity varies by checkpoint and state, so validate the override on your own data. A two-option `choice` with neutral keys remains another workaround:\n\n```\n{\"type\": \"choice\", \"instructions\": \"Is this review positive?\",\n \"criteria\": {\"A\": \"yes, the review is positive\", \"B\": \"no, the review is negative\"}}\n```\n\n `criteria` on a`noul` must be keyed`true` /`false` — those two keys*are* the option text the\nmodel reads, so any other key is rejected instead of being quietly replaced with the defaults.\nBefore that check,`criteria: {\"yes\": ..., \"no\": ...}` was accepted, dropped, and answered\nagainst`false:` /`true:` anyway, which cost 2 of 3 clearly positive reviews on the English\ncheckpoint ([#156](https://github.com/NandhaKishorM/laya/issues/156) ). Use`labels` as above to\nchange the wording without touching the option text.\n- \n**`laya-multilingual` has a position bias on `score` questions** (#131): it rarely picks the first-listed level, in any language. For English score questions, route to`model=\"english\"` , and for other languages validate score outputs on your own data before relying on them.\n- \n**`action.act_probability` carries no usable signal yet** (#185). It reads 1.0 for almost every input, and its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions). Gate on`confidence` instead, which reaches an AUROC of 0.77 on the same items.\n- \n`laya` collapses outside English;`laya-multilingual` is weaker on English. Route, or pick\ndeliberately.\n\n- **[omp-laya-judge](https://github.com/F0Rextasy/omp-laya-judge)** : an[oh-my-pi](https://github.com/can1357/oh-my-pi) plugin with a local System-1 judge MCP server and skill (`choice` /`bool` /`score` , 0 tokens, about 0.3 s on CPU), confidence-gated escalation, and reproducible quiz and Snake demos.\n- **[laya-adk-toolkit](https://github.com/Ashfaqbs/laya-adk-toolkit)** :[Google ADK](https://google.github.io/adk-docs/) tools that let an agent call Laya's`classify` /`score` /`detect` typed decisions directly as tools, instead of asking an LLM to guess at structured output.\n- **[laya-Ascend](https://github.com/zzhdbw/laya-Ascend)** : Laya on Huawei Ascend NPUs through`torch-npu` , with a CPU vs NPU benchmark (34x to 71x faster at batch size 1), a setup guide, and Snake and Tetris demos.\n- **[laya-apple](https://github.com/tc3oliver/laya-apple)** : a correctness-validated Laya runtime for Apple silicon that uses the MLX GPU and the Apple Neural Engine, with automatic routing and concurrent heterogeneous serving.\n- **[stuntd](https://github.com/bladedevoff/stuntd)** : runs Laya locally behind the Jev API (`POST /v1/systemone` , no key) and trains a head per decision on the frozen encoder from your own labelled rows, with a calibrated confidence threshold (a 12-label intent task: 89.5% zero-shot to 100% trained).\n\n- **Hugging Face Model:**[convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya)\n- **Interactive Web Demo:**[convaiinnovations/laya-demo](https://huggingface.co/spaces/convaiinnovations/laya-demo)\n- **Engineering Writeup:**[Read the full story on Dev.to](https://dev.to/nandakishor_m_6cc0adfde9f/i-built-non-autoregressive-decision-models-a-year-ago-then-a-frontier-lab-called-it-a-18me)\n\nFine-tune Laya on your own domain data. The notebook runs on Kaggle's free 2xT4 GPUs and does the whole loop: build the dataset, train with RLCD (proper-scoring-rule rewards, GRPO-style policy gradient), fit calibration temperatures, evaluate, and push the result to the Hub.\n\nThe notebook enables gradient checkpointing on both the encoder and the decision head.\nFor custom training loops, `model.head_checkpointing = True` enables activation\ncheckpointing for the decision-head layers; enable the encoder's gradient checkpointing\nseparately. During gradient-enabled training, this reduces stored intermediate activations\nby recomputing them during backward, trading extra computation for lower activation memory.\nThe head flag defaults to `False` and is bypassed in evaluation and under `torch.no_grad()`.\n\nThe notebook fits one `temperature` per type (`choice`, `score`, `noul`) and removes inherited\n`temperature_by_options` from the exported config. Otherwise those old bucket values take\nprecedence at inference and silently mask the new fit. Existing checkpoints still honor\nintentional bucket-specific temperatures, falling back to the corresponding per-type value\nwhen a bucket is absent; the runtime's temperature clamp is unchanged.\n\nThis fixes configuration persistence, not measured model accuracy or calibration quality.\nThe notebook's calibration samples come from its training items; evaluate on separate held-out\ndata before claiming an improvement. Already published checkpoints are not rewritten.\nRun the CPU-only regression checks with `python tests/test_calibration_persistence.py`\n(synthetic configs and tiny local fixtures; no pretrained downloads or training).\n\nFine-tuning is where most of the value is. On the typed-decisions benchmark the base\ncheckpoints score near chance zero-shot (0.36 and 0.35 against a 0.318 random baseline),\nwhile the fine-tuned checkpoint reaches **0.766** on the same 2,000 decisions -- above\nTypeSafe Jev's published 0.727 and above the 0.735 teacher self-agreement ceiling. Treat Laya\nas a fast base to specialise, not as a zero-shot decision engine.\n\nRuntime on 2xT4 is roughly 4-5 hours for 4 epochs over ~30k questions.\n\n[`docs/finetune_browser_agent.md`](https://github.com/NandhaKishorM/laya/blob/main/docs/finetune_browser_agent.md) records a complete specialisation\non a single 16 GB GPU with no paid API: Laya as the operation/target decider for\n[browser-use/jev-ultrafast](https://github.com/browser-use/jev-ultrafast) (same request format as\nTypeSafe Jev). Element top-1 among ~45 candidates goes from 0.10 zero-shot to 0.66, real-task\nsuccess from 0 % to 62 % at 17-23 ms per step; weights, pipeline code and per-run results are on\nthe Hub at [cklxx/laya-browser](https://huggingface.co/cklxx/laya-browser). The write-up covers the\ndata recipe (reverse-generated goals, executed DONE states, Mind2Web, on-policy corrections), the\ninput-format change that mattered most, and the things that did not work.\n\nIf Laya helps your research or products, consider supporting independent research:\n\nApache 2.0. Developed by Convai Innovations.", "url": "https://wpnews.pro/news/laya-multilingual-non-autoregressive-system-1-decision-engine", "canonical_source": "https://github.com/NandhaKishorM/laya", "published_at": "2026-10-01 09:18:41+00:00", "updated_at": "2026-10-01 09:45:44.895437+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-tools", "developer-tools", "machine-learning"], "entities": ["Laya", "laya-multilingual", "laya-ts", "Router", "PyTorch", "ONNX Runtime", "TileLang", "LangChain"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/laya-multilingual-non-autoregressive-system-1-decision-engine", "markdown": "https://wpnews.pro/news/laya-multilingual-non-autoregressive-system-1-decision-engine.md", "text": "https://wpnews.pro/news/laya-multilingual-non-autoregressive-system-1-decision-engine.txt", "jsonld": "https://wpnews.pro/news/laya-multilingual-non-autoregressive-system-1-decision-engine.jsonld"}}