{"slug": "not-another-llm-i-tried-laya-a-421m-parameter-ai-decision-engine", "title": "Not Another LLM: I Tried Laya, a 421M-Parameter AI Decision Engine", "summary": "A developer tested Laya, a 322M–421M parameter non-autoregressive decision model, on a Mac mini M4 with 16GB of memory, running it locally for English and Hindi inputs, multiple decisions, longer texts, and larger choice sets, and also built a simple Gradio interface. Unlike traditional LLMs, Laya does not generate text; it answers typed questions and returns structured decisions with probabilities, leading the developer to conclude it could serve as a lightweight decision layer alongside a larger LLM rather than replacing one.", "body_md": "I spend a lot of time trying different AI models, especially models that can run locally. Usually, the process is pretty familiar: download the model, give it a prompt, and wait for it to generate something.\n\nLaya caught my attention because it doesn't work that way.\n\nIt isn't built to write an answer, generate code, or have a conversation. Instead, you give it some text or a piece of information and ask it specific questions. It then returns a decision and a probability for that decision.\n\nThat sounded simple, but also quite different from the models I normally experiment with.\n\nSo I decided to install Laya on my Mac mini M4 with 16GB memory and see how it actually performs. I wanted to start from the basics, run it locally, try a few real examples, and understand where a small decision-focused model makes sense compared with using a much larger language model for everything.\n\nThis post is my hands-on experience with that process, from installation and the first prediction to multilingual input, model routing, and some of the limitations I found along the way.\n\nLaya is a 322M–421M parameter decision model that works very differently from a traditional LLM. It doesn't generate text; it takes an input, answers typed questions, and returns structured decisions with probabilities. I installed it on my M4 Mac mini, tested English, Hindi, multiple decisions, longer inputs, and larger choice sets, and also built a simple Gradio interface. The interesting part is that Laya could work as a lightweight decision layer alongside a larger LLM rather than replacing one.\n\nBefore installing it, it helps to understand what makes Laya different from the models I usually run locally.\n\nMost language models are built around generating text. You send them a prompt, they process it one token at a time, and eventually you get a response. That makes sense for chat, coding, writing, summarization, and many other tasks.\n\nLaya takes a different approach.\n\nIt is a non-autoregressive decision model. Instead of generating a response, it looks at the information you provide and answers a set of questions about it. It returns the answers in a structured format, along with probabilities.\n\nFor example, imagine I receive this support message:\n\n```\nHi, we were charged twice for March.\nPlease refund the duplicate payment today.\n```\n\nWith a normal LLM, I might ask:\n\n```\nWhich department should handle this request?\n```\n\nand get a natural-language response such as:\n\n```\nThis looks like a billing issue, so it should be handled\nby the billing department.\n```\n\nWith Laya, I can define the possible decisions beforehand:\n\n```\n{\n    \"billing\": \"invoices, payments, refunds\",\n    \"technical\": \"bugs, outages, system errors\",\n    \"other\": \"everything else\"\n}\n```\n\nLaya can then return the selected option and its probability rather than generating an explanation.\n\nThat's the part I found interesting.\n\nThere is no long response to parse, no generated paragraph sitting between the model and my application, and no need to ask the model to format its answer as JSON after the fact.\n\nIt's a small model too\n\nLaya currently comes with three checkpoints.\n\nThe main English model has 421 million parameters, while the multilingual checkpoint has 322 million parameters. There is also a 421M-parameter checkpoint fine-tuned for typed decision tasks.\n\nFor comparison, the local language models I normally experiment with are measured in billions of parameters. Laya is operating on a completely different scale.\n\nBut the parameter count isn't really the main story here.\n\nThe important thing is that Laya is designed for a much narrower job.\n\nIt isn't trying to write an email for me or build an application from a prompt. It is trying to answer questions such as:\n\n```\nWhich category is this?\n\nHow urgent is this?\n\nDoes the user want a refund?\n\nShould this request be routed to another system?\n```\n\nThat makes it feel less like a chatbot and more like a decision layer that can sit inside a larger application.\n\nAnd that was enough to make me want to try it locally.\n\nOnce I understood what Laya was actually built for, the next step was simply getting it running.\n\nI wanted to keep the setup clean, so I used a separate Python virtual environment instead of installing everything directly into my system Python.\n\nMy Mac mini has an M4 chip with 16GB of unified memory, so I wasn't expecting the model to be particularly demanding. The main Laya checkpoint is also only a few hundred million parameters, which makes it much smaller than the local language models I usually work with.\n\nI started by creating a new folder for the experiment:\n\n```\nmkdir laya-test\ncd laya-test\n```\n\nThen I created a virtual environment:\n\n```\npython3 -m venv .venv\nsource .venv/bin/activate\n```\n\nOnce the environment was active, my terminal showed (.venv) at the beginning of the command line.\n\nThe installation itself is surprisingly simple:\n\n```\npython -m pip install laya\n```\n\nThat's it.\n\nThere was no separate model download command at this stage. The package installs first, and the actual checkpoint is downloaded the first time I use it.\n\nBefore loading a model, I wanted to make sure the package was installed correctly:\n\n``` python\npython -c \"import laya; print(laya.__version__)\"\n```\n\nThis is a useful little check because it confirms that Python can find the package and tells me which version I'm actually running.\n\nA small thing worth knowing\n\nLaya doesn't download every checkpoint immediately.\n\nThe project has three checkpoints, but the one I request is what gets loaded. This is useful on a machine like mine because I don't need to spend storage space downloading models that I haven't decided to use yet.\n\nThe English checkpoint is around 843 MB, while the multilingual checkpoint is around 647 MB. That's tiny compared with many of the local models I've worked with.\n\nSo even with limited free space on my internal drive, Laya itself wasn't going to be the problem.\n\nNow for the interesting part\n\nThe package was installed, the environment was ready, and there was only one thing left to do:\n\nActually make Laya take a decision.\n\nNow that Laya is installed, I wanted to run a small example and see how it handles a real decision.\n\nInside the laya-test folder, I created a Python file:\n\n```\ntouch test_laya.py\n```\n\nThen I opened the file and added the following code:\n\n```\nnano test_laya.py\n```\n\nI added this code:\n\n``` python\nfrom laya import Router\n\nrouter = Router()\n\nstate = \"\"\"\nHi, we were billed twice for March.\nPlease refund the duplicate payment today.\n\"\"\"\n\nquestions = {\n    \"department\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which department should handle this?\",\n        \"criteria\": {\n            \"billing\": \"invoices, payments, refunds\",\n            \"technical\": \"bugs, outages, system errors\",\n            \"other\": \"everything else\"\n        }\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(result)\n```\n\nAfter adding the code, I saved the file and closed the editor.\n\nThe example is intentionally simple. I give Laya a short customer-support message and define three possible departments. Laya then has to decide which department is the right one.\n\nWith the file ready, I ran it from the same terminal:\n\n```\npython test_laya.py\n```\n\nOn the first run, Laya downloaded the required model checkpoint from Hugging Face. In my case, the download was around 846 MB.\n\nAfter the model finished loading, Laya processed the message and returned a structured result.\n\nThe important part of my output was:\n\n```\ndepartment → billing\n```\n\nIt also returned probabilities for all three choices, rather than generating a normal text response.\n\nThis was my first successful Laya inference running locally on my M4 Mac mini.\n\nThe first test was successful, but the output from Laya contains more information than just the final category.\n\nMy terminal returned a result similar to this:\n\n```\n'answers': {\n    'department': {\n        'type': 'choice',\n        'choice': 'billing',\n        'probabilities': {\n            'billing': 0.9875,\n            'technical': 0.0063,\n            'other': 0.0062\n        },\n        'confidence': 0.9309,\n        'answer_confidence': 0.9875\n    }\n}\n```\n\nThe first thing to look at is:\n\n```\nchoice: billing\n```\n\nThat's the actual decision Laya made.\n\nThen there are the probabilities:\n\n```\nbilling   → 0.9875\ntechnical → 0.0063\nother     → 0.0062\n```\n\nSo, for this particular input, the model strongly preferred the billing category.\n\nAnother interesting part of the result is:\n\n```\noutput_tokens: 0\n```\n\nThis is one of the easiest ways to see how Laya differs from a traditional text-generating model. It didn't generate a sentence explaining its answer. It evaluated the input and returned the structured result.\n\nThe response also included routing information:\n\n```\nmodel: english\nreason: English Latin text\n```\n\nThis means the built-in Router detected my input as English and selected the English checkpoint automatically.\n\nOne thing I noticed during this test was a warning about calibration. The runtime reported that one temperature value was outside its allowed range and was adjusted. Because of that, I won't treat the displayed probability as a guaranteed measure of real-world accuracy. It is better to validate and calibrate these probabilities on the data you're actually using.\n\nFor a first local test, though, the important part was clear: Laya was installed, the model loaded successfully, and it made a structured decision on my Mac mini.\n\nAfter the first test worked, I wanted to see what would happen if I asked Laya more than one question about the same message.\n\nThe nice thing here is that I don't have to send the same text to the model again and again. I can define several questions and pass them together.\n\nI kept the same customer-support example and added two more decisions: urgency and churn risk.\n\nI updated test_laya.py with this:\n\n``` python\nfrom laya import Router\n\nrouter = Router()\n\nstate = \"\"\"\nHi, we were billed twice for March.\nPlease refund the duplicate payment today\nor we will cancel our plan.\n\"\"\"\n\nquestions = {\n    \"department\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which department should handle this?\",\n        \"criteria\": {\n            \"billing\": \"invoices, payments, refunds\",\n            \"technical\": \"bugs, outages, system errors\",\n            \"other\": \"everything else\"\n        }\n    },\n\n    \"urgency\": {\n        \"type\": \"score\",\n        \"instructions\": \"How urgent is this?\",\n        \"criteria\": [\n            \"not urgent\",\n            \"soon\",\n            \"critical\"\n        ]\n    },\n\n    \"churn_risk\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does the user threaten to cancel or leave?\"\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(result)\n```\n\nThe input now contains three things at once:\n\n```\nDepartment\nUrgency\nChurn risk\n```\n\nSo instead of asking Laya:\n\n`Which department should handle this?`\n\nI'm also asking:\n\n`How urgent is this?`\n\nand:\n\n`Is the customer threatening to leave?`\n\nI saved the file and ran it again:\n\n```\npython test_laya.py\n```\n\nThis time, the result included answers for all three questions.\n\nThis time, Laya returned all three decisions from the same input.\n\nFor the department, it selected:\n\n```\nbilling\n```\n\nwith:\n\n```\nbilling   → 0.9801\ntechnical → 0.0113\nother     → 0.0086\n```\n\nThe model's reported confidence was 0.8989.\n\nFor urgency, the result was:\n\n```\nscore: 1.2514\n```\n\nThe three possible levels had these probabilities:\n\n```\nnot urgent → 0.1502\nsoon       → 0.4482\ncritical   → 0.4016\n```\n\nSo the model leaned toward “soon”, although the probabilities were much closer here than they were for the department decision.\n\nFinally, for churn risk, Laya returned:\n\n```\n0.8553\n```\n\nSo in this particular test, it assigned an 85.53% probability to the user threatening to cancel or leave.\n\nOne detail that stood out was:\n\n```\ninput_tokens: 179\noutput_tokens: 0\n```\n\nEven though I asked three different questions, there was still no generated text output.\n\nThat's probably the simplest way to explain what makes Laya interesting. I'm giving it one piece of information and several decisions to make, and it returns those decisions in a structured format.\n\nThe Router also reported:\n\n```\nmodel: english\nreason: English Latin text\n```\n\nSo the English checkpoint was selected automatically.\n\nOne thing I noticed\n\nThe runtime again showed the warning about an invalid temperature value and said the affected confidence should be treated as uncalibrated. So while these probabilities are useful for seeing what the model is doing, I wouldn't present them as a measure of actual accuracy.\n\nFor me, the more interesting result was the shape of the output: one input, three decisions, and zero generated tokens.\n\nAfter testing the English example, I wanted to see what happens when I give Laya a message in another language.\n\nI used a simple Hindi sentence:\n\n```\nमुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें।\n```\n\nThe meaning is roughly:\n\n`I was charged twice; please refund the money.`\n\nThis is where the Router becomes useful. I don't have to manually decide which checkpoint to load. Laya can inspect the input and choose the appropriate model.\n\nI updated my test_laya.py file and replaced the English state with the Hindi example:\n\n``` python\nfrom laya import Router\n\nrouter = Router()\n\nstate = \"\"\"\nमुझसे दो बार शुल्क लिया गया,\nकृपया पैसे वापस करें।\n\"\"\"\n\nquestions = {\n    \"department\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which department should handle this?\",\n        \"criteria\": {\n            \"billing\": \"invoices, payments, refunds\",\n            \"technical\": \"bugs, outages, system errors\",\n            \"other\": \"everything else\"\n        }\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(result)\n```\n\nI kept the same billing question and ran the file again:\n\n```\npython test_laya.py\n```\n\nThis time, Laya had to download its multilingual checkpoint. The download was around 653 MB, and the reconstructed checkpoint was about 678 MB.\n\nThe prediction itself was:\n\n```\ndepartment → billing\n```\n\nwith these probabilities:\n\n```\nbilling   → 0.9923\ntechnical → 0.0026\nother     → 0.0051\n```\n\nThe reported confidence was 0.9544.\n\nBut the part I was really interested in was the routing information.\n\nLaya returned:\n\n```\nmodel: multilingual\n```\n\nand gave the reason:\n\n```\nnon-Latin script (devanagari, 100% of letters);\nthe English checkpoint cannot read it\n```\n\nSo the Router recognized the script as Devanagari and automatically sent the request to the multilingual checkpoint.\n\nThe routing metadata also showed:\n\n```\nscript: devanagari\nlanguage: None\nis_english: False\n```\n\nThat last part is interesting. Laya didn't need to identify the exact language as Hindi first. The script itself was enough for the Router to decide that the English checkpoint shouldn't be used.\n\nThe complete flow looked like this:\n\n```\nHindi text\n    ↓\nRouter\n    ↓\nDevanagari detected\n    ↓\nMultilingual checkpoint\n    ↓\nBilling decision\n```\n\nAnd once again, the model reported:\n\n```\ninput_tokens: 55\noutput_tokens: 0\n```\n\nSo even with the multilingual model, there was still no text generation involved.\n\nThe multilingual checkpoint was a little smaller on disk than the English one in my tests:\n\nEnglish checkpoint      → ~846 MB\n\nMultilingual checkpoint → ~678 MB reconstructed\n\nThat made this test even more interesting for my setup, because both checkpoints are small enough to work comfortably on my Mac mini.\n\nThe Hindi test gave me the prediction I expected, but I was more curious about what happened before the prediction.\n\nLaya's Router doesn't just return the final answer. It also gives back information about the checkpoint it selected and why it selected it.\n\nI already had this in my result:\n\n```\nresult = router.predict(state, questions)\n```\n\nSo I added a couple of simple print statements to see the routing details more clearly:\n\n```\nprint(\"Selected model:\", result[\"routing\"][\"model\"])\nprint(\"Reason:\", result[\"routing\"][\"reason\"])\nprint(\"Detection:\", result[\"routing\"][\"detection\"])\n```\n\nFor my Hindi test, the output showed:\n\n```\nSelected model: multilingual\n\nReason:\nnon-Latin script (devanagari, 100% of letters);\nthe English checkpoint cannot read it\n```\n\nAnd the detection information looked like:\n\n```\nscript: devanagari\nlanguage: None\nis_english: False\n```\n\nThat was useful because it showed me that the Router didn't need to first figure out that the sentence was specifically Hindi. It could see that the text was written in Devanagari, which was enough to avoid sending it to the English checkpoint.\n\nI can also check the route separately\n\nAnother thing I found useful is that routing can be inspected without actually running the model.\n\nFor example:\n\n```\nroute = router.route(state, questions)\n\nprint(\"Model:\", route.model)\nprint(\"Reason:\", route.reason)\n```\n\nThis gives me the routing decision before inference happens.\n\nThat means the overall flow is roughly:\n\n```\nInput\n  ↓\nRouter\n  ↓\nDetect script / language\n  ↓\nSelect checkpoint\n  ↓\nRun inference\n  ↓\nReturn decision\n```\n\nI like this separation because it makes the behavior easier to understand when you're building a real application. The routing step isn't hidden inside some giant prompt. You can actually see which checkpoint was selected and the reason for it.\n\nEnglish and Hindi side by side\n\nAfter running both tests, I had a simple comparison:\n\n```\nEnglish\n  ↓\nEnglish / Laya checkpoint\n\nHindi\n  ↓\nDevanagari detected\n  ↓\nMultilingual checkpoint\n```\n\nSo far, the whole thing had been surprisingly lightweight.\n\nI had one model around 846 MB, another around 678 MB in my local test, and both were running directly on my Mac mini without needing a remote API.\n\nThe first few tests were intentionally simple. I wanted to make sure the installation worked, understand the output, and see how the Router handled different languages.\n\nNow I wanted to make the example a little closer to something I might actually use in an application.\n\nInstead of asking Laya only for the department, I decided to extract several useful decisions from the same support message.\n\nI used this example:\n\n```\nHi, we were billed twice for March.\nPlease refund the duplicate payment today\nor we will cancel our plan.\n```\n\nFrom that single message, I wanted Laya to determine:\n\nI updated test_laya.py with:\n\n``` python\nfrom laya import Router\n\nrouter = Router()\n\nstate = \"\"\"\nHi, we were billed twice for March.\nPlease refund the duplicate payment today\nor we will cancel our plan.\n\"\"\"\n\nquestions = {\n    \"department\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which department should handle this?\",\n        \"criteria\": {\n            \"billing\": \"invoices, payments, refunds\",\n            \"technical\": \"bugs, outages, system errors\",\n            \"other\": \"everything else\"\n        }\n    },\n\n    \"urgency\": {\n        \"type\": \"score\",\n        \"instructions\": \"How urgent is this?\",\n        \"criteria\": [\n            \"not urgent\",\n            \"soon\",\n            \"critical\"\n        ]\n    },\n\n    \"refund_requested\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does the user explicitly request a refund?\"\n    },\n\n    \"churn_risk\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does the user threaten to cancel or leave?\"\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(result)\n```\n\nI updated my test_laya.py file with the four questions and ran:\n\n```\npython test_laya.py\n```\n\nThe result was interesting.\n\n`Department`\n\nLaya selected:\n\n```\nbilling\n```\n\nThe probabilities were:\n\n```\nbilling   → 0.9801\ntechnical → 0.0113\nother     → 0.0086\n```\n\nSo the model clearly associated the message with a billing issue.\n\n`Urgency`\n\nFor urgency, I defined three levels:\n\n```\n0 → not urgent\n1 → soon\n2 → critical\nscore: 1.2514\nnot urgent → 0.1502\nsoon       → 0.4482\ncritical   → 0.4016\n```\n\nThe model leaned toward “soon”, although the probabilities were much closer together than they were for the department decision.\n\nThe reported confidence here was also quite low at 0.0799, which is a good reminder not to treat every model output as equally certain.\n\n`Refund requested`\n\nFor the refund question, Laya returned:\n\n```\n0.8822\n```\n\nSo it assigned an 88.22% probability to the user explicitly requesting a refund.\n\n`Churn risk`\n\nThe final question was whether the customer was threatening to leave.\n\n```\n0.8553\n```\n\nThat corresponds to an 85.53% probability for that decision.\n\nThe overall result can be summarized like this:\n\n```\n                    Customer message\n                           ↓\n                         Laya\n                           ↓\n        ┌──────────┬──────────┬───────────┬──────────┐\n        ↓          ↓          ↓           ↓\n     Billing     Urgency    Refund      Churn\n      98.01%      1.25      88.22%      85.53%\n```\n\nThere was another detail I found useful in the output:\n\n```\ninput_tokens: 239\noutput_tokens: 0\n```\n\nEven with four separate decisions, Laya still generated zero output tokens.\n\nThat is probably the clearest practical difference I've seen so far between Laya and a conventional text-generating model. I give it a piece of text, define the decisions I care about, and it returns structured results instead of generating a response.\n\nThe Router also continued to select the English checkpoint automatically:\n\n```\nmodel: english\nreason: English Latin text\n```\n\nAt this point, Laya was starting to look less like a small chatbot and more like a component I could put inside a larger application.\n\nThere is still one limitation worth keeping in mind. The runtime again warned about an out-of-range temperature value and said the affected confidence values should be treated as uncalibrated. So these numbers are useful for understanding this particular prediction, but I wouldn't use them as evidence that the model is 98% or 85% accurate in general.\n\nFor a local experiment, though, this was a useful result: one input, four decisions, and no generated text.\n\nSo far, most of my tests used short messages. That's useful for understanding the basics, but real applications rarely receive just two or three lines of text.\n\nA support ticket can contain the original complaint, previous replies, order details, and a few extra paragraphs. So my next test was to give Laya a longer piece of text and see how it handles it.\n\nFor this test, I kept the same questions but replaced the short message with a more detailed customer request.\n\nI updated test_laya.py like this:\n\n``` python\nfrom laya import Router\n\nrouter = Router()\n\nstate = \"\"\"\nHi support team,\n\nI am contacting you because I noticed that my March subscription\npayment was charged twice on my account. I checked my bank statement\nand can see two separate charges for the same amount on the same day.\n\nI have already checked my account dashboard, but I can only see one\ninvoice there. I would like the duplicate payment to be refunded as\nsoon as possible.\n\nThis has been quite frustrating because I need the money back before\nthe end of the week. If this issue cannot be resolved soon, I may have\nto cancel my subscription and move to another service.\n\nPlease check the payment history, confirm what happened, and let me\nknow when the duplicate amount will be returned.\n\"\"\"\n\nquestions = {\n    \"department\": {\n        \"type\": \"choice\",\n        \"instructions\": \"Which department should handle this?\",\n        \"criteria\": {\n            \"billing\": \"invoices, payments, refunds\",\n            \"technical\": \"bugs, outages, system errors\",\n            \"other\": \"everything else\"\n        }\n    },\n\n    \"urgency\": {\n        \"type\": \"score\",\n        \"instructions\": \"How urgent is this?\",\n        \"criteria\": [\n            \"not urgent\",\n            \"soon\",\n            \"critical\"\n        ]\n    },\n\n    \"refund_requested\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does the user explicitly request a refund?\"\n    },\n\n    \"churn_risk\": {\n        \"type\": \"noul\",\n        \"instructions\": \"Does the user threaten to cancel or leave?\"\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(result)\n```\n\nThen I ran the same file again:\n\n```\npython test_laya.py\n```\n\nThis time, the model processed 751 input tokens and still returned:\n\n```\noutput_tokens: 0\n```\n\nThat part stayed the same. Even with a much longer input, Laya wasn't generating a written response.\n\nThe department result was:\n\n```\nbilling → 0.9115\ntechnical → 0.0524\nother → 0.0361\n```\n\nSo Laya still identified the request as a billing issue, although the probability was lower than in my shorter test.\n\nFor urgency, it returned:\n\n```\nnot urgent → 0.2238\nsoon       → 0.7585\ncritical   → 0.0177\n```\n\nThe resulting score was:\n\n```\n0.7939\n```\n\nSo this time the model clearly leaned toward “soon.”\n\nThe refund question returned:\n\n```\n0.8231\n```\n\nwhich means the model assigned an 82.31% probability to the user explicitly requesting a refund.\n\nThen I got one result that I found quite interesting.\n\nThe churn-risk question returned:\n\n```\n0.1494\n```\n\nSo the model assigned only 14.94% probability to the user threatening to cancel, even though the text explicitly included:\n\n`“If this issue cannot be resolved soon, I may have to cancel my subscription.”`\n\nThis was a useful reminder that I shouldn't assume the model will interpret every question exactly the way I expect.\n\nThe Laya documentation also points out that its noul decision can sometimes be affected by the labels used for the two possible outcomes. So this result is something I would want to investigate rather than simply accepting it as a correct or incorrect verdict.\n\nThe longer input changed the results\n\nComparing this with my earlier short test was also interesting.\n\nThe shorter version gave me:\n\n```\nbilling → 98.01%\n```\n\nwhile the longer version gave:\n\n```\nbilling → 91.15%\n```\n\nThe model still selected the same category, but the probability changed once more context was added.\n\nThat's actually something I wanted to see. A longer input isn't automatically going to produce the same confidence as a short one, even when the underlying topic hasn't changed.\n\nFor this particular test, the 751-token input was still handled without any problem by the English checkpoint.\n\nAt this point, I had tested short inputs, multiple questions, different languages, and a longer support ticket.\n\nSo far, I had been giving Laya only a few possible answers.\n\nThat works nicely for a simple example like:\n\n```\nbilling\ntechnical\nother\n```\n\nBut real classification systems can have many more categories. A support system, for example, might have separate labels for refunds, card problems, failed transfers, account access, delivery issues, verification, and many other cases.\n\nSo I wanted to see how Laya behaves when the number of choices starts increasing.\n\nInstead of changing the whole project, I kept the same setup and created a larger choice question.\n\nI used this:\n\n``` python\nfrom laya import Router\n\nrouter = Router()\n\nstate = \"\"\"\nI tried to transfer money to my bank account this morning.\nThe transfer failed, but I can see the amount has already\nbeen deducted from my account. I want to know when the money\nwill be returned.\n\"\"\"\n\nquestions = {\n    \"intent\": {\n        \"type\": \"choice\",\n        \"instructions\": \"What is the main reason for this support request?\",\n        \"criteria\": {\n            \"card_payment\": \"problem with a card payment\",\n            \"cash_withdrawal\": \"problem withdrawing cash\",\n            \"bank_transfer\": \"problem with a bank transfer\",\n            \"refund\": \"asking for a refund\",\n            \"account_access\": \"cannot access the account\",\n            \"verification\": \"identity or account verification problem\",\n            \"subscription\": \"subscription or recurring payment problem\",\n            \"fees\": \"question about fees or charges\",\n            \"exchange_rate\": \"question about currency exchange rates\",\n            \"card_delivery\": \"problem with card delivery\",\n            \"cash_deposit\": \"problem depositing cash\",\n            \"other\": \"something not covered by the other categories\"\n        }\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(result)\n```\n\nThe input was:\n\n```\nI tried to transfer money to my bank account this morning.\nThe transfer failed, but I can see the amount has already\nbeen deducted from my account. I want to know when the money\nwill be returned.\n```\n\nI then asked Laya to choose the main reason for the request from these categories:\n\n```\ncard_payment\ncash_withdrawal\nbank_transfer\nrefund\naccount_access\nverification\nsubscription\nfees\nexchange_rate\ncard_delivery\ncash_deposit\nother\n```\n\nI ran the same command as before:\n\n```\npython test_laya.py\nbank_transfer\n```\n\nThe result was extremely strong:\n\n```\nbank_transfer → 1.0\n```\n\nThe other 11 categories were returned as 0.0 in this particular run.\n\nThe reported confidence was:\n\n```\nconfidence: 0.9998\nanswer_confidence: 1.0\n```\n\nIt also processed:\n\n```\ninput_tokens: 174\noutput_tokens: 0\n```\n\nSo once again, there was no generated text.\n\nI expected adding more options to make the decision noticeably harder, but with 12 choices, Laya still identified the correct category very clearly.\n\nThat said, there is an important detail here.\n\nThe runtime again showed the warning about an invalid temperature value and said that the affected confidence values were uncalibrated. Because of that, I wouldn't interpret the 1.0 probability as meaning that the model is literally certain or that it will be correct 100% of the time.\n\nWhat I can say from my test is much simpler:\n\n`With this particular 12-option example, Laya selected bank_transfer and assigned virtually all of the probability mass to that category.`\n\nThis test also showed me why the number of choices matters.\n\nLaya has a fixed budget for representing the options in a question. With 12 categories, there is still enough room for the descriptions to remain distinct.\n\nBut what happens when the number jumps from 12 choices to dozens of choices?\n\nThe project's own benchmarks show that high-cardinality questions can become a problem. The documentation specifically discusses a 77-option Banking77 test, where the available option budget becomes much tighter.\n\nSo my next experiment is going to push this much further.\n\nAfter my 12-choice test worked, I wanted to push the model much further.\n\nThe Laya documentation talks about an important limitation with high-cardinality choice questions. When a question has a large number of options, the model has to fit all of those option descriptions into a fixed part of its input budget. With 50 or more choices, this can make the options harder to distinguish. The project's benchmark specifically uses Banking77, a dataset with 77 banking intents, to demonstrate this limitation.\n\nSo I decided to recreate a similar test locally.\n\nSetting up the 77 choices\n\nI used the 77 Banking77 intent names as the choices. Banking77 is a fine-grained banking intent dataset containing 77 categories, including things such as card_arrival, failed_transfer, request_refund, transaction_charged_twice, and verify_my_identity.\n\nFor the experiment, I used a straightforward banking query:\n\n```\nI tried to make a bank transfer this morning.\nThe transfer failed, but the money has already\nbeen deducted from my account.\n```\n\nI then created a choice question containing all 77 categories.\n\nThe code looks like this:\n\n``` python\nfrom laya import Router\n\nrouter = Router()\n\nstate = \"\"\"\nI tried to make a bank transfer this morning.\nThe transfer failed, but the money has already\nbeen deducted from my account.\n\"\"\"\n\nintents = {\n    \"activate_my_card\": \"activating a card\",\n    \"age_limit\": \"age requirements\",\n    \"apple_pay_or_google_pay\": \"using Apple Pay or Google Pay\",\n    \"atm_support\": \"ATM support\",\n    \"automatic_top_up\": \"automatic top up\",\n    \"balance_not_updated_after_bank_transfer\": \"balance not updated after bank transfer\",\n    \"balance_not_updated_after_cheque_or_cash_deposit\": \"balance not updated after cheque or cash deposit\",\n    \"beneficiary_not_allowed\": \"beneficiary is not allowed\",\n    \"cancel_transfer\": \"cancelling a transfer\",\n    \"card_about_to_expire\": \"card is about to expire\",\n    \"card_acceptance\": \"where a card is accepted\",\n    \"card_arrival\": \"card arrival\",\n    \"card_delivery_estimate\": \"estimated card delivery time\",\n    \"card_linking\": \"linking a card\",\n    \"card_not_working\": \"card not working\",\n    \"card_payment_fee_charged\": \"fee charged for a card payment\",\n    \"card_payment_not_recognised\": \"unrecognised card payment\",\n    \"card_payment_wrong_exchange_rate\": \"wrong exchange rate for card payment\",\n    \"card_swallowed\": \"card retained by an ATM\",\n    \"cash_withdrawal_charge\": \"cash withdrawal charge\",\n    \"cash_withdrawal_not_recognised\": \"unrecognised cash withdrawal\",\n    \"change_pin\": \"changing PIN\",\n    \"compromised_card\": \"compromised card\",\n    \"contactless_not_working\": \"contactless not working\",\n    \"country_support\": \"supported countries\",\n    \"declined_card_payment\": \"declined card payment\",\n    \"declined_cash_withdrawal\": \"declined cash withdrawal\",\n    \"declined_transfer\": \"declined transfer\",\n    \"direct_debit_payment_not_recognised\": \"unrecognised direct debit\",\n    \"disposable_card_limits\": \"disposable card limits\",\n    \"edit_personal_details\": \"editing personal details\",\n    \"exchange_charge\": \"currency exchange charge\",\n    \"exchange_rate\": \"exchange rate\",\n    \"exchange_via_app\": \"exchanging currency through the app\",\n    \"extra_charge_on_statement\": \"extra charge on statement\",\n    \"failed_transfer\": \"failed bank transfer\",\n    \"fiat_currency_support\": \"supported fiat currencies\",\n    \"get_disposable_virtual_card\": \"getting a disposable virtual card\",\n    \"get_physical_card\": \"getting a physical card\",\n    \"getting_spare_card\": \"getting a spare card\",\n    \"getting_virtual_card\": \"getting a virtual card\",\n    \"lost_or_stolen_card\": \"lost or stolen card\",\n    \"lost_or_stolen_phone\": \"lost or stolen phone\",\n    \"order_physical_card\": \"ordering a physical card\",\n    \"passcode_forgotten\": \"forgotten passcode\",\n    \"pending_card_payment\": \"pending card payment\",\n    \"pending_cash_withdrawal\": \"pending cash withdrawal\",\n    \"pending_top_up\": \"pending top up\",\n    \"pending_transfer\": \"pending transfer\",\n    \"pin_blocked\": \"blocked PIN\",\n    \"receiving_money\": \"receiving money\",\n    \"refund_not_showing_up\": \"refund not showing up\",\n    \"request_refund\": \"requesting a refund\",\n    \"reverted_card_payment\": \"reverted card payment\",\n    \"supported_cards_and_currencies\": \"supported cards and currencies\",\n    \"terminate_account\": \"closing the account\",\n    \"top_up_by_bank_transfer_charge\": \"charge for topping up by bank transfer\",\n    \"top_up_by_card_charge\": \"charge for topping up by card\",\n    \"top_up_by_cash_or_cheque\": \"topping up by cash or cheque\",\n    \"top_up_failed\": \"failed top up\",\n    \"top_up_limits\": \"top up limits\",\n    \"top_up_reverted\": \"reverted top up\",\n    \"topping_up_by_card\": \"topping up by card\",\n    \"transaction_charged_twice\": \"transaction charged twice\",\n    \"transfer_fee_charged\": \"fee charged for a transfer\",\n    \"transfer_into_account\": \"transferring money into the account\",\n    \"transfer_not_received_by_recipient\": \"recipient did not receive the transfer\",\n    \"transfer_timing\": \"transfer timing\",\n    \"unable_to_verify_identity\": \"unable to verify identity\",\n    \"verify_my_identity\": \"identity verification\",\n    \"verify_source_of_funds\": \"verifying source of funds\",\n    \"verify_top_up\": \"top up verification\",\n    \"virtual_card_not_working\": \"virtual card not working\",\n    \"visa_or_mastercard\": \"Visa or Mastercard\",\n    \"why_verify_identity\": \"why identity verification is required\",\n    \"wrong_amount_of_cash_received\": \"wrong amount of cash received\",\n    \"wrong_exchange_rate_for_cash_withdrawal\": \"wrong exchange rate for cash withdrawal\"\n}\n\nquestions = {\n    \"intent\": {\n        \"type\": \"choice\",\n        \"instructions\": \"What is the main reason for this banking request?\",\n        \"criteria\": intents\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(result)\n```\n\nThis is very different from my earlier 12-choice experiment.\n\nWith 77 options, the model has to distinguish between many closely related banking intents. For example:\n\n```\nfailed_transfer\ndeclined_transfer\npending_transfer\ntransfer_not_received_by_recipient\ncancel_transfer\ntransfer_timing\n```\n\nThese categories are much closer to each other than something obvious like card_arrival versus exchange_rate.\n\nThe Laya documentation says this is where its default option budget becomes a real constraint. In the documented Banking77 comparison, Laya's 77-choice result was substantially lower than its performance on smaller choice sets, and the authors explain that the option descriptions can become too compressed within the fixed head budget.\n\n```\npython test_laya.py\n```\n\nThe result surprised me\n\n```\nfailed_transfer\n```\n\nThat was the category I would expect from this particular message.\n\n```\nfailed_transfer   → 0.9753\ndeclined_transfer → 0.0056\nunable_to_verify_identity → 0.0191\n```\n\nMost of the remaining categories were returned as 0.0 in this particular run.\n\nThe reported values were:\n\n```\nconfidence: 0.9703\nanswer_confidence: 0.9753\n```\n\nThe input contained 350 tokens, while the output was again:\n\n```\noutput_tokens: 0\n```\n\nSo even with 77 possible choices, Laya still didn't generate any text.\n\nI expected the larger choice set to make the result obviously worse, but that didn't happen with this particular example.\n\nLaya still selected the correct-looking category and put most of the probability on it.\n\nHowever, this doesn't mean that Laya solves 77-choice classification reliably in general.\n\nThe reason is that this was one manually constructed test. The official project benchmarks report a much more difficult Banking77 evaluation where the model's performance drops substantially with a large number of options. The project explains that the option descriptions have to share a limited token budget, which becomes increasingly restrictive as the number of choices grows.\n\nSo my local experiment gave me one useful result:\n\n`77 choices did not break the model on this example.`\n\nBut it is not enough to claim that Laya performs well across a full 77-class benchmark.\n\nThat's an important distinction, especially when writing about model experiments.\n\nThe runtime continued to display the same temperature warning:\n\n```\nthis checkpoint ships invalid temperatures...\nTreat confidence from the affected entries as uncalibrated.\n```\n\nSo again, I'm treating the probabilities as the output of this particular run, not as a guarantee of real-world accuracy.\n\nAt this point, I had a much better idea of how Laya behaves with different input sizes and different numbers of choices.\n\nUp to this point, I had been using the default Router:\n\n```\nrouter = Router()\n```\n\nWith that setup, Laya loads a checkpoint when it is needed. That works well when you're experimenting because you don't have to load every model into memory at startup.\n\nFor the next test, I wanted to try a different approach.\n\nLaya also supports preloading, which means the checkpoints are loaded when the Router starts.\n\nI changed my code to:\n\n``` python\nfrom laya import Router\n\nrouter = Router(preload=True)\n\nstate = \"\"\"\nI tried to make a bank transfer this morning.\nThe transfer failed, but the money has already\nbeen deducted from my account.\n\"\"\"\n\nquestions = {\n    \"intent\": {\n        \"type\": \"choice\",\n        \"instructions\": \"What is the main reason for this banking request?\",\n        \"criteria\": {\n            \"card_payment\": \"problem with a card payment\",\n            \"cash_withdrawal\": \"problem withdrawing cash\",\n            \"bank_transfer\": \"problem with a bank transfer\",\n            \"refund\": \"asking for a refund\",\n            \"account_access\": \"cannot access the account\",\n            \"verification\": \"identity or account verification problem\",\n            \"subscription\": \"subscription or recurring payment problem\",\n            \"fees\": \"question about fees or charges\",\n            \"exchange_rate\": \"question about currency exchange rates\",\n            \"card_delivery\": \"problem with card delivery\",\n            \"cash_deposit\": \"problem depositing cash\",\n            \"other\": \"something not covered by the other categories\"\n        }\n    }\n}\n\nresult = router.predict(state, questions)\n\nprint(\"Selected intent:\", result[\"answers\"][\"intent\"][\"choice\"])\nprint(\"Probabilities:\", result[\"answers\"][\"intent\"][\"probabilities\"])\nprint(\"Confidence:\", result[\"answers\"][\"intent\"][\"confidence\"])\nprint(\"Routing model:\", result[\"routing\"][\"model\"])\nprint(\"Routing reason:\", result[\"routing\"][\"reason\"])\nprint(\"Input tokens:\", result[\"usage\"][\"input_tokens\"])\nprint(\"Output tokens:\", result[\"usage\"][\"output_tokens\"])\n```\n\nI then ran:\n\n```\npython test_laya.py\n```\n\nThe result\n\nThe prediction was:\n\n```\nSelected intent: failed_transfer\n```\n\nLaya assigned:\n\n```\nfailed_transfer → 0.9753\n```\n\nThe other notable probabilities were:\n\n```\ndeclined_transfer          → 0.0056\nunable_to_verify_identity  → 0.0191\n0.9703\n```\n\nThe request contained 350 input tokens, while the output was again:\n\n```\nOutput tokens: 0\n```\n\nThe Router also selected:\n\n```\nRouting model: english\n```\n\nwith the reason:\n\n```\nEnglish Latin text\n```\n\nThis time, the terminal didn't download the model again.\n\nThe fetching step showed:\n\n```\n5/5\n00:00\n```\n\nand the download/reconstruction both reported 0.00B.\n\nThat makes sense because the checkpoint had already been downloaded during my earlier experiments. Laya was using the cached files instead of downloading another copy.\n\nThe important thing to understand is that preload=True doesn't change the kind of answer Laya produces. It changes how the model is prepared for inference.\n\nWith the normal setup:\n\n```\nRequest\n   ↓\nLoad checkpoint if needed\n   ↓\nPrediction\n```\n\nWith preload enabled:\n\n```\nApplication starts\n       ↓\nCheckpoint(s) loaded\n       ↓\nRequest arrives\n       ↓\nPrediction\n```\n\nThat makes more sense for something that is going to stay running, such as a local API, a support-ticket classifier, or another application making repeated decisions.\n\nThere is a trade-off, though. Keeping models ready means using more memory. So preload mode is more useful when the machine has enough RAM to keep the required checkpoints resident.\n\nFor my Mac mini, the interesting thing was that Laya was already operating in a completely different memory class from the large language models I normally experiment with.\n\nI didn't measure the exact RAM usage in this run, so I'm not going to make up a number. What I can confirm is that the preloaded setup ran successfully on my M4 Mac mini with 16GB unified memory.\n\nAnd at this point, I had tested Laya with short text, long text, multiple decisions, Hindi input, and a wide choice set.\n\nAfter running several different tests, I wanted to check something more practical.\n\nI already knew the model files were relatively small, but I wanted to see what they actually looked like on my Mac rather than relying only on the numbers from the model page.\n\nThere are two different things to measure here:\n\nStorage — how much space the downloaded checkpoints occupy on the SSD.\n\nMemory — how much RAM the Python process needs while Laya is running.\n\nThese are easy to confuse, because an 800 MB model file does not necessarily mean the application will use exactly 800 MB of memory.\n\nHugging Face keeps downloaded model files in its local cache. I checked the cache with:\n\n```\ndu -sh ~/.cache/huggingface/hub\n```\n\nTo look specifically for Laya models:\n\n```\ndu -sh ~/.cache/huggingface/hub/models--convaiinnovations--*\n```\n\nDepending on how the cache is configured on the Mac, the files may be stored in a different location. You can check that with:\n\n```\necho $HF_HOME\n```\n\nIf nothing is printed, the standard Hugging Face cache location is normally used.\n\nFor the memory side, I used macOS's built-in /usr/bin/time utility.\n\nInstead of:\n\n```\npython test_laya.py\n```\n\nI ran:\n\n```\n/usr/bin/time -l python test_laya.py\n```\n\nAt the end of the output, macOS reports:\n\n```\nmaximum resident set size\n```\n\nThat gives me the peak memory used by the Python process during the run.\n\nThis is a much more useful number for my blog than simply saying:\n\n`“The model is 846 MB.”`\n\nThe model file and the actual runtime memory are two different measurements.\n\nAfter testing Laya from the terminal, I wanted to make the experiment a little more practical.\n\nRunning a Python script every time works, but I wanted a small interface where I could paste a customer message, click a button, and immediately see what Laya thinks about it.\n\nSo I added Gradio on top of the same Laya code.\n\nThe flow now looks like this:\n\n```\nBrowser\n   ↓\nGradio\n   ↓\nLaya Router\n   ↓\nStructured decisions\n```\n\nI kept the interface simple. I wanted to see the department, refund request, churn risk, routing information, and token usage without having to read a large JSON response every time.\n\nGradio is a simple way to do that.\n\nInside the same virtual environment:\n\n```\npython -m pip install gradio\n```\n\nCreate another file:\n\n```\ntouch app.py\n```\n\nThen open it:\n\n```\nnano app.py\n```\n\nAdd this:\n\n``` python\nimport gradio as gr\nfrom laya import Router\n\n# Load Laya once when the app starts\nrouter = Router(preload=True)\n\ndef predict(text):\n    if not text.strip():\n        return {\n            \"error\": \"Please enter some text.\"\n        }\n\n    questions = {\n        \"department\": {\n            \"type\": \"choice\",\n            \"instructions\": \"Which department should handle this?\",\n            \"criteria\": {\n                \"billing\": \"invoices, payments, refunds\",\n                \"technical\": \"bugs, outages, system errors\",\n                \"other\": \"everything else\"\n            }\n        },\n        \"refund_requested\": {\n            \"type\": \"noul\",\n            \"instructions\": \"Does the user explicitly request a refund?\"\n        },\n        \"churn_risk\": {\n            \"type\": \"noul\",\n            \"instructions\": \"Does the user threaten to cancel or leave?\"\n        }\n    }\n\n    result = router.predict(text, questions)\n\n    return {\n        \"department\": result[\"answers\"][\"department\"],\n        \"refund_requested\": result[\"answers\"][\"refund_requested\"],\n        \"churn_risk\": result[\"answers\"][\"churn_risk\"],\n        \"routing\": result[\"routing\"],\n        \"usage\": result[\"usage\"]\n    }\n\nwith gr.Blocks(title=\"Laya Decision Demo\") as demo:\n    gr.Markdown(\"# Laya Decision Demo\")\n    gr.Markdown(\n        \"Enter a message and let Laya make structured decisions.\"\n    )\n\n    text_input = gr.Textbox(\n        label=\"Input\",\n        placeholder=\"Example: I was charged twice and want a refund.\",\n        lines=6\n    )\n\n    run_button = gr.Button(\"Run Laya\")\n\n    output = gr.JSON(label=\"Laya Result\")\n\n    run_button.click(\n        fn=predict,\n        inputs=text_input,\n        outputs=output\n    )\n\nif __name__ == \"__main__\":\n    demo.launch()\n```\n\nNow start it with:\n\n```\npython app.py\n```\n\nGradio will start a local web server and give you a local URL, typically something like:\n\n```\nhttp://127.0.0.1:7860\n```\n\nOpen that URL in your browser.\n\nYou'll get a simple interface where you can enter something like:\n\n```\nHi, I was charged twice for my March subscription.\nPlease refund the duplicate payment.\n```\n\nand click:\n\nRun Laya\n\nThe UI will then show the structured result returned by Laya.\n\nFor one of my tests, I entered a customer-support message and got this result:\n\n```\nDepartment: billing\nRefund requested: 0.8829\nChurn risk: 0.0916\nModel: english\nInput tokens: 161\nOutput tokens: 0\n```\n\nThe department probabilities were:\n\n```\nbilling   → 0.9637\ntechnical → 0.0185\nother     → 0.0178\n```\n\nThe refund decision returned:\n\n```\n0.8829\n```\n\nAnd the churn-risk value was:\n\n```\n0.0916\n```\n\nOne thing I found particularly interesting was the token count:\n\n```\ninput_tokens: 161\noutput_tokens: 0\n```\n\nAgain, Laya wasn't generating a paragraph in response to my input. The interface was simply displaying the structured decisions produced by the model.\n\nThe Router also identified the input as English:\n\n```\nmodel: english\nreason: English Latin text\n```\n\nAfter playing with Laya locally, I wanted to look beyond my own small experiments and see how the different checkpoints perform on larger benchmark runs.\n\nThe project currently has three checkpoints:\n\n```\n| Checkpoint               | Parameters |            Context | Main use                 |\n| ------------------------ | ---------: | -----------------: | ------------------------ |\n| `convaiinnovations/laya` |       421M |                512 | English                  |\n| `laya-multilingual`      |       322M | 1,024, up to 8,192 | 100+ languages           |\n| `laya-typed-decisions`   |       421M |              1,024 | Typed decision workflows |\n```\n\nThe first thing that stands out is how small these models are.\n\nThe main Laya checkpoint is only 421 million parameters, and the multilingual version is even smaller at 322 million. That's a very different scale from the multi-billion-parameter models I normally think about when talking about local AI.\n\nThe project's benchmark runs on a Tesla T4 show a few interesting numbers.\n\nFor typed decisions, the fine-tuned laya-typed-decisions checkpoint reached 0.766 accuracy across 2,000 decisions.\n\nThe base English and multilingual checkpoints were much lower on this particular benchmark:\n\n```\nlaya                  → 0.362\nlaya-multilingual     → 0.342\nlaya-typed-decisions  → 0.766\n```\n\nThat difference is important.\n\nIt tells me that the impressive 0.766 number shouldn't be interpreted as what the base Laya model does out of the box. The result comes from a checkpoint that was specifically fine-tuned for those typed-decision workflows.\n\nThat's actually one of the more useful lessons from the benchmarks: fine-tuning matters a lot for this kind of model.\n\nThe speed numbers are also interesting.\n\nOn a Tesla T4, the project reports:\n\n```\n| Questions in one call |     Laya | Multilingual |\n| --------------------: | -------: | -----------: |\n|                     1 |  39.5 ms |      32.8 ms |\n|                     5 |  84.5 ms |      40.1 ms |\n|                    10 | 158.6 ms |      72.3 ms |\n|                    50 |   771 ms |       337 ms |\n```\n\nBecause multiple questions can be answered in one forward pass, the cost per question drops as the batch gets larger.\n\nFor example, the multilingual checkpoint reaches about 7.2 ms per question when 10 questions are processed together.\n\nThat's one of the places where the smaller decision-oriented architecture starts to make sense.\n\n`Laya benchmark results across accuracy, latency, calibration, multilingual performance, and application workflows.`\n\nOne of the things I was particularly interested in was how Laya behaves when the input gets much longer.\n\nThe multilingual checkpoint can be configured with:\n\n```\nmax_len=8192\n```\n\nThe project's long-context experiment placed the request at the end of documents with different amounts of unrelated text before it.\n\nThe results were:\n\n```\n| Text before request | Correct | Time per request |\n| ------------------- | ------: | ---------------: |\n| Short input         | 19 / 20 |           0.02 s |\n| ~1,000 tokens       | 16 / 20 |           0.21 s |\n| ~2,000 tokens       | 17 / 20 |           0.50 s |\n| ~3,000 tokens       | 17 / 20 |           0.90 s |\n| ~4,000 tokens       | 18 / 20 |           1.71 s |\n| ~5,000 tokens       | 11 / 20 |           2.59 s |\n| ~6,000 tokens       | 17 / 20 |           3.50 s |\n| ~7,000 tokens       |  8 / 20 |           4.50 s |\n```\n\nThe results aren't perfectly smooth, but they show the basic trade-off clearly: longer context costs more time, and accuracy can become more variable as the document gets very long.\n\nThat matches what I saw in my own test earlier. My 751-token support ticket worked without any issue, but that doesn't mean I should throw an enormous document at the model and expect the same behavior.\n\nFor a real application, I would test the exact input lengths and document types I care about.\n\n`Long-context test for laya-multilingual with requests placed at different positions inside longer documents.`\n\nAfter all the experiments, I think the most useful way to look at Laya is not as a replacement for a large language model.\n\nIt's a different tool.\n\nThe project reports strong results on some tasks. For example, its benchmark suite reports around 0.993 accuracy for email spam and phishing on the tested data, while several other workflows are considerably weaker.\n\nThe same benchmark also shows an important limitation with large choice sets.\n\nOn Banking77, which has 77 labels, the benchmark reports:\n\n```\nLaya → 0.425\nTypeSafe Jev → 0.870\n```\n\nThe project attributes Laya's result to the way the available token budget is shared across the many option descriptions.\n\nThat matches something I started noticing during my own experiments: three or twelve choices are very different from seventy-seven choices.\n\nSo I wouldn't design a system around hundreds of labels in one giant choice question without testing it carefully.\n\nThere are a few other things I would keep in mind too.\n\nThe base checkpoints are not automatically excellent at every zero-shot decision task. The model's probability outputs can also require calibration, and some decision types perform better than others.\n\nFor me, that's actually a positive part of the project rather than something to hide.\n\nThe documentation is fairly open about where the model works and where it needs more care.\n\nAfter running all these tests, this is the comparison that makes the most sense to me.\n\nA traditional LLM generally looks like:\n\n```\nInput\n  ↓\nLarge language model\n  ↓\nGenerate tokens\n  ↓\nText response\n```\n\nLaya looks more like:\n\n```\nInput\n  ↓\nLaya\n  ↓\nTyped decision\n  ↓\nProbability\n```\n\nThat's a much narrower job.\n\nI wouldn't use Laya to write a blog post, generate code, summarize a long meeting, or have a conversation.\n\nBut I could imagine using something like Laya for the smaller decisions that happen around those tasks.\n\n```\nUser request\n      ↓\n     Laya\n      ↓\n  Is this relevant?\n      ↓\n  Is it urgent?\n      ↓\n Which workflow?\n      ↓\n    LLM / Tool\n```\n\nIn other words, the interesting idea isn't:\n\n`“Laya can replace my LLM.”`\n\nFor me, it's:\n\n`“Maybe my LLM doesn't need to make every decision.”`\n\nI started this experiment because Laya looked unusual.\n\nMost of the local models I come across are trying to become better at generation: better coding, better reasoning, better conversations, better multimodal capabilities.\n\nLaya went in a different direction.\n\nIt takes a relatively small model, gives it a structured decision problem, and focuses on making that decision quickly.\n\nI tested it on my M4 Mac mini, ran the English checkpoint, switched to the multilingual model with Hindi input, asked multiple questions against the same text, tried a longer support ticket, increased the number of choices, and finally put the whole thing behind a small Gradio interface.\n\nWhat surprised me most was how little infrastructure was needed to get from zero to a working local demo.\n\nI didn't need a giant GPU server.\n\nI didn't need an API key for inference.\n\nI didn't need to run a multi-billion-parameter model.\n\nI just installed the package, downloaded the checkpoint, and started asking it structured questions.\n\nMaybe Every AI System Doesn't Need One Giant Model\n\nAfter spending so much time experimenting with large language models, I found Laya refreshing simply because it isn't trying to be one.\n\nIt doesn't generate a long answer.\n\nIt doesn't try to write code.\n\nIt doesn't try to become a chatbot.\n\nIt makes decisions.\n\nRunning a 421M-parameter model locally on my M4 Mac mini made that idea feel much more practical to me.\n\nFor something like customer-support routing, moderation, classification, guardrails, or other structured decisions, using a smaller specialized model can make more sense than sending every single request through a large generative model.\n\nAt the same time, my experiments also showed that Laya isn't magic. Its performance depends heavily on the task, the number of choices, the input size, and whether the model has been fine-tuned for the problem. The confidence values also need to be treated carefully rather than taken as guaranteed accuracy.\n\nSo I wouldn't look at Laya and ask:\n\n```\n“Is this the next replacement for Llama or Qwen?”\n```\n\nThat's the wrong comparison.\n\nThe more interesting question for me is:\n\n```\nWhat if we stop expecting one model to do everything?\n```\n\nA large language model can handle generation and reasoning.\n\nA small decision model can handle a narrow decision.\n\nA routing model can decide which path to take.\n\nAnd an application can combine all of them.\n\nThat feels like a much more interesting direction for local AI.\n\nLaya may be small, but the idea behind it is much bigger: sometimes the best model for a job isn't the model that can do everything — it's the one designed to do that one job well.", "url": "https://wpnews.pro/news/not-another-llm-i-tried-laya-a-421m-parameter-ai-decision-engine", "canonical_source": "https://dev.to/ayush7614/not-another-llm-i-tried-laya-a-421m-parameter-ai-decision-engine-6af", "published_at": "2026-09-25 11:42:47+00:00", "updated_at": "2026-09-25 12:01:21.608952+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools"], "entities": ["Laya", "Mac mini M4", "Gradio"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/not-another-llm-i-tried-laya-a-421m-parameter-ai-decision-engine", "markdown": "https://wpnews.pro/news/not-another-llm-i-tried-laya-a-421m-parameter-ai-decision-engine.md", "text": "https://wpnews.pro/news/not-another-llm-i-tried-laya-a-421m-parameter-ai-decision-engine.txt", "jsonld": "https://wpnews.pro/news/not-another-llm-i-tried-laya-a-421m-parameter-ai-decision-engine.jsonld"}}