Not Another LLM: I Tried Laya, a 421M-Parameter AI Decision Engine

A developer tested Laya, a 322M–421M parameter non-autoregressive decision model, on a Mac mini M4 with 16GB of memory, running it locally for English and Hindi inputs, multiple decisions, longer texts, and larger choice sets, and also built a simple Gradio interface. Unlike traditional LLMs, Laya does not generate text; it answers typed questions and returns structured decisions with probabilities, leading the developer to conclude it could serve as a lightweight decision layer alongside a larger LLM rather than replacing one.

I spend a lot of time trying different AI models, especially models that can run locally. Usually, the process is pretty familiar: download the model, give it a prompt, and wait for it to generate something. Laya caught my attention because it doesn't work that way. It isn't built to write an answer, generate code, or have a conversation. Instead, you give it some text or a piece of information and ask it specific questions. It then returns a decision and a probability for that decision. That sounded simple, but also quite different from the models I normally experiment with. So I decided to install Laya on my Mac mini M4 with 16GB memory and see how it actually performs. I wanted to start from the basics, run it locally, try a few real examples, and understand where a small decision-focused model makes sense compared with using a much larger language model for everything. This post is my hands-on experience with that process, from installation and the first prediction to multilingual input, model routing, and some of the limitations I found along the way. Laya is a 322M–421M parameter decision model that works very differently from a traditional LLM. It doesn't generate text; it takes an input, answers typed questions, and returns structured decisions with probabilities. I installed it on my M4 Mac mini, tested English, Hindi, multiple decisions, longer inputs, and larger choice sets, and also built a simple Gradio interface. The interesting part is that Laya could work as a lightweight decision layer alongside a larger LLM rather than replacing one. Before installing it, it helps to understand what makes Laya different from the models I usually run locally. Most language models are built around generating text. You send them a prompt, they process it one token at a time, and eventually you get a response. That makes sense for chat, coding, writing, summarization, and many other tasks. Laya takes a different approach. It is a non-autoregressive decision model. Instead of generating a response, it looks at the information you provide and answers a set of questions about it. It returns the answers in a structured format, along with probabilities. For example, imagine I receive this support message: Hi, we were charged twice for March. Please refund the duplicate payment today. With a normal LLM, I might ask: Which department should handle this request? and get a natural-language response such as: This looks like a billing issue, so it should be handled by the billing department. With Laya, I can define the possible decisions beforehand: { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else" } Laya can then return the selected option and its probability rather than generating an explanation. That's the part I found interesting. There is no long response to parse, no generated paragraph sitting between the model and my application, and no need to ask the model to format its answer as JSON after the fact. It's a small model too Laya currently comes with three checkpoints. The main English model has 421 million parameters, while the multilingual checkpoint has 322 million parameters. There is also a 421M-parameter checkpoint fine-tuned for typed decision tasks. For comparison, the local language models I normally experiment with are measured in billions of parameters. Laya is operating on a completely different scale. But the parameter count isn't really the main story here. The important thing is that Laya is designed for a much narrower job. It isn't trying to write an email for me or build an application from a prompt. It is trying to answer questions such as: Which category is this? How urgent is this? Does the user want a refund? Should this request be routed to another system? That makes it feel less like a chatbot and more like a decision layer that can sit inside a larger application. And that was enough to make me want to try it locally. Once I understood what Laya was actually built for, the next step was simply getting it running. I wanted to keep the setup clean, so I used a separate Python virtual environment instead of installing everything directly into my system Python. My Mac mini has an M4 chip with 16GB of unified memory, so I wasn't expecting the model to be particularly demanding. The main Laya checkpoint is also only a few hundred million parameters, which makes it much smaller than the local language models I usually work with. I started by creating a new folder for the experiment: mkdir laya-test cd laya-test Then I created a virtual environment: python3 -m venv .venv source .venv/bin/activate Once the environment was active, my terminal showed .venv at the beginning of the command line. The installation itself is surprisingly simple: python -m pip install laya That's it. There was no separate model download command at this stage. The package installs first, and the actual checkpoint is downloaded the first time I use it. Before loading a model, I wanted to make sure the package was installed correctly: python python -c "import laya; print laya. version " This is a useful little check because it confirms that Python can find the package and tells me which version I'm actually running. A small thing worth knowing Laya doesn't download every checkpoint immediately. The project has three checkpoints, but the one I request is what gets loaded. This is useful on a machine like mine because I don't need to spend storage space downloading models that I haven't decided to use yet. The English checkpoint is around 843 MB, while the multilingual checkpoint is around 647 MB. That's tiny compared with many of the local models I've worked with. So even with limited free space on my internal drive, Laya itself wasn't going to be the problem. Now for the interesting part The package was installed, the environment was ready, and there was only one thing left to do: Actually make Laya take a decision. Now that Laya is installed, I wanted to run a small example and see how it handles a real decision. Inside the laya-test folder, I created a Python file: touch test laya.py Then I opened the file and added the following code: nano test laya.py I added this code: python from laya import Router router = Router state = """ Hi, we were billed twice for March. Please refund the duplicate payment today. """ questions = { "department": { "type": "choice", "instructions": "Which department should handle this?", "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else" } } } result = router.predict state, questions print result After adding the code, I saved the file and closed the editor. The example is intentionally simple. I give Laya a short customer-support message and define three possible departments. Laya then has to decide which department is the right one. With the file ready, I ran it from the same terminal: python test laya.py On the first run, Laya downloaded the required model checkpoint from Hugging Face. In my case, the download was around 846 MB. After the model finished loading, Laya processed the message and returned a structured result. The important part of my output was: department → billing It also returned probabilities for all three choices, rather than generating a normal text response. This was my first successful Laya inference running locally on my M4 Mac mini. The first test was successful, but the output from Laya contains more information than just the final category. My terminal returned a result similar to this: 'answers': { 'department': { 'type': 'choice', 'choice': 'billing', 'probabilities': { 'billing': 0.9875, 'technical': 0.0063, 'other': 0.0062 }, 'confidence': 0.9309, 'answer confidence': 0.9875 } } The first thing to look at is: choice: billing That's the actual decision Laya made. Then there are the probabilities: billing → 0.9875 technical → 0.0063 other → 0.0062 So, for this particular input, the model strongly preferred the billing category. Another interesting part of the result is: output tokens: 0 This is one of the easiest ways to see how Laya differs from a traditional text-generating model. It didn't generate a sentence explaining its answer. It evaluated the input and returned the structured result. The response also included routing information: model: english reason: English Latin text This means the built-in Router detected my input as English and selected the English checkpoint automatically. One thing I noticed during this test was a warning about calibration. The runtime reported that one temperature value was outside its allowed range and was adjusted. Because of that, I won't treat the displayed probability as a guaranteed measure of real-world accuracy. It is better to validate and calibrate these probabilities on the data you're actually using. For a first local test, though, the important part was clear: Laya was installed, the model loaded successfully, and it made a structured decision on my Mac mini. After the first test worked, I wanted to see what would happen if I asked Laya more than one question about the same message. The nice thing here is that I don't have to send the same text to the model again and again. I can define several questions and pass them together. I kept the same customer-support example and added two more decisions: urgency and churn risk. I updated test laya.py with this: python from laya import Router router = Router state = """ Hi, we were billed twice for March. Please refund the duplicate payment today or we will cancel our plan. """ questions = { "department": { "type": "choice", "instructions": "Which department should handle this?", "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else" } }, "urgency": { "type": "score", "instructions": "How urgent is this?", "criteria": "not urgent", "soon", "critical" }, "churn risk": { "type": "noul", "instructions": "Does the user threaten to cancel or leave?" } } result = router.predict state, questions print result The input now contains three things at once: Department Urgency Churn risk So instead of asking Laya: Which department should handle this? I'm also asking: How urgent is this? and: Is the customer threatening to leave? I saved the file and ran it again: python test laya.py This time, the result included answers for all three questions. This time, Laya returned all three decisions from the same input. For the department, it selected: billing with: billing → 0.9801 technical → 0.0113 other → 0.0086 The model's reported confidence was 0.8989. For urgency, the result was: score: 1.2514 The three possible levels had these probabilities: not urgent → 0.1502 soon → 0.4482 critical → 0.4016 So the model leaned toward “soon”, although the probabilities were much closer here than they were for the department decision. Finally, for churn risk, Laya returned: 0.8553 So in this particular test, it assigned an 85.53% probability to the user threatening to cancel or leave. One detail that stood out was: input tokens: 179 output tokens: 0 Even though I asked three different questions, there was still no generated text output. That's probably the simplest way to explain what makes Laya interesting. I'm giving it one piece of information and several decisions to make, and it returns those decisions in a structured format. The Router also reported: model: english reason: English Latin text So the English checkpoint was selected automatically. One thing I noticed The runtime again showed the warning about an invalid temperature value and said the affected confidence should be treated as uncalibrated. So while these probabilities are useful for seeing what the model is doing, I wouldn't present them as a measure of actual accuracy. For me, the more interesting result was the shape of the output: one input, three decisions, and zero generated tokens. After testing the English example, I wanted to see what happens when I give Laya a message in another language. I used a simple Hindi sentence: मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें। The meaning is roughly: I was charged twice; please refund the money. This is where the Router becomes useful. I don't have to manually decide which checkpoint to load. Laya can inspect the input and choose the appropriate model. I updated my test laya.py file and replaced the English state with the Hindi example: python from laya import Router router = Router state = """ मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें। """ questions = { "department": { "type": "choice", "instructions": "Which department should handle this?", "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else" } } } result = router.predict state, questions print result I kept the same billing question and ran the file again: python test laya.py This time, Laya had to download its multilingual checkpoint. The download was around 653 MB, and the reconstructed checkpoint was about 678 MB. The prediction itself was: department → billing with these probabilities: billing → 0.9923 technical → 0.0026 other → 0.0051 The reported confidence was 0.9544. But the part I was really interested in was the routing information. Laya returned: model: multilingual and gave the reason: non-Latin script devanagari, 100% of letters ; the English checkpoint cannot read it So the Router recognized the script as Devanagari and automatically sent the request to the multilingual checkpoint. The routing metadata also showed: script: devanagari language: None is english: False That last part is interesting. Laya didn't need to identify the exact language as Hindi first. The script itself was enough for the Router to decide that the English checkpoint shouldn't be used. The complete flow looked like this: Hindi text ↓ Router ↓ Devanagari detected ↓ Multilingual checkpoint ↓ Billing decision And once again, the model reported: input tokens: 55 output tokens: 0 So even with the multilingual model, there was still no text generation involved. The multilingual checkpoint was a little smaller on disk than the English one in my tests: English checkpoint → ~846 MB Multilingual checkpoint → ~678 MB reconstructed That made this test even more interesting for my setup, because both checkpoints are small enough to work comfortably on my Mac mini. The Hindi test gave me the prediction I expected, but I was more curious about what happened before the prediction. Laya's Router doesn't just return the final answer. It also gives back information about the checkpoint it selected and why it selected it. I already had this in my result: result = router.predict state, questions So I added a couple of simple print statements to see the routing details more clearly: print "Selected model:", result "routing" "model" print "Reason:", result "routing" "reason" print "Detection:", result "routing" "detection" For my Hindi test, the output showed: Selected model: multilingual Reason: non-Latin script devanagari, 100% of letters ; the English checkpoint cannot read it And the detection information looked like: script: devanagari language: None is english: False That was useful because it showed me that the Router didn't need to first figure out that the sentence was specifically Hindi. It could see that the text was written in Devanagari, which was enough to avoid sending it to the English checkpoint. I can also check the route separately Another thing I found useful is that routing can be inspected without actually running the model. For example: route = router.route state, questions print "Model:", route.model print "Reason:", route.reason This gives me the routing decision before inference happens. That means the overall flow is roughly: Input ↓ Router ↓ Detect script / language ↓ Select checkpoint ↓ Run inference ↓ Return decision I like this separation because it makes the behavior easier to understand when you're building a real application. The routing step isn't hidden inside some giant prompt. You can actually see which checkpoint was selected and the reason for it. English and Hindi side by side After running both tests, I had a simple comparison: English ↓ English / Laya checkpoint Hindi ↓ Devanagari detected ↓ Multilingual checkpoint So far, the whole thing had been surprisingly lightweight. I had one model around 846 MB, another around 678 MB in my local test, and both were running directly on my Mac mini without needing a remote API. The first few tests were intentionally simple. I wanted to make sure the installation worked, understand the output, and see how the Router handled different languages. Now I wanted to make the example a little closer to something I might actually use in an application. Instead of asking Laya only for the department, I decided to extract several useful decisions from the same support message. I used this example: Hi, we were billed twice for March. Please refund the duplicate payment today or we will cancel our plan. From that single message, I wanted Laya to determine: I updated test laya.py with: python from laya import Router router = Router state = """ Hi, we were billed twice for March. Please refund the duplicate payment today or we will cancel our plan. """ questions = { "department": { "type": "choice", "instructions": "Which department should handle this?", "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else" } }, "urgency": { "type": "score", "instructions": "How urgent is this?", "criteria": "not urgent", "soon", "critical" }, "refund requested": { "type": "noul", "instructions": "Does the user explicitly request a refund?" }, "churn risk": { "type": "noul", "instructions": "Does the user threaten to cancel or leave?" } } result = router.predict state, questions print result I updated my test laya.py file with the four questions and ran: python test laya.py The result was interesting. Department Laya selected: billing The probabilities were: billing → 0.9801 technical → 0.0113 other → 0.0086 So the model clearly associated the message with a billing issue. Urgency For urgency, I defined three levels: 0 → not urgent 1 → soon 2 → critical score: 1.2514 not urgent → 0.1502 soon → 0.4482 critical → 0.4016 The model leaned toward “soon”, although the probabilities were much closer together than they were for the department decision. The reported confidence here was also quite low at 0.0799, which is a good reminder not to treat every model output as equally certain. Refund requested For the refund question, Laya returned: 0.8822 So it assigned an 88.22% probability to the user explicitly requesting a refund. Churn risk The final question was whether the customer was threatening to leave. 0.8553 That corresponds to an 85.53% probability for that decision. The overall result can be summarized like this: Customer message ↓ Laya ↓ ┌──────────┬──────────┬───────────┬──────────┐ ↓ ↓ ↓ ↓ Billing Urgency Refund Churn 98.01% 1.25 88.22% 85.53% There was another detail I found useful in the output: input tokens: 239 output tokens: 0 Even with four separate decisions, Laya still generated zero output tokens. That is probably the clearest practical difference I've seen so far between Laya and a conventional text-generating model. I give it a piece of text, define the decisions I care about, and it returns structured results instead of generating a response. The Router also continued to select the English checkpoint automatically: model: english reason: English Latin text At this point, Laya was starting to look less like a small chatbot and more like a component I could put inside a larger application. There is still one limitation worth keeping in mind. The runtime again warned about an out-of-range temperature value and said the affected confidence values should be treated as uncalibrated. So these numbers are useful for understanding this particular prediction, but I wouldn't use them as evidence that the model is 98% or 85% accurate in general. For a local experiment, though, this was a useful result: one input, four decisions, and no generated text. So far, most of my tests used short messages. That's useful for understanding the basics, but real applications rarely receive just two or three lines of text. A support ticket can contain the original complaint, previous replies, order details, and a few extra paragraphs. So my next test was to give Laya a longer piece of text and see how it handles it. For this test, I kept the same questions but replaced the short message with a more detailed customer request. I updated test laya.py like this: python from laya import Router router = Router state = """ Hi support team, I am contacting you because I noticed that my March subscription payment was charged twice on my account. I checked my bank statement and can see two separate charges for the same amount on the same day. I have already checked my account dashboard, but I can only see one invoice there. I would like the duplicate payment to be refunded as soon as possible. This has been quite frustrating because I need the money back before the end of the week. If this issue cannot be resolved soon, I may have to cancel my subscription and move to another service. Please check the payment history, confirm what happened, and let me know when the duplicate amount will be returned. """ questions = { "department": { "type": "choice", "instructions": "Which department should handle this?", "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else" } }, "urgency": { "type": "score", "instructions": "How urgent is this?", "criteria": "not urgent", "soon", "critical" }, "refund requested": { "type": "noul", "instructions": "Does the user explicitly request a refund?" }, "churn risk": { "type": "noul", "instructions": "Does the user threaten to cancel or leave?" } } result = router.predict state, questions print result Then I ran the same file again: python test laya.py This time, the model processed 751 input tokens and still returned: output tokens: 0 That part stayed the same. Even with a much longer input, Laya wasn't generating a written response. The department result was: billing → 0.9115 technical → 0.0524 other → 0.0361 So Laya still identified the request as a billing issue, although the probability was lower than in my shorter test. For urgency, it returned: not urgent → 0.2238 soon → 0.7585 critical → 0.0177 The resulting score was: 0.7939 So this time the model clearly leaned toward “soon.” The refund question returned: 0.8231 which means the model assigned an 82.31% probability to the user explicitly requesting a refund. Then I got one result that I found quite interesting. The churn-risk question returned: 0.1494 So the model assigned only 14.94% probability to the user threatening to cancel, even though the text explicitly included: “If this issue cannot be resolved soon, I may have to cancel my subscription.” This was a useful reminder that I shouldn't assume the model will interpret every question exactly the way I expect. The Laya documentation also points out that its noul decision can sometimes be affected by the labels used for the two possible outcomes. So this result is something I would want to investigate rather than simply accepting it as a correct or incorrect verdict. The longer input changed the results Comparing this with my earlier short test was also interesting. The shorter version gave me: billing → 98.01% while the longer version gave: billing → 91.15% The model still selected the same category, but the probability changed once more context was added. That's actually something I wanted to see. A longer input isn't automatically going to produce the same confidence as a short one, even when the underlying topic hasn't changed. For this particular test, the 751-token input was still handled without any problem by the English checkpoint. At this point, I had tested short inputs, multiple questions, different languages, and a longer support ticket. So far, I had been giving Laya only a few possible answers. That works nicely for a simple example like: billing technical other But real classification systems can have many more categories. A support system, for example, might have separate labels for refunds, card problems, failed transfers, account access, delivery issues, verification, and many other cases. So I wanted to see how Laya behaves when the number of choices starts increasing. Instead of changing the whole project, I kept the same setup and created a larger choice question. I used this: python from laya import Router router = Router state = """ I tried to transfer money to my bank account this morning. The transfer failed, but I can see the amount has already been deducted from my account. I want to know when the money will be returned. """ questions = { "intent": { "type": "choice", "instructions": "What is the main reason for this support request?", "criteria": { "card payment": "problem with a card payment", "cash withdrawal": "problem withdrawing cash", "bank transfer": "problem with a bank transfer", "refund": "asking for a refund", "account access": "cannot access the account", "verification": "identity or account verification problem", "subscription": "subscription or recurring payment problem", "fees": "question about fees or charges", "exchange rate": "question about currency exchange rates", "card delivery": "problem with card delivery", "cash deposit": "problem depositing cash", "other": "something not covered by the other categories" } } } result = router.predict state, questions print result The input was: I tried to transfer money to my bank account this morning. The transfer failed, but I can see the amount has already been deducted from my account. I want to know when the money will be returned. I then asked Laya to choose the main reason for the request from these categories: card payment cash withdrawal bank transfer refund account access verification subscription fees exchange rate card delivery cash deposit other I ran the same command as before: python test laya.py bank transfer The result was extremely strong: bank transfer → 1.0 The other 11 categories were returned as 0.0 in this particular run. The reported confidence was: confidence: 0.9998 answer confidence: 1.0 It also processed: input tokens: 174 output tokens: 0 So once again, there was no generated text. I expected adding more options to make the decision noticeably harder, but with 12 choices, Laya still identified the correct category very clearly. That said, there is an important detail here. The runtime again showed the warning about an invalid temperature value and said that the affected confidence values were uncalibrated. Because of that, I wouldn't interpret the 1.0 probability as meaning that the model is literally certain or that it will be correct 100% of the time. What I can say from my test is much simpler: With this particular 12-option example, Laya selected bank transfer and assigned virtually all of the probability mass to that category. This test also showed me why the number of choices matters. Laya has a fixed budget for representing the options in a question. With 12 categories, there is still enough room for the descriptions to remain distinct. But what happens when the number jumps from 12 choices to dozens of choices? The project's own benchmarks show that high-cardinality questions can become a problem. The documentation specifically discusses a 77-option Banking77 test, where the available option budget becomes much tighter. So my next experiment is going to push this much further. After my 12-choice test worked, I wanted to push the model much further. The Laya documentation talks about an important limitation with high-cardinality choice questions. When a question has a large number of options, the model has to fit all of those option descriptions into a fixed part of its input budget. With 50 or more choices, this can make the options harder to distinguish. The project's benchmark specifically uses Banking77, a dataset with 77 banking intents, to demonstrate this limitation. So I decided to recreate a similar test locally. Setting up the 77 choices I used the 77 Banking77 intent names as the choices. Banking77 is a fine-grained banking intent dataset containing 77 categories, including things such as card arrival, failed transfer, request refund, transaction charged twice, and verify my identity. For the experiment, I used a straightforward banking query: I tried to make a bank transfer this morning. The transfer failed, but the money has already been deducted from my account. I then created a choice question containing all 77 categories. The code looks like this: python from laya import Router router = Router state = """ I tried to make a bank transfer this morning. The transfer failed, but the money has already been deducted from my account. """ intents = { "activate my card": "activating a card", "age limit": "age requirements", "apple pay or google pay": "using Apple Pay or Google Pay", "atm support": "ATM support", "automatic top up": "automatic top up", "balance not updated after bank transfer": "balance not updated after bank transfer", "balance not updated after cheque or cash deposit": "balance not updated after cheque or cash deposit", "beneficiary not allowed": "beneficiary is not allowed", "cancel transfer": "cancelling a transfer", "card about to expire": "card is about to expire", "card acceptance": "where a card is accepted", "card arrival": "card arrival", "card delivery estimate": "estimated card delivery time", "card linking": "linking a card", "card not working": "card not working", "card payment fee charged": "fee charged for a card payment", "card payment not recognised": "unrecognised card payment", "card payment wrong exchange rate": "wrong exchange rate for card payment", "card swallowed": "card retained by an ATM", "cash withdrawal charge": "cash withdrawal charge", "cash withdrawal not recognised": "unrecognised cash withdrawal", "change pin": "changing PIN", "compromised card": "compromised card", "contactless not working": "contactless not working", "country support": "supported countries", "declined card payment": "declined card payment", "declined cash withdrawal": "declined cash withdrawal", "declined transfer": "declined transfer", "direct debit payment not recognised": "unrecognised direct debit", "disposable card limits": "disposable card limits", "edit personal details": "editing personal details", "exchange charge": "currency exchange charge", "exchange rate": "exchange rate", "exchange via app": "exchanging currency through the app", "extra charge on statement": "extra charge on statement", "failed transfer": "failed bank transfer", "fiat currency support": "supported fiat currencies", "get disposable virtual card": "getting a disposable virtual card", "get physical card": "getting a physical card", "getting spare card": "getting a spare card", "getting virtual card": "getting a virtual card", "lost or stolen card": "lost or stolen card", "lost or stolen phone": "lost or stolen phone", "order physical card": "ordering a physical card", "passcode forgotten": "forgotten passcode", "pending card payment": "pending card payment", "pending cash withdrawal": "pending cash withdrawal", "pending top up": "pending top up", "pending transfer": "pending transfer", "pin blocked": "blocked PIN", "receiving money": "receiving money", "refund not showing up": "refund not showing up", "request refund": "requesting a refund", "reverted card payment": "reverted card payment", "supported cards and currencies": "supported cards and currencies", "terminate account": "closing the account", "top up by bank transfer charge": "charge for topping up by bank transfer", "top up by card charge": "charge for topping up by card", "top up by cash or cheque": "topping up by cash or cheque", "top up failed": "failed top up", "top up limits": "top up limits", "top up reverted": "reverted top up", "topping up by card": "topping up by card", "transaction charged twice": "transaction charged twice", "transfer fee charged": "fee charged for a transfer", "transfer into account": "transferring money into the account", "transfer not received by recipient": "recipient did not receive the transfer", "transfer timing": "transfer timing", "unable to verify identity": "unable to verify identity", "verify my identity": "identity verification", "verify source of funds": "verifying source of funds", "verify top up": "top up verification", "virtual card not working": "virtual card not working", "visa or mastercard": "Visa or Mastercard", "why verify identity": "why identity verification is required", "wrong amount of cash received": "wrong amount of cash received", "wrong exchange rate for cash withdrawal": "wrong exchange rate for cash withdrawal" } questions = { "intent": { "type": "choice", "instructions": "What is the main reason for this banking request?", "criteria": intents } } result = router.predict state, questions print result This is very different from my earlier 12-choice experiment. With 77 options, the model has to distinguish between many closely related banking intents. For example: failed transfer declined transfer pending transfer transfer not received by recipient cancel transfer transfer timing These categories are much closer to each other than something obvious like card arrival versus exchange rate. The Laya documentation says this is where its default option budget becomes a real constraint. In the documented Banking77 comparison, Laya's 77-choice result was substantially lower than its performance on smaller choice sets, and the authors explain that the option descriptions can become too compressed within the fixed head budget. python test laya.py The result surprised me failed transfer That was the category I would expect from this particular message. failed transfer → 0.9753 declined transfer → 0.0056 unable to verify identity → 0.0191 Most of the remaining categories were returned as 0.0 in this particular run. The reported values were: confidence: 0.9703 answer confidence: 0.9753 The input contained 350 tokens, while the output was again: output tokens: 0 So even with 77 possible choices, Laya still didn't generate any text. I expected the larger choice set to make the result obviously worse, but that didn't happen with this particular example. Laya still selected the correct-looking category and put most of the probability on it. However, this doesn't mean that Laya solves 77-choice classification reliably in general. The reason is that this was one manually constructed test. The official project benchmarks report a much more difficult Banking77 evaluation where the model's performance drops substantially with a large number of options. The project explains that the option descriptions have to share a limited token budget, which becomes increasingly restrictive as the number of choices grows. So my local experiment gave me one useful result: 77 choices did not break the model on this example. But it is not enough to claim that Laya performs well across a full 77-class benchmark. That's an important distinction, especially when writing about model experiments. The runtime continued to display the same temperature warning: this checkpoint ships invalid temperatures... Treat confidence from the affected entries as uncalibrated. So again, I'm treating the probabilities as the output of this particular run, not as a guarantee of real-world accuracy. At this point, I had a much better idea of how Laya behaves with different input sizes and different numbers of choices. Up to this point, I had been using the default Router: router = Router With that setup, Laya loads a checkpoint when it is needed. That works well when you're experimenting because you don't have to load every model into memory at startup. For the next test, I wanted to try a different approach. Laya also supports preloading, which means the checkpoints are loaded when the Router starts. I changed my code to: python from laya import Router router = Router preload=True state = """ I tried to make a bank transfer this morning. The transfer failed, but the money has already been deducted from my account. """ questions = { "intent": { "type": "choice", "instructions": "What is the main reason for this banking request?", "criteria": { "card payment": "problem with a card payment", "cash withdrawal": "problem withdrawing cash", "bank transfer": "problem with a bank transfer", "refund": "asking for a refund", "account access": "cannot access the account", "verification": "identity or account verification problem", "subscription": "subscription or recurring payment problem", "fees": "question about fees or charges", "exchange rate": "question about currency exchange rates", "card delivery": "problem with card delivery", "cash deposit": "problem depositing cash", "other": "something not covered by the other categories" } } } result = router.predict state, questions print "Selected intent:", result "answers" "intent" "choice" print "Probabilities:", result "answers" "intent" "probabilities" print "Confidence:", result "answers" "intent" "confidence" print "Routing model:", result "routing" "model" print "Routing reason:", result "routing" "reason" print "Input tokens:", result "usage" "input tokens" print "Output tokens:", result "usage" "output tokens" I then ran: python test laya.py The result The prediction was: Selected intent: failed transfer Laya assigned: failed transfer → 0.9753 The other notable probabilities were: declined transfer → 0.0056 unable to verify identity → 0.0191 0.9703 The request contained 350 input tokens, while the output was again: Output tokens: 0 The Router also selected: Routing model: english with the reason: English Latin text This time, the terminal didn't download the model again. The fetching step showed: 5/5 00:00 and the download/reconstruction both reported 0.00B. That makes sense because the checkpoint had already been downloaded during my earlier experiments. Laya was using the cached files instead of downloading another copy. The important thing to understand is that preload=True doesn't change the kind of answer Laya produces. It changes how the model is prepared for inference. With the normal setup: Request ↓ Load checkpoint if needed ↓ Prediction With preload enabled: Application starts ↓ Checkpoint s loaded ↓ Request arrives ↓ Prediction That makes more sense for something that is going to stay running, such as a local API, a support-ticket classifier, or another application making repeated decisions. There is a trade-off, though. Keeping models ready means using more memory. So preload mode is more useful when the machine has enough RAM to keep the required checkpoints resident. For my Mac mini, the interesting thing was that Laya was already operating in a completely different memory class from the large language models I normally experiment with. I didn't measure the exact RAM usage in this run, so I'm not going to make up a number. What I can confirm is that the preloaded setup ran successfully on my M4 Mac mini with 16GB unified memory. And at this point, I had tested Laya with short text, long text, multiple decisions, Hindi input, and a wide choice set. After running several different tests, I wanted to check something more practical. I already knew the model files were relatively small, but I wanted to see what they actually looked like on my Mac rather than relying only on the numbers from the model page. There are two different things to measure here: Storage — how much space the downloaded checkpoints occupy on the SSD. Memory — how much RAM the Python process needs while Laya is running. These are easy to confuse, because an 800 MB model file does not necessarily mean the application will use exactly 800 MB of memory. Hugging Face keeps downloaded model files in its local cache. I checked the cache with: du -sh ~/.cache/huggingface/hub To look specifically for Laya models: du -sh ~/.cache/huggingface/hub/models--convaiinnovations-- Depending on how the cache is configured on the Mac, the files may be stored in a different location. You can check that with: echo $HF HOME If nothing is printed, the standard Hugging Face cache location is normally used. For the memory side, I used macOS's built-in /usr/bin/time utility. Instead of: python test laya.py I ran: /usr/bin/time -l python test laya.py At the end of the output, macOS reports: maximum resident set size That gives me the peak memory used by the Python process during the run. This is a much more useful number for my blog than simply saying: “The model is 846 MB.” The model file and the actual runtime memory are two different measurements. After testing Laya from the terminal, I wanted to make the experiment a little more practical. Running a Python script every time works, but I wanted a small interface where I could paste a customer message, click a button, and immediately see what Laya thinks about it. So I added Gradio on top of the same Laya code. The flow now looks like this: Browser ↓ Gradio ↓ Laya Router ↓ Structured decisions I kept the interface simple. I wanted to see the department, refund request, churn risk, routing information, and token usage without having to read a large JSON response every time. Gradio is a simple way to do that. Inside the same virtual environment: python -m pip install gradio Create another file: touch app.py Then open it: nano app.py Add this: python import gradio as gr from laya import Router Load Laya once when the app starts router = Router preload=True def predict text : if not text.strip : return { "error": "Please enter some text." } questions = { "department": { "type": "choice", "instructions": "Which department should handle this?", "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "other": "everything else" } }, "refund requested": { "type": "noul", "instructions": "Does the user explicitly request a refund?" }, "churn risk": { "type": "noul", "instructions": "Does the user threaten to cancel or leave?" } } result = router.predict text, questions return { "department": result "answers" "department" , "refund requested": result "answers" "refund requested" , "churn risk": result "answers" "churn risk" , "routing": result "routing" , "usage": result "usage" } with gr.Blocks title="Laya Decision Demo" as demo: gr.Markdown " Laya Decision Demo" gr.Markdown "Enter a message and let Laya make structured decisions." text input = gr.Textbox label="Input", placeholder="Example: I was charged twice and want a refund.", lines=6 run button = gr.Button "Run Laya" output = gr.JSON label="Laya Result" run button.click fn=predict, inputs=text input, outputs=output if name == " main ": demo.launch Now start it with: python app.py Gradio will start a local web server and give you a local URL, typically something like: http://127.0.0.1:7860 Open that URL in your browser. You'll get a simple interface where you can enter something like: Hi, I was charged twice for my March subscription. Please refund the duplicate payment. and click: Run Laya The UI will then show the structured result returned by Laya. For one of my tests, I entered a customer-support message and got this result: Department: billing Refund requested: 0.8829 Churn risk: 0.0916 Model: english Input tokens: 161 Output tokens: 0 The department probabilities were: billing → 0.9637 technical → 0.0185 other → 0.0178 The refund decision returned: 0.8829 And the churn-risk value was: 0.0916 One thing I found particularly interesting was the token count: input tokens: 161 output tokens: 0 Again, Laya wasn't generating a paragraph in response to my input. The interface was simply displaying the structured decisions produced by the model. The Router also identified the input as English: model: english reason: English Latin text After playing with Laya locally, I wanted to look beyond my own small experiments and see how the different checkpoints perform on larger benchmark runs. The project currently has three checkpoints: | Checkpoint | Parameters | Context | Main use | | ------------------------ | ---------: | -----------------: | ------------------------ | | convaiinnovations/laya | 421M | 512 | English | | laya-multilingual | 322M | 1,024, up to 8,192 | 100+ languages | | laya-typed-decisions | 421M | 1,024 | Typed decision workflows | The first thing that stands out is how small these models are. The main Laya checkpoint is only 421 million parameters, and the multilingual version is even smaller at 322 million. That's a very different scale from the multi-billion-parameter models I normally think about when talking about local AI. The project's benchmark runs on a Tesla T4 show a few interesting numbers. For typed decisions, the fine-tuned laya-typed-decisions checkpoint reached 0.766 accuracy across 2,000 decisions. The base English and multilingual checkpoints were much lower on this particular benchmark: laya → 0.362 laya-multilingual → 0.342 laya-typed-decisions → 0.766 That difference is important. It tells me that the impressive 0.766 number shouldn't be interpreted as what the base Laya model does out of the box. The result comes from a checkpoint that was specifically fine-tuned for those typed-decision workflows. That's actually one of the more useful lessons from the benchmarks: fine-tuning matters a lot for this kind of model. The speed numbers are also interesting. On a Tesla T4, the project reports: | Questions in one call | Laya | Multilingual | | --------------------: | -------: | -----------: | | 1 | 39.5 ms | 32.8 ms | | 5 | 84.5 ms | 40.1 ms | | 10 | 158.6 ms | 72.3 ms | | 50 | 771 ms | 337 ms | Because multiple questions can be answered in one forward pass, the cost per question drops as the batch gets larger. For example, the multilingual checkpoint reaches about 7.2 ms per question when 10 questions are processed together. That's one of the places where the smaller decision-oriented architecture starts to make sense. Laya benchmark results across accuracy, latency, calibration, multilingual performance, and application workflows. One of the things I was particularly interested in was how Laya behaves when the input gets much longer. The multilingual checkpoint can be configured with: max len=8192 The project's long-context experiment placed the request at the end of documents with different amounts of unrelated text before it. The results were: | Text before request | Correct | Time per request | | ------------------- | ------: | ---------------: | | Short input | 19 / 20 | 0.02 s | | ~1,000 tokens | 16 / 20 | 0.21 s | | ~2,000 tokens | 17 / 20 | 0.50 s | | ~3,000 tokens | 17 / 20 | 0.90 s | | ~4,000 tokens | 18 / 20 | 1.71 s | | ~5,000 tokens | 11 / 20 | 2.59 s | | ~6,000 tokens | 17 / 20 | 3.50 s | | ~7,000 tokens | 8 / 20 | 4.50 s | The results aren't perfectly smooth, but they show the basic trade-off clearly: longer context costs more time, and accuracy can become more variable as the document gets very long. That matches what I saw in my own test earlier. My 751-token support ticket worked without any issue, but that doesn't mean I should throw an enormous document at the model and expect the same behavior. For a real application, I would test the exact input lengths and document types I care about. Long-context test for laya-multilingual with requests placed at different positions inside longer documents. After all the experiments, I think the most useful way to look at Laya is not as a replacement for a large language model. It's a different tool. The project reports strong results on some tasks. For example, its benchmark suite reports around 0.993 accuracy for email spam and phishing on the tested data, while several other workflows are considerably weaker. The same benchmark also shows an important limitation with large choice sets. On Banking77, which has 77 labels, the benchmark reports: Laya → 0.425 TypeSafe Jev → 0.870 The project attributes Laya's result to the way the available token budget is shared across the many option descriptions. That matches something I started noticing during my own experiments: three or twelve choices are very different from seventy-seven choices. So I wouldn't design a system around hundreds of labels in one giant choice question without testing it carefully. There are a few other things I would keep in mind too. The base checkpoints are not automatically excellent at every zero-shot decision task. The model's probability outputs can also require calibration, and some decision types perform better than others. For me, that's actually a positive part of the project rather than something to hide. The documentation is fairly open about where the model works and where it needs more care. After running all these tests, this is the comparison that makes the most sense to me. A traditional LLM generally looks like: Input ↓ Large language model ↓ Generate tokens ↓ Text response Laya looks more like: Input ↓ Laya ↓ Typed decision ↓ Probability That's a much narrower job. I wouldn't use Laya to write a blog post, generate code, summarize a long meeting, or have a conversation. But I could imagine using something like Laya for the smaller decisions that happen around those tasks. User request ↓ Laya ↓ Is this relevant? ↓ Is it urgent? ↓ Which workflow? ↓ LLM / Tool In other words, the interesting idea isn't: “Laya can replace my LLM.” For me, it's: “Maybe my LLM doesn't need to make every decision.” I started this experiment because Laya looked unusual. Most of the local models I come across are trying to become better at generation: better coding, better reasoning, better conversations, better multimodal capabilities. Laya went in a different direction. It takes a relatively small model, gives it a structured decision problem, and focuses on making that decision quickly. I tested it on my M4 Mac mini, ran the English checkpoint, switched to the multilingual model with Hindi input, asked multiple questions against the same text, tried a longer support ticket, increased the number of choices, and finally put the whole thing behind a small Gradio interface. What surprised me most was how little infrastructure was needed to get from zero to a working local demo. I didn't need a giant GPU server. I didn't need an API key for inference. I didn't need to run a multi-billion-parameter model. I just installed the package, downloaded the checkpoint, and started asking it structured questions. Maybe Every AI System Doesn't Need One Giant Model After spending so much time experimenting with large language models, I found Laya refreshing simply because it isn't trying to be one. It doesn't generate a long answer. It doesn't try to write code. It doesn't try to become a chatbot. It makes decisions. Running a 421M-parameter model locally on my M4 Mac mini made that idea feel much more practical to me. For something like customer-support routing, moderation, classification, guardrails, or other structured decisions, using a smaller specialized model can make more sense than sending every single request through a large generative model. At the same time, my experiments also showed that Laya isn't magic. Its performance depends heavily on the task, the number of choices, the input size, and whether the model has been fine-tuned for the problem. The confidence values also need to be treated carefully rather than taken as guaranteed accuracy. So I wouldn't look at Laya and ask: “Is this the next replacement for Llama or Qwen?” That's the wrong comparison. The more interesting question for me is: What if we stop expecting one model to do everything? A large language model can handle generation and reasoning. A small decision model can handle a narrow decision. A routing model can decide which path to take. And an application can combine all of them. That feels like a much more interesting direction for local AI. Laya may be small, but the idea behind it is much bigger: sometimes the best model for a job isn't the model that can do everything — it's the one designed to do that one job well.