I spend a lot of time trying different AI models, especially models that can run locally. Usually, the process is pretty familiar: download the model, give it a prompt, and wait for it to generate something.
Laya caught my attention because it doesn't work that way.
It isn't built to write an answer, generate code, or have a conversation. Instead, you give it some text or a piece of information and ask it specific questions. It then returns a decision and a probability for that decision.
That sounded simple, but also quite different from the models I normally experiment with.
So I decided to install Laya on my Mac mini M4 with 16GB memory and see how it actually performs. I wanted to start from the basics, run it locally, try a few real examples, and understand where a small decision-focused model makes sense compared with using a much larger language model for everything.
This post is my hands-on experience with that process, from installation and the first prediction to multilingual input, model routing, and some of the limitations I found along the way.
Laya is a 322M–421M parameter decision model that works very differently from a traditional LLM. It doesn't generate text; it takes an input, answers typed questions, and returns structured decisions with probabilities. I installed it on my M4 Mac mini, tested English, Hindi, multiple decisions, longer inputs, and larger choice sets, and also built a simple Gradio interface. The interesting part is that Laya could work as a lightweight decision layer alongside a larger LLM rather than replacing one.
Before installing it, it helps to understand what makes Laya different from the models I usually run locally.
Most language models are built around generating text. You send them a prompt, they process it one token at a time, and eventually you get a response. That makes sense for chat, coding, writing, summarization, and many other tasks.
Laya takes a different approach.
It is a non-autoregressive decision model. Instead of generating a response, it looks at the information you provide and answers a set of questions about it. It returns the answers in a structured format, along with probabilities.
For example, imagine I receive this support message:
Hi, we were charged twice for March.
Please refund the duplicate payment today.
With a normal LLM, I might ask:
Which department should handle this request?
and get a natural-language response such as:
This looks like a billing issue, so it should be handled
by the billing department.
With Laya, I can define the possible decisions beforehand:
{
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"
}
Laya can then return the selected option and its probability rather than generating an explanation.
That's the part I found interesting.
There is no long response to parse, no generated paragraph sitting between the model and my application, and no need to ask the model to format its answer as JSON after the fact.
It's a small model too
Laya currently comes with three checkpoints.
The main English model has 421 million parameters, while the multilingual checkpoint has 322 million parameters. There is also a 421M-parameter checkpoint fine-tuned for typed decision tasks.
For comparison, the local language models I normally experiment with are measured in billions of parameters. Laya is operating on a completely different scale.
But the parameter count isn't really the main story here.
The important thing is that Laya is designed for a much narrower job.
It isn't trying to write an email for me or build an application from a prompt. It is trying to answer questions such as:
Which category is this?
How urgent is this?
Does the user want a refund?
Should this request be routed to another system?
That makes it feel less like a chatbot and more like a decision layer that can sit inside a larger application.
And that was enough to make me want to try it locally.
Once I understood what Laya was actually built for, the next step was simply getting it running.
I wanted to keep the setup clean, so I used a separate Python virtual environment instead of installing everything directly into my system Python.
My Mac mini has an M4 chip with 16GB of unified memory, so I wasn't expecting the model to be particularly demanding. The main Laya checkpoint is also only a few hundred million parameters, which makes it much smaller than the local language models I usually work with.
I started by creating a new folder for the experiment:
mkdir laya-test
cd laya-test
Then I created a virtual environment:
python3 -m venv .venv
source .venv/bin/activate
Once the environment was active, my terminal showed (.venv) at the beginning of the command line.
The installation itself is surprisingly simple:
python -m pip install laya
That's it.
There was no separate model download command at this stage. The package installs first, and the actual checkpoint is downloaded the first time I use it.
Before a model, I wanted to make sure the package was installed correctly:
python -c "import laya; print(laya.__version__)"
This is a useful little check because it confirms that Python can find the package and tells me which version I'm actually running.
A small thing worth knowing
Laya doesn't download every checkpoint immediately.
The project has three checkpoints, but the one I request is what gets loaded. This is useful on a machine like mine because I don't need to spend storage space down models that I haven't decided to use yet.
The English checkpoint is around 843 MB, while the multilingual checkpoint is around 647 MB. That's tiny compared with many of the local models I've worked with.
So even with limited free space on my internal drive, Laya itself wasn't going to be the problem.
Now for the interesting part
The package was installed, the environment was ready, and there was only one thing left to do:
Actually make Laya take a decision.
Now that Laya is installed, I wanted to run a small example and see how it handles a real decision.
Inside the laya-test folder, I created a Python file:
touch test_laya.py
Then I opened the file and added the following code:
nano test_laya.py
I added this code:
from laya import Router
router = Router()
state = """
Hi, we were billed twice for March.
Please refund the duplicate payment today.
"""
questions = {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": {
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"
}
}
}
result = router.predict(state, questions)
print(result)
After adding the code, I saved the file and closed the editor.
The example is intentionally simple. I give Laya a short customer-support message and define three possible departments. Laya then has to decide which department is the right one.
With the file ready, I ran it from the same terminal:
python test_laya.py
On the first run, Laya downloaded the required model checkpoint from Hugging Face. In my case, the download was around 846 MB.
After the model finished , Laya processed the message and returned a structured result.
The important part of my output was:
department → billing
It also returned probabilities for all three choices, rather than generating a normal text response.
This was my first successful Laya inference running locally on my M4 Mac mini.
The first test was successful, but the output from Laya contains more information than just the final category.
My terminal returned a result similar to this:
'answers': {
'department': {
'type': 'choice',
'choice': 'billing',
'probabilities': {
'billing': 0.9875,
'technical': 0.0063,
'other': 0.0062
},
'confidence': 0.9309,
'answer_confidence': 0.9875
}
}
The first thing to look at is:
choice: billing
That's the actual decision Laya made.
Then there are the probabilities:
billing → 0.9875
technical → 0.0063
other → 0.0062
So, for this particular input, the model strongly preferred the billing category.
Another interesting part of the result is:
output_tokens: 0
This is one of the easiest ways to see how Laya differs from a traditional text-generating model. It didn't generate a sentence explaining its answer. It evaluated the input and returned the structured result.
The response also included routing information:
model: english
reason: English Latin text
This means the built-in Router detected my input as English and selected the English checkpoint automatically.
One thing I noticed during this test was a warning about calibration. The runtime reported that one temperature value was outside its allowed range and was adjusted. Because of that, I won't treat the displayed probability as a guaranteed measure of real-world accuracy. It is better to validate and calibrate these probabilities on the data you're actually using.
For a first local test, though, the important part was clear: Laya was installed, the model loaded successfully, and it made a structured decision on my Mac mini.
After the first test worked, I wanted to see what would happen if I asked Laya more than one question about the same message.
The nice thing here is that I don't have to send the same text to the model again and again. I can define several questions and pass them together.
I kept the same customer-support example and added two more decisions: urgency and churn risk.
I updated test_laya.py with this:
from laya import Router
router = Router()
state = """
Hi, we were billed twice for March.
Please refund the duplicate payment today
or we will cancel our plan.
"""
questions = {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": {
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"
}
},
"urgency": {
"type": "score",
"instructions": "How urgent is this?",
"criteria": [
"not urgent",
"soon",
"critical"
]
},
"churn_risk": {
"type": "noul",
"instructions": "Does the user threaten to cancel or leave?"
}
}
result = router.predict(state, questions)
print(result)
The input now contains three things at once:
Department
Urgency
Churn risk
So instead of asking Laya:
Which department should handle this?
I'm also asking:
How urgent is this?
and:
Is the customer threatening to leave?
I saved the file and ran it again:
python test_laya.py
This time, the result included answers for all three questions.
This time, Laya returned all three decisions from the same input.
For the department, it selected:
billing
with:
billing → 0.9801
technical → 0.0113
other → 0.0086
The model's reported confidence was 0.8989.
For urgency, the result was:
score: 1.2514
The three possible levels had these probabilities:
not urgent → 0.1502
soon → 0.4482
critical → 0.4016
So the model leaned toward “soon”, although the probabilities were much closer here than they were for the department decision.
Finally, for churn risk, Laya returned:
0.8553
So in this particular test, it assigned an 85.53% probability to the user threatening to cancel or leave.
One detail that stood out was:
input_tokens: 179
output_tokens: 0
Even though I asked three different questions, there was still no generated text output.
That's probably the simplest way to explain what makes Laya interesting. I'm giving it one piece of information and several decisions to make, and it returns those decisions in a structured format.
The Router also reported:
model: english
reason: English Latin text
So the English checkpoint was selected automatically.
One thing I noticed
The runtime again showed the warning about an invalid temperature value and said the affected confidence should be treated as uncalibrated. So while these probabilities are useful for seeing what the model is doing, I wouldn't present them as a measure of actual accuracy.
For me, the more interesting result was the shape of the output: one input, three decisions, and zero generated tokens.
After testing the English example, I wanted to see what happens when I give Laya a message in another language.
I used a simple Hindi sentence:
मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें।
The meaning is roughly:
I was charged twice; please refund the money.
This is where the Router becomes useful. I don't have to manually decide which checkpoint to load. Laya can inspect the input and choose the appropriate model.
I updated my test_laya.py file and replaced the English state with the Hindi example:
from laya import Router
router = Router()
state = """
मुझसे दो बार शुल्क लिया गया,
कृपया पैसे वापस करें।
"""
questions = {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": {
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"
}
}
}
result = router.predict(state, questions)
print(result)
I kept the same billing question and ran the file again:
python test_laya.py
This time, Laya had to download its multilingual checkpoint. The download was around 653 MB, and the reconstructed checkpoint was about 678 MB.
The prediction itself was:
department → billing
with these probabilities:
billing → 0.9923
technical → 0.0026
other → 0.0051
The reported confidence was 0.9544.
But the part I was really interested in was the routing information.
Laya returned:
model: multilingual
and gave the reason:
non-Latin script (devanagari, 100% of letters);
the English checkpoint cannot read it
So the Router recognized the script as Devanagari and automatically sent the request to the multilingual checkpoint.
The routing metadata also showed:
script: devanagari
language: None
is_english: False
That last part is interesting. Laya didn't need to identify the exact language as Hindi first. The script itself was enough for the Router to decide that the English checkpoint shouldn't be used.
The complete flow looked like this:
Hindi text
↓
Router
↓
Devanagari detected
↓
Multilingual checkpoint
↓
Billing decision
And once again, the model reported:
input_tokens: 55
output_tokens: 0
So even with the multilingual model, there was still no text generation involved.
The multilingual checkpoint was a little smaller on disk than the English one in my tests:
English checkpoint → ~846 MB
Multilingual checkpoint → ~678 MB reconstructed
That made this test even more interesting for my setup, because both checkpoints are small enough to work comfortably on my Mac mini.
The Hindi test gave me the prediction I expected, but I was more curious about what happened before the prediction.
Laya's Router doesn't just return the final answer. It also gives back information about the checkpoint it selected and why it selected it.
I already had this in my result:
result = router.predict(state, questions)
So I added a couple of simple print statements to see the routing details more clearly:
print("Selected model:", result["routing"]["model"])
print("Reason:", result["routing"]["reason"])
print("Detection:", result["routing"]["detection"])
For my Hindi test, the output showed:
Selected model: multilingual
Reason:
non-Latin script (devanagari, 100% of letters);
the English checkpoint cannot read it
And the detection information looked like:
script: devanagari
language: None
is_english: False
That was useful because it showed me that the Router didn't need to first figure out that the sentence was specifically Hindi. It could see that the text was written in Devanagari, which was enough to avoid sending it to the English checkpoint.
I can also check the route separately
Another thing I found useful is that routing can be inspected without actually running the model.
For example:
route = router.route(state, questions)
print("Model:", route.model)
print("Reason:", route.reason)
This gives me the routing decision before inference happens.
That means the overall flow is roughly:
Input
↓
Router
↓
Detect script / language
↓
Select checkpoint
↓
Run inference
↓
Return decision
I like this separation because it makes the behavior easier to understand when you're building a real application. The routing step isn't hidden inside some giant prompt. You can actually see which checkpoint was selected and the reason for it.
English and Hindi side by side
After running both tests, I had a simple comparison:
English
↓
English / Laya checkpoint
Hindi
↓
Devanagari detected
↓
Multilingual checkpoint
So far, the whole thing had been surprisingly lightweight.
I had one model around 846 MB, another around 678 MB in my local test, and both were running directly on my Mac mini without needing a remote API.
The first few tests were intentionally simple. I wanted to make sure the installation worked, understand the output, and see how the Router handled different languages.
Now I wanted to make the example a little closer to something I might actually use in an application.
Instead of asking Laya only for the department, I decided to extract several useful decisions from the same support message.
I used this example:
Hi, we were billed twice for March.
Please refund the duplicate payment today
or we will cancel our plan.
From that single message, I wanted Laya to determine:
I updated test_laya.py with:
from laya import Router
router = Router()
state = """
Hi, we were billed twice for March.
Please refund the duplicate payment today
or we will cancel our plan.
"""
questions = {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": {
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"
}
},
"urgency": {
"type": "score",
"instructions": "How urgent is this?",
"criteria": [
"not urgent",
"soon",
"critical"
]
},
"refund_requested": {
"type": "noul",
"instructions": "Does the user explicitly request a refund?"
},
"churn_risk": {
"type": "noul",
"instructions": "Does the user threaten to cancel or leave?"
}
}
result = router.predict(state, questions)
print(result)
I updated my test_laya.py file with the four questions and ran:
python test_laya.py
The result was interesting.
Department
Laya selected:
billing
The probabilities were:
billing → 0.9801
technical → 0.0113
other → 0.0086
So the model clearly associated the message with a billing issue.
Urgency
For urgency, I defined three levels:
0 → not urgent
1 → soon
2 → critical
score: 1.2514
not urgent → 0.1502
soon → 0.4482
critical → 0.4016
The model leaned toward “soon”, although the probabilities were much closer together than they were for the department decision.
The reported confidence here was also quite low at 0.0799, which is a good reminder not to treat every model output as equally certain.
Refund requested
For the refund question, Laya returned:
0.8822
So it assigned an 88.22% probability to the user explicitly requesting a refund.
Churn risk
The final question was whether the customer was threatening to leave.
0.8553
That corresponds to an 85.53% probability for that decision.
The overall result can be summarized like this:
Customer message
↓
Laya
↓
┌──────────┬──────────┬───────────┬──────────┐
↓ ↓ ↓ ↓
Billing Urgency Refund Churn
98.01% 1.25 88.22% 85.53%
There was another detail I found useful in the output:
input_tokens: 239
output_tokens: 0
Even with four separate decisions, Laya still generated zero output tokens.
That is probably the clearest practical difference I've seen so far between Laya and a conventional text-generating model. I give it a piece of text, define the decisions I care about, and it returns structured results instead of generating a response.
The Router also continued to select the English checkpoint automatically:
model: english
reason: English Latin text
At this point, Laya was starting to look less like a small chatbot and more like a component I could put inside a larger application.
There is still one limitation worth keeping in mind. The runtime again warned about an out-of-range temperature value and said the affected confidence values should be treated as uncalibrated. So these numbers are useful for understanding this particular prediction, but I wouldn't use them as evidence that the model is 98% or 85% accurate in general.
For a local experiment, though, this was a useful result: one input, four decisions, and no generated text.
So far, most of my tests used short messages. That's useful for understanding the basics, but real applications rarely receive just two or three lines of text.
A support ticket can contain the original complaint, previous replies, order details, and a few extra paragraphs. So my next test was to give Laya a longer piece of text and see how it handles it.
For this test, I kept the same questions but replaced the short message with a more detailed customer request.
I updated test_laya.py like this:
from laya import Router
router = Router()
state = """
Hi support team,
I am contacting you because I noticed that my March subscription
payment was charged twice on my account. I checked my bank statement
and can see two separate charges for the same amount on the same day.
I have already checked my account dashboard, but I can only see one
invoice there. I would like the duplicate payment to be refunded as
soon as possible.
This has been quite frustrating because I need the money back before
the end of the week. If this issue cannot be resolved soon, I may have
to cancel my subscription and move to another service.
Please check the payment history, confirm what happened, and let me
know when the duplicate amount will be returned.
"""
questions = {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": {
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"
}
},
"urgency": {
"type": "score",
"instructions": "How urgent is this?",
"criteria": [
"not urgent",
"soon",
"critical"
]
},
"refund_requested": {
"type": "noul",
"instructions": "Does the user explicitly request a refund?"
},
"churn_risk": {
"type": "noul",
"instructions": "Does the user threaten to cancel or leave?"
}
}
result = router.predict(state, questions)
print(result)
Then I ran the same file again:
python test_laya.py
This time, the model processed 751 input tokens and still returned:
output_tokens: 0
That part stayed the same. Even with a much longer input, Laya wasn't generating a written response.
The department result was:
billing → 0.9115
technical → 0.0524
other → 0.0361
So Laya still identified the request as a billing issue, although the probability was lower than in my shorter test.
For urgency, it returned:
not urgent → 0.2238
soon → 0.7585
critical → 0.0177
The resulting score was:
0.7939
So this time the model clearly leaned toward “soon.”
The refund question returned:
0.8231
which means the model assigned an 82.31% probability to the user explicitly requesting a refund.
Then I got one result that I found quite interesting.
The churn-risk question returned:
0.1494
So the model assigned only 14.94% probability to the user threatening to cancel, even though the text explicitly included:
“If this issue cannot be resolved soon, I may have to cancel my subscription.”
This was a useful reminder that I shouldn't assume the model will interpret every question exactly the way I expect.
The Laya documentation also points out that its noul decision can sometimes be affected by the labels used for the two possible outcomes. So this result is something I would want to investigate rather than simply accepting it as a correct or incorrect verdict.
The longer input changed the results
Comparing this with my earlier short test was also interesting.
The shorter version gave me:
billing → 98.01%
while the longer version gave:
billing → 91.15%
The model still selected the same category, but the probability changed once more context was added.
That's actually something I wanted to see. A longer input isn't automatically going to produce the same confidence as a short one, even when the underlying topic hasn't changed.
For this particular test, the 751-token input was still handled without any problem by the English checkpoint.
At this point, I had tested short inputs, multiple questions, different languages, and a longer support ticket.
So far, I had been giving Laya only a few possible answers.
That works nicely for a simple example like:
billing
technical
other
But real classification systems can have many more categories. A support system, for example, might have separate labels for refunds, card problems, failed transfers, account access, delivery issues, verification, and many other cases.
So I wanted to see how Laya behaves when the number of choices starts increasing.
Instead of changing the whole project, I kept the same setup and created a larger choice question.
I used this:
from laya import Router
router = Router()
state = """
I tried to transfer money to my bank account this morning.
The transfer failed, but I can see the amount has already
been deducted from my account. I want to know when the money
will be returned.
"""
questions = {
"intent": {
"type": "choice",
"instructions": "What is the main reason for this support request?",
"criteria": {
"card_payment": "problem with a card payment",
"cash_withdrawal": "problem withdrawing cash",
"bank_transfer": "problem with a bank transfer",
"refund": "asking for a refund",
"account_access": "cannot access the account",
"verification": "identity or account verification problem",
"subscription": "subscription or recurring payment problem",
"fees": "question about fees or charges",
"exchange_rate": "question about currency exchange rates",
"card_delivery": "problem with card delivery",
"cash_deposit": "problem depositing cash",
"other": "something not covered by the other categories"
}
}
}
result = router.predict(state, questions)
print(result)
The input was:
I tried to transfer money to my bank account this morning.
The transfer failed, but I can see the amount has already
been deducted from my account. I want to know when the money
will be returned.
I then asked Laya to choose the main reason for the request from these categories:
card_payment
cash_withdrawal
bank_transfer
refund
account_access
verification
subscription
fees
exchange_rate
card_delivery
cash_deposit
other
I ran the same command as before:
python test_laya.py
bank_transfer
The result was extremely strong:
bank_transfer → 1.0
The other 11 categories were returned as 0.0 in this particular run.
The reported confidence was:
confidence: 0.9998
answer_confidence: 1.0
It also processed:
input_tokens: 174
output_tokens: 0
So once again, there was no generated text.
I expected adding more options to make the decision noticeably harder, but with 12 choices, Laya still identified the correct category very clearly.
That said, there is an important detail here.
The runtime again showed the warning about an invalid temperature value and said that the affected confidence values were uncalibrated. Because of that, I wouldn't interpret the 1.0 probability as meaning that the model is literally certain or that it will be correct 100% of the time.
What I can say from my test is much simpler:
With this particular 12-option example, Laya selected bank_transfer and assigned virtually all of the probability mass to that category.
This test also showed me why the number of choices matters.
Laya has a fixed budget for representing the options in a question. With 12 categories, there is still enough room for the descriptions to remain distinct.
But what happens when the number jumps from 12 choices to dozens of choices?
The project's own benchmarks show that high-cardinality questions can become a problem. The documentation specifically discusses a 77-option Banking77 test, where the available option budget becomes much tighter.
So my next experiment is going to push this much further.
After my 12-choice test worked, I wanted to push the model much further.
The Laya documentation talks about an important limitation with high-cardinality choice questions. When a question has a large number of options, the model has to fit all of those option descriptions into a fixed part of its input budget. With 50 or more choices, this can make the options harder to distinguish. The project's benchmark specifically uses Banking77, a dataset with 77 banking intents, to demonstrate this limitation.
So I decided to recreate a similar test locally.
Setting up the 77 choices
I used the 77 Banking77 intent names as the choices. Banking77 is a fine-grained banking intent dataset containing 77 categories, including things such as card_arrival, failed_transfer, request_refund, transaction_charged_twice, and verify_my_identity.
For the experiment, I used a straightforward banking query:
I tried to make a bank transfer this morning.
The transfer failed, but the money has already
been deducted from my account.
I then created a choice question containing all 77 categories.
The code looks like this:
from laya import Router
router = Router()
state = """
I tried to make a bank transfer this morning.
The transfer failed, but the money has already
been deducted from my account.
"""
intents = {
"activate_my_card": "activating a card",
"age_limit": "age requirements",
"apple_pay_or_google_pay": "using Apple Pay or Google Pay",
"atm_support": "ATM support",
"automatic_top_up": "automatic top up",
"balance_not_updated_after_bank_transfer": "balance not updated after bank transfer",
"balance_not_updated_after_cheque_or_cash_deposit": "balance not updated after cheque or cash deposit",
"beneficiary_not_allowed": "beneficiary is not allowed",
"cancel_transfer": "cancelling a transfer",
"card_about_to_expire": "card is about to expire",
"card_acceptance": "where a card is accepted",
"card_arrival": "card arrival",
"card_delivery_estimate": "estimated card delivery time",
"card_linking": "linking a card",
"card_not_working": "card not working",
"card_payment_fee_charged": "fee charged for a card payment",
"card_payment_not_recognised": "unrecognised card payment",
"card_payment_wrong_exchange_rate": "wrong exchange rate for card payment",
"card_swallowed": "card retained by an ATM",
"cash_withdrawal_charge": "cash withdrawal charge",
"cash_withdrawal_not_recognised": "unrecognised cash withdrawal",
"change_pin": "changing PIN",
"compromised_card": "compromised card",
"contactless_not_working": "contactless not working",
"country_support": "supported countries",
"declined_card_payment": "declined card payment",
"declined_cash_withdrawal": "declined cash withdrawal",
"declined_transfer": "declined transfer",
"direct_debit_payment_not_recognised": "unrecognised direct debit",
"disposable_card_limits": "disposable card limits",
"edit_personal_details": "editing personal details",
"exchange_charge": "currency exchange charge",
"exchange_rate": "exchange rate",
"exchange_via_app": "exchanging currency through the app",
"extra_charge_on_statement": "extra charge on statement",
"failed_transfer": "failed bank transfer",
"fiat_currency_support": "supported fiat currencies",
"get_disposable_virtual_card": "getting a disposable virtual card",
"get_physical_card": "getting a physical card",
"getting_spare_card": "getting a spare card",
"getting_virtual_card": "getting a virtual card",
"lost_or_stolen_card": "lost or stolen card",
"lost_or_stolen_phone": "lost or stolen phone",
"order_physical_card": "ordering a physical card",
"passcode_forgotten": "forgotten passcode",
"pending_card_payment": "pending card payment",
"pending_cash_withdrawal": "pending cash withdrawal",
"pending_top_up": "pending top up",
"pending_transfer": "pending transfer",
"pin_blocked": "blocked PIN",
"receiving_money": "receiving money",
"refund_not_showing_up": "refund not showing up",
"request_refund": "requesting a refund",
"reverted_card_payment": "reverted card payment",
"supported_cards_and_currencies": "supported cards and currencies",
"terminate_account": "closing the account",
"top_up_by_bank_transfer_charge": "charge for topping up by bank transfer",
"top_up_by_card_charge": "charge for topping up by card",
"top_up_by_cash_or_cheque": "topping up by cash or cheque",
"top_up_failed": "failed top up",
"top_up_limits": "top up limits",
"top_up_reverted": "reverted top up",
"topping_up_by_card": "topping up by card",
"transaction_charged_twice": "transaction charged twice",
"transfer_fee_charged": "fee charged for a transfer",
"transfer_into_account": "transferring money into the account",
"transfer_not_received_by_recipient": "recipient did not receive the transfer",
"transfer_timing": "transfer timing",
"unable_to_verify_identity": "unable to verify identity",
"verify_my_identity": "identity verification",
"verify_source_of_funds": "verifying source of funds",
"verify_top_up": "top up verification",
"virtual_card_not_working": "virtual card not working",
"visa_or_mastercard": "Visa or Mastercard",
"why_verify_identity": "why identity verification is required",
"wrong_amount_of_cash_received": "wrong amount of cash received",
"wrong_exchange_rate_for_cash_withdrawal": "wrong exchange rate for cash withdrawal"
}
questions = {
"intent": {
"type": "choice",
"instructions": "What is the main reason for this banking request?",
"criteria": intents
}
}
result = router.predict(state, questions)
print(result)
This is very different from my earlier 12-choice experiment.
With 77 options, the model has to distinguish between many closely related banking intents. For example:
failed_transfer
declined_transfer
pending_transfer
transfer_not_received_by_recipient
cancel_transfer
transfer_timing
These categories are much closer to each other than something obvious like card_arrival versus exchange_rate.
The Laya documentation says this is where its default option budget becomes a real constraint. In the documented Banking77 comparison, Laya's 77-choice result was substantially lower than its performance on smaller choice sets, and the authors explain that the option descriptions can become too compressed within the fixed head budget.
python test_laya.py
The result surprised me
failed_transfer
That was the category I would expect from this particular message.
failed_transfer → 0.9753
declined_transfer → 0.0056
unable_to_verify_identity → 0.0191
Most of the remaining categories were returned as 0.0 in this particular run.
The reported values were:
confidence: 0.9703
answer_confidence: 0.9753
The input contained 350 tokens, while the output was again:
output_tokens: 0
So even with 77 possible choices, Laya still didn't generate any text.
I expected the larger choice set to make the result obviously worse, but that didn't happen with this particular example.
Laya still selected the correct-looking category and put most of the probability on it.
However, this doesn't mean that Laya solves 77-choice classification reliably in general.
The reason is that this was one manually constructed test. The official project benchmarks report a much more difficult Banking77 evaluation where the model's performance drops substantially with a large number of options. The project explains that the option descriptions have to share a limited token budget, which becomes increasingly restrictive as the number of choices grows.
So my local experiment gave me one useful result:
77 choices did not break the model on this example.
But it is not enough to claim that Laya performs well across a full 77-class benchmark.
That's an important distinction, especially when writing about model experiments.
The runtime continued to display the same temperature warning:
this checkpoint ships invalid temperatures...
Treat confidence from the affected entries as uncalibrated.
So again, I'm treating the probabilities as the output of this particular run, not as a guarantee of real-world accuracy.
At this point, I had a much better idea of how Laya behaves with different input sizes and different numbers of choices.
Up to this point, I had been using the default Router:
router = Router()
With that setup, Laya loads a checkpoint when it is needed. That works well when you're experimenting because you don't have to load every model into memory at startup.
For the next test, I wanted to try a different approach.
Laya also supports pre, which means the checkpoints are loaded when the Router starts.
I changed my code to:
from laya import Router
router = Router(preload=True)
state = """
I tried to make a bank transfer this morning.
The transfer failed, but the money has already
been deducted from my account.
"""
questions = {
"intent": {
"type": "choice",
"instructions": "What is the main reason for this banking request?",
"criteria": {
"card_payment": "problem with a card payment",
"cash_withdrawal": "problem withdrawing cash",
"bank_transfer": "problem with a bank transfer",
"refund": "asking for a refund",
"account_access": "cannot access the account",
"verification": "identity or account verification problem",
"subscription": "subscription or recurring payment problem",
"fees": "question about fees or charges",
"exchange_rate": "question about currency exchange rates",
"card_delivery": "problem with card delivery",
"cash_deposit": "problem depositing cash",
"other": "something not covered by the other categories"
}
}
}
result = router.predict(state, questions)
print("Selected intent:", result["answers"]["intent"]["choice"])
print("Probabilities:", result["answers"]["intent"]["probabilities"])
print("Confidence:", result["answers"]["intent"]["confidence"])
print("Routing model:", result["routing"]["model"])
print("Routing reason:", result["routing"]["reason"])
print("Input tokens:", result["usage"]["input_tokens"])
print("Output tokens:", result["usage"]["output_tokens"])
I then ran:
python test_laya.py
The result
The prediction was:
Selected intent: failed_transfer
Laya assigned:
failed_transfer → 0.9753
The other notable probabilities were:
declined_transfer → 0.0056
unable_to_verify_identity → 0.0191
0.9703
The request contained 350 input tokens, while the output was again:
Output tokens: 0
The Router also selected:
Routing model: english
with the reason:
English Latin text
This time, the terminal didn't download the model again.
The fetching step showed:
5/5
00:00
and the download/reconstruction both reported 0.00B.
That makes sense because the checkpoint had already been downloaded during my earlier experiments. Laya was using the cached files instead of down another copy.
The important thing to understand is that preload=True doesn't change the kind of answer Laya produces. It changes how the model is prepared for inference.
With the normal setup:
Request
↓
Load checkpoint if needed
↓
Prediction
With preload enabled:
Application starts
↓
Checkpoint(s) loaded
↓
Request arrives
↓
Prediction
That makes more sense for something that is going to stay running, such as a local API, a support-ticket classifier, or another application making repeated decisions.
There is a trade-off, though. Keeping models ready means using more memory. So preload mode is more useful when the machine has enough RAM to keep the required checkpoints resident.
For my Mac mini, the interesting thing was that Laya was already operating in a completely different memory class from the large language models I normally experiment with.
I didn't measure the exact RAM usage in this run, so I'm not going to make up a number. What I can confirm is that the preloaded setup ran successfully on my M4 Mac mini with 16GB unified memory.
And at this point, I had tested Laya with short text, long text, multiple decisions, Hindi input, and a wide choice set.
After running several different tests, I wanted to check something more practical.
I already knew the model files were relatively small, but I wanted to see what they actually looked like on my Mac rather than relying only on the numbers from the model page.
There are two different things to measure here:
Storage — how much space the downloaded checkpoints occupy on the SSD.
Memory — how much RAM the Python process needs while Laya is running.
These are easy to confuse, because an 800 MB model file does not necessarily mean the application will use exactly 800 MB of memory.
Hugging Face keeps downloaded model files in its local cache. I checked the cache with:
du -sh ~/.cache/huggingface/hub
To look specifically for Laya models:
du -sh ~/.cache/huggingface/hub/models--convaiinnovations--*
Depending on how the cache is configured on the Mac, the files may be stored in a different location. You can check that with:
echo $HF_HOME
If nothing is printed, the standard Hugging Face cache location is normally used.
For the memory side, I used macOS's built-in /usr/bin/time utility.
Instead of:
python test_laya.py
I ran:
/usr/bin/time -l python test_laya.py
At the end of the output, macOS reports:
maximum resident set size
That gives me the peak memory used by the Python process during the run.
This is a much more useful number for my blog than simply saying:
“The model is 846 MB.”
The model file and the actual runtime memory are two different measurements.
After testing Laya from the terminal, I wanted to make the experiment a little more practical.
Running a Python script every time works, but I wanted a small interface where I could paste a customer message, click a button, and immediately see what Laya thinks about it.
So I added Gradio on top of the same Laya code.
The flow now looks like this:
Browser
↓
Gradio
↓
Laya Router
↓
Structured decisions
I kept the interface simple. I wanted to see the department, refund request, churn risk, routing information, and token usage without having to read a large JSON response every time.
Gradio is a simple way to do that.
Inside the same virtual environment:
python -m pip install gradio
Create another file:
touch app.py
Then open it:
nano app.py
Add this:
import gradio as gr
from laya import Router
router = Router(preload=True)
def predict(text):
if not text.strip():
return {
"error": "Please enter some text."
}
questions = {
"department": {
"type": "choice",
"instructions": "Which department should handle this?",
"criteria": {
"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"
}
},
"refund_requested": {
"type": "noul",
"instructions": "Does the user explicitly request a refund?"
},
"churn_risk": {
"type": "noul",
"instructions": "Does the user threaten to cancel or leave?"
}
}
result = router.predict(text, questions)
return {
"department": result["answers"]["department"],
"refund_requested": result["answers"]["refund_requested"],
"churn_risk": result["answers"]["churn_risk"],
"routing": result["routing"],
"usage": result["usage"]
}
with gr.Blocks(title="Laya Decision Demo") as demo:
gr.Markdown("# Laya Decision Demo")
gr.Markdown(
"Enter a message and let Laya make structured decisions."
)
text_input = gr.Textbox(
label="Input",
placeholder="Example: I was charged twice and want a refund.",
lines=6
)
run_button = gr.Button("Run Laya")
output = gr.JSON(label="Laya Result")
run_button.click(
fn=predict,
inputs=text_input,
outputs=output
)
if __name__ == "__main__":
demo.launch()
Now start it with:
python app.py
Gradio will start a local web server and give you a local URL, typically something like:
http://127.0.0.1:7860
Open that URL in your browser.
You'll get a simple interface where you can enter something like:
Hi, I was charged twice for my March subscription.
Please refund the duplicate payment.
and click:
Run Laya
The UI will then show the structured result returned by Laya.
For one of my tests, I entered a customer-support message and got this result:
Department: billing
Refund requested: 0.8829
Churn risk: 0.0916
Model: english
Input tokens: 161
Output tokens: 0
The department probabilities were:
billing → 0.9637
technical → 0.0185
other → 0.0178
The refund decision returned:
0.8829
And the churn-risk value was:
0.0916
One thing I found particularly interesting was the token count:
input_tokens: 161
output_tokens: 0
Again, Laya wasn't generating a paragraph in response to my input. The interface was simply displaying the structured decisions produced by the model.
The Router also identified the input as English:
model: english
reason: English Latin text
After playing with Laya locally, I wanted to look beyond my own small experiments and see how the different checkpoints perform on larger benchmark runs.
The project currently has three checkpoints:
| Checkpoint | Parameters | Context | Main use |
| ------------------------ | ---------: | -----------------: | ------------------------ |
| `convaiinnovations/laya` | 421M | 512 | English |
| `laya-multilingual` | 322M | 1,024, up to 8,192 | 100+ languages |
| `laya-typed-decisions` | 421M | 1,024 | Typed decision workflows |
The first thing that stands out is how small these models are.
The main Laya checkpoint is only 421 million parameters, and the multilingual version is even smaller at 322 million. That's a very different scale from the multi-billion-parameter models I normally think about when talking about local AI.
The project's benchmark runs on a Tesla T4 show a few interesting numbers.
For typed decisions, the fine-tuned laya-typed-decisions checkpoint reached 0.766 accuracy across 2,000 decisions.
The base English and multilingual checkpoints were much lower on this particular benchmark:
laya → 0.362
laya-multilingual → 0.342
laya-typed-decisions → 0.766
That difference is important.
It tells me that the impressive 0.766 number shouldn't be interpreted as what the base Laya model does out of the box. The result comes from a checkpoint that was specifically fine-tuned for those typed-decision workflows.
That's actually one of the more useful lessons from the benchmarks: fine-tuning matters a lot for this kind of model.
The speed numbers are also interesting.
On a Tesla T4, the project reports:
| Questions in one call | Laya | Multilingual |
| --------------------: | -------: | -----------: |
| 1 | 39.5 ms | 32.8 ms |
| 5 | 84.5 ms | 40.1 ms |
| 10 | 158.6 ms | 72.3 ms |
| 50 | 771 ms | 337 ms |
Because multiple questions can be answered in one forward pass, the cost per question drops as the batch gets larger.
For example, the multilingual checkpoint reaches about 7.2 ms per question when 10 questions are processed together.
That's one of the places where the smaller decision-oriented architecture starts to make sense.
Laya benchmark results across accuracy, latency, calibration, multilingual performance, and application workflows.
One of the things I was particularly interested in was how Laya behaves when the input gets much longer.
The multilingual checkpoint can be configured with:
max_len=8192
The project's long-context experiment placed the request at the end of documents with different amounts of unrelated text before it.
The results were:
| Text before request | Correct | Time per request |
| ------------------- | ------: | ---------------: |
| Short input | 19 / 20 | 0.02 s |
| ~1,000 tokens | 16 / 20 | 0.21 s |
| ~2,000 tokens | 17 / 20 | 0.50 s |
| ~3,000 tokens | 17 / 20 | 0.90 s |
| ~4,000 tokens | 18 / 20 | 1.71 s |
| ~5,000 tokens | 11 / 20 | 2.59 s |
| ~6,000 tokens | 17 / 20 | 3.50 s |
| ~7,000 tokens | 8 / 20 | 4.50 s |
The results aren't perfectly smooth, but they show the basic trade-off clearly: longer context costs more time, and accuracy can become more variable as the document gets very long.
That matches what I saw in my own test earlier. My 751-token support ticket worked without any issue, but that doesn't mean I should throw an enormous document at the model and expect the same behavior.
For a real application, I would test the exact input lengths and document types I care about.
Long-context test for laya-multilingual with requests placed at different positions inside longer documents.
After all the experiments, I think the most useful way to look at Laya is not as a replacement for a large language model.
It's a different tool.
The project reports strong results on some tasks. For example, its benchmark suite reports around 0.993 accuracy for email spam and phishing on the tested data, while several other workflows are considerably weaker.
The same benchmark also shows an important limitation with large choice sets.
On Banking77, which has 77 labels, the benchmark reports:
Laya → 0.425
TypeSafe Jev → 0.870
The project attributes Laya's result to the way the available token budget is shared across the many option descriptions.
That matches something I started noticing during my own experiments: three or twelve choices are very different from seventy-seven choices.
So I wouldn't design a system around hundreds of labels in one giant choice question without testing it carefully.
There are a few other things I would keep in mind too.
The base checkpoints are not automatically excellent at every zero-shot decision task. The model's probability outputs can also require calibration, and some decision types perform better than others.
For me, that's actually a positive part of the project rather than something to hide.
The documentation is fairly open about where the model works and where it needs more care.
After running all these tests, this is the comparison that makes the most sense to me.
A traditional LLM generally looks like:
Input
↓
Large language model
↓
Generate tokens
↓
Text response
Laya looks more like:
Input
↓
Laya
↓
Typed decision
↓
Probability
That's a much narrower job.
I wouldn't use Laya to write a blog post, generate code, summarize a long meeting, or have a conversation.
But I could imagine using something like Laya for the smaller decisions that happen around those tasks.
User request
↓
Laya
↓
Is this relevant?
↓
Is it urgent?
↓
Which workflow?
↓
LLM / Tool
In other words, the interesting idea isn't:
“Laya can replace my LLM.”
For me, it's:
“Maybe my LLM doesn't need to make every decision.”
I started this experiment because Laya looked unusual.
Most of the local models I come across are trying to become better at generation: better coding, better reasoning, better conversations, better multimodal capabilities.
Laya went in a different direction.
It takes a relatively small model, gives it a structured decision problem, and focuses on making that decision quickly.
I tested it on my M4 Mac mini, ran the English checkpoint, switched to the multilingual model with Hindi input, asked multiple questions against the same text, tried a longer support ticket, increased the number of choices, and finally put the whole thing behind a small Gradio interface.
What surprised me most was how little infrastructure was needed to get from zero to a working local demo.
I didn't need a giant GPU server.
I didn't need an API key for inference.
I didn't need to run a multi-billion-parameter model.
I just installed the package, downloaded the checkpoint, and started asking it structured questions.
Maybe Every AI System Doesn't Need One Giant Model
After spending so much time experimenting with large language models, I found Laya refreshing simply because it isn't trying to be one.
It doesn't generate a long answer.
It doesn't try to write code.
It doesn't try to become a chatbot.
It makes decisions.
Running a 421M-parameter model locally on my M4 Mac mini made that idea feel much more practical to me.
For something like customer-support routing, moderation, classification, guardrails, or other structured decisions, using a smaller specialized model can make more sense than sending every single request through a large generative model.
At the same time, my experiments also showed that Laya isn't magic. Its performance depends heavily on the task, the number of choices, the input size, and whether the model has been fine-tuned for the problem. The confidence values also need to be treated carefully rather than taken as guaranteed accuracy.
So I wouldn't look at Laya and ask:
“Is this the next replacement for Llama or Qwen?”
That's the wrong comparison.
The more interesting question for me is:
What if we stop expecting one model to do everything?
A large language model can handle generation and reasoning.
A small decision model can handle a narrow decision.
A routing model can decide which path to take.
And an application can combine all of them.
That feels like a much more interesting direction for local AI.
Laya may be small, but the idea behind it is much bigger: sometimes the best model for a job isn't the model that can do everything — it's the one designed to do that one job well.