# Before You Invent a New Architecture, Try a Bigger Encoder

> Source: <https://pub.towardsai.net/before-you-invent-a-new-architecture-try-a-bigger-encoder-8f2b39fc50a8?source=rss----98111c9905da---4>
> Published: 2026-10-22 15:31:02+00:00

*Disclosure: this article was drafted with help from an AI assistant, and the charts were made with AI-assisted code from my own results. The model, experiments, results and decisions are my own. Every number was checked against the model card or my training logs.*

*Yama returns a calibrated probability for every label you define. Image by the author.*

**TL;DR**

Here is a typical customer message:

*Kal se mera UPI payment fail ho raha hai, paise kat gaye par merchant ko nahi mile*

(“My UPI payment has been failing since yesterday; the money was debited but the merchant never got it.”)

The questions you want answered change all the time. Today: is this a failed payment, a refund request or a login problem? Is it urgent? How annoyed is the customer? Tomorrow: the same kind of questions about Tamil product reviews or Bengali headlines, with different labels.

The usual tools all have a catch for Indian languages:

I wanted one small, open model that takes the question and labels at inference time and returns one calibrated probability per label in a single forward pass. “Calibrated” means that when it says 0.8, it is right about 80% of the time. That model is Yama. It succeeds my earlier model, laya-indic, which is still available.

``` python
import thx01   # pip install thx01agent = thx01.load("arunp0/yama-indic")
text = "Kal se mera UPI payment fail ho raha hai, paise kat gaye par merchant ko nahi mile"out = agent.system_one(text, {    "topic":  {"type": "choice", "instructions": "What is the customer writing about?",               "criteria": {"failed_payment": "payment failed or money debited but not received",                            "refund": "asking for a refund", "account": "login or account access", "other": None}},    "urgent": {"type": "noul", "instructions": "Does the customer need help urgently?"},    "anger":  {"type": "score", "instructions": "How frustrated is the customer?",               "criteria": ["calm", "mildly annoyed", "clearly frustrated", "furious"]},})print(out["answers"]["topic"]["probabilities"])# {'failed_payment': 0.9629, 'refund': 0.0125, 'account': 0.0123, 'other': 0.0124}# other answers from the same call: urgent P(yes) = 0.74, anger "clearly frustrated" 0.88
```

That is real output. One call answered three kinds of question:

Write the questions in English and leave the text in its own language and script. Yama covers:

It is an encoder, not a text generator, so it is fast and deterministic:

The first two versions used mmBERT-base [5] as the encoder, and v2 reached 0.800. Every number in this section is on the same 67-task benchmark the model card uses; the intermediate runs are internal and not on the card. Here is what I tried to push that score higher.

**Targeted data.** Reasoning was the weakest area. I added 15k cause/effect and passage yes/no questions, written by Gemma 4 from real Indian-language text. Under the same three-epoch schedule, the overall score went from 0.804 without them to 0.809 with them. The two tasks they targeted barely moved.

**Looped transformers in the head.** I let the decision head take several passes over its own output and let the options attend to each other. The new blocks sat behind gates initialised to zero, so training started exactly at the baseline. The result was 0.806, against 0.809 for the same model trained for the same extra epoch without loops. The gates learned to stay almost closed.

**Looped transformers in the encoder.** I re-ran the top six encoder layers three times. The result was 0.808 against 0.809. Again, the gates barely opened.

One caveat: I added the loops to an already-trained model at a low learning rate, which favours the existing weights. Trained from scratch, they might behave differently.

**A bigger encoder.** At this point the pattern suggested that depth and wiring weren’t the bottleneck; capacity was:

So I trained XLM-R-large on the same data.

*mmBERT-base and XLM-RoBERTa-large on the same training data, both after three epochs. The subtitle shows where v3’s six points came from. These are internal ablation runs; only v2 and v3 are on the model card. Chart by the author.*

The settings weren’t perfectly matched: the XLM-R run needed bf16 and a lower learning rate (more on that below). One difference favours mmBERT: its encoder had extra Indian-language pretraining, and XLM-R’s did not. IndicCOPA went from 0.531, barely above the 0.5 coin-flip line, to 0.624.

**1. Check the encoder before you invent architecture.** Everything clever I tried was worth less than a point. The swap was worth more than five. I’d now run the bigger-encoder baseline first.

**2. Big encoders can fail silently.** My first XLM-R run collapsed. Around step 28,000 the running loss roughly doubled, from about 0.65 to 1.35, and never recovered. Large encoders often overflow in fp16, the 16-bit format I was using; bf16 has a wider range. Switching to bf16 and halving the learning rate fixed it. I also added a guard that rolls back and lowers the learning rate on a loss spike. It never had to fire, but I’d keep it.

**3. Continued pretraining helped the small encoder, not the large one.** Extra masked-language-model training on Indian-language text [7] gave mmBERT about +0.7 points in a pilot. On XLM-R, it cut that pretraining loss by 6–14% in every language, but the downstream result was a tie (0.861 against 0.862). My guess is that XLM-R had already seen plenty of Indian-language text.

I scored Yama and four popular open zero-shot classifiers with the same harness on the same test samples. The headline set contains only datasets Yama never saw any data from. That is 33 pick-one-label tasks:

*Zero-shot accuracy on 33 tasks from datasets Yama never trained on (it did train on similar task types). On the 22 Indian-language tasks alone: Yama 0.813, bge-m3-zeroshot-v2.0 0.753. Chart by the author.*

The closest competitor, bge-m3-zeroshot-v2.0, is the same size class (568M). Yama beats it on 29 of the 33 tasks, loses on 3 (AG News, Emotion and Marathi IndicCOPA) and ties on 1.

For a rough sense of scale I also asked an LLM the same questions zero-shot: Gemma 4, a 26B mixture-of-experts model with 4B active parameters [12]. On a slightly different 64-task set, with 200-sample subsets, it scored 0.850 against Yama’s 0.855. Treat that as “roughly level”, not a ranking.

Benchmark tables are easy to tilt, so here is what to keep in mind:

The model card lists every per-task number.

*How Yama scores a question (simplified). Architecture and training method from THX-01 [13]. Diagram by the author.*

Yama uses the THX-01 decision architecture, its training method and its open-source thx01 library [13]. It does not use THX-01's weights, and I'm not affiliated with its authors.

The labels are part of the input, so a new label set needs no retraining. Yes/no questions get two markers (“false” and “true”), and ratings get one marker per level.

Training uses THX-01’s RLCD method [13]:

Expected calibration error is 0.070 on the 67-task benchmark and 0.092 on the unseen tasks above.

All training data is public and permissively licensed: CC0, CC-BY, Apache-2.0 or MIT, with no ShareAlike or NonCommercial sources. As far as I can verify, none of it is output from proprietary models. The one exception I couldn’t fully check is NVIDIA’s Nemotron Safety Guard v3 [16], which doesn’t name the models used to generate it.

The training set has four parts:

I built in one experiment. Telugu and Gujarati were left out of the cause/effect questions, and Hindi out of the yes/no questions. Any gain in those languages would then be transfer, not format memorisation. The result was mixed:

All training text was deduplicated against every test set by normalised exact match. The new data is published as [arunp0/yama-indic-gemma](https://huggingface.co/datasets/arunp0/yama-indic-gemma).

In Indian tradition Yama is also *Dharmaraja*, the judge who weighs what is placed before him and gives a verdict. A model that weighs the options you give it and returns a verdict with a confidence seemed a fair fit.

This is a personal project. If you try it on your own data, tell me where it breaks. That’s the most useful feedback I can get.

*Thanks to AI4Bharat for Sangraha, to the THX-01 authors for the architecture, training method and* *thx01 library, and to the teams behind XLM-RoBERTa, Qwen and Gemma for releasing open weights.*

[1] W. Yin et al. Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. EMNLP 2019. [https://arxiv.org/abs/1909.00161](https://arxiv.org/abs/1909.00161)

[2] M. Lewis et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL 2020. [https://arxiv.org/abs/1910.13461](https://arxiv.org/abs/1910.13461)

[3] M. Laurer et al. Building Efficient Universal Classifiers with Natural Language Inference. 2023. [https://arxiv.org/abs/2312.17543](https://arxiv.org/abs/2312.17543)

[4] J. Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. Findings of ACL 2024. [https://arxiv.org/abs/2402.03216](https://arxiv.org/abs/2402.03216)

[5] M. Marone et al. mmBERT: A Modern Multilingual Encoder with Annealed Language Learning. 2025. [https://arxiv.org/abs/2509.06888](https://arxiv.org/abs/2509.06888)

[6] A. Conneau et al. Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020. [https://arxiv.org/abs/1911.02116](https://arxiv.org/abs/1911.02116)

[7] S. Gururangan et al. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL 2020. [https://arxiv.org/abs/2004.10964](https://arxiv.org/abs/2004.10964)

[8] D. I. Adelani et al. SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. EACL 2024. [https://arxiv.org/abs/2309.07445](https://arxiv.org/abs/2309.07445)

[9] S. Doddapaneni et al. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. ACL 2023. [https://arxiv.org/abs/2212.05409](https://arxiv.org/abs/2212.05409)

[10] X. Zhang, J. Zhao and Y. LeCun. Character-level Convolutional Networks for Text Classification. NeurIPS 2015. [https://arxiv.org/abs/1509.01626](https://arxiv.org/abs/1509.01626)

[11] E. Saravia et al. CARER: Contextualized Affect Representations for Emotion Recognition. EMNLP 2018. [https://aclanthology.org/D18-1404/](https://aclanthology.org/D18-1404/)

[12] Gemma 4 26B-A4B instruct, model card. [https://huggingface.co/google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it)

[13] THX-01 decision model, RLCD training method and thx01 library. HAL-X AI. [https://huggingface.co/doofz/THX-01](https://huggingface.co/doofz/THX-01)

[14] T. Gneiting and A. E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. JASA, 2007.

[15] C. Guo et al. On Calibration of Modern Neural Networks. ICML 2017. [https://arxiv.org/abs/1706.04599](https://arxiv.org/abs/1706.04599)

[16] NVIDIA Nemotron Safety Guard dataset v3. [https://huggingface.co/datasets/nvidia/Nemotron-Safety-Guard-Dataset-v3](https://huggingface.co/datasets/nvidia/Nemotron-Safety-Guard-Dataset-v3)

[17] J. FitzGerald et al. MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages. ACL 2023. [https://arxiv.org/abs/2204.08582](https://arxiv.org/abs/2204.08582)

[18] A. Yang et al. Qwen3 Technical Report. 2025. [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388)

[19] M. S. U. R. Khan et al. IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages. ACL 2024. [https://arxiv.org/abs/2403.06350](https://arxiv.org/abs/2403.06350)

[20] Qwen3.6–35B-A3B, model card. [https://huggingface.co/Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)

[Before You Invent a New Architecture, Try a Bigger Encoder](https://pub.towardsai.net/before-you-invent-a-new-architecture-try-a-bigger-encoder-8f2b39fc50a8) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.