Before You Invent a New Architecture, Try a Bigger Encoder

Independent developer Arun P. released Yama (arunp0/yama-indic), an open encoder model that returns a calibrated probability for every user-defined label in a single forward pass, succeeding the earlier laya-indic model. Yama scored 0.800 on the 67-task benchmark used by its model card with an mmBERT-base encoder, and the author reports that swapping in XLM-RoBERTa-large on the same training data outperformed attempts to raise the score through targeted data (0.804 to 0.809), looped transformer heads (0.806) and looped encoder layers (0.808). The model is aimed at Indian-language text classification, letting users write questions in English while leaving the input in its own language and script, and is distributed via the thx01 package.

Disclosure: this article was drafted with help from an AI assistant, and the charts were made with AI-assisted code from my own results. The model, experiments, results and decisions are my own. Every number was checked against the model card or my training logs. Yama returns a calibrated probability for every label you define. Image by the author. TL;DR Here is a typical customer message: Kal se mera UPI payment fail ho raha hai, paise kat gaye par merchant ko nahi mile “My UPI payment has been failing since yesterday; the money was debited but the merchant never got it.” The questions you want answered change all the time. Today: is this a failed payment, a refund request or a login problem? Is it urgent? How annoyed is the customer? Tomorrow: the same kind of questions about Tamil product reviews or Bengali headlines, with different labels. The usual tools all have a catch for Indian languages: I wanted one small, open model that takes the question and labels at inference time and returns one calibrated probability per label in a single forward pass. “Calibrated” means that when it says 0.8, it is right about 80% of the time. That model is Yama. It succeeds my earlier model, laya-indic, which is still available. python import thx01 pip install thx01agent = thx01.load "arunp0/yama-indic" text = "Kal se mera UPI payment fail ho raha hai, paise kat gaye par merchant ko nahi mile"out = agent.system one text, { "topic": {"type": "choice", "instructions": "What is the customer writing about?", "criteria": {"failed payment": "payment failed or money debited but not received", "refund": "asking for a refund", "account": "login or account access", "other": None}}, "urgent": {"type": "noul", "instructions": "Does the customer need help urgently?"}, "anger": {"type": "score", "instructions": "How frustrated is the customer?", "criteria": "calm", "mildly annoyed", "clearly frustrated", "furious" },} print out "answers" "topic" "probabilities" {'failed payment': 0.9629, 'refund': 0.0125, 'account': 0.0123, 'other': 0.0124} other answers from the same call: urgent P yes = 0.74, anger "clearly frustrated" 0.88 That is real output. One call answered three kinds of question: Write the questions in English and leave the text in its own language and script. Yama covers: It is an encoder, not a text generator, so it is fast and deterministic: The first two versions used mmBERT-base 5 as the encoder, and v2 reached 0.800. Every number in this section is on the same 67-task benchmark the model card uses; the intermediate runs are internal and not on the card. Here is what I tried to push that score higher. Targeted data. Reasoning was the weakest area. I added 15k cause/effect and passage yes/no questions, written by Gemma 4 from real Indian-language text. Under the same three-epoch schedule, the overall score went from 0.804 without them to 0.809 with them. The two tasks they targeted barely moved. Looped transformers in the head. I let the decision head take several passes over its own output and let the options attend to each other. The new blocks sat behind gates initialised to zero, so training started exactly at the baseline. The result was 0.806, against 0.809 for the same model trained for the same extra epoch without loops. The gates learned to stay almost closed. Looped transformers in the encoder. I re-ran the top six encoder layers three times. The result was 0.808 against 0.809. Again, the gates barely opened. One caveat: I added the loops to an already-trained model at a low learning rate, which favours the existing weights. Trained from scratch, they might behave differently. A bigger encoder. At this point the pattern suggested that depth and wiring weren’t the bottleneck; capacity was: So I trained XLM-R-large on the same data. mmBERT-base and XLM-RoBERTa-large on the same training data, both after three epochs. The subtitle shows where v3’s six points came from. These are internal ablation runs; only v2 and v3 are on the model card. Chart by the author. The settings weren’t perfectly matched: the XLM-R run needed bf16 and a lower learning rate more on that below . One difference favours mmBERT: its encoder had extra Indian-language pretraining, and XLM-R’s did not. IndicCOPA went from 0.531, barely above the 0.5 coin-flip line, to 0.624. 1. Check the encoder before you invent architecture. Everything clever I tried was worth less than a point. The swap was worth more than five. I’d now run the bigger-encoder baseline first. 2. Big encoders can fail silently. My first XLM-R run collapsed. Around step 28,000 the running loss roughly doubled, from about 0.65 to 1.35, and never recovered. Large encoders often overflow in fp16, the 16-bit format I was using; bf16 has a wider range. Switching to bf16 and halving the learning rate fixed it. I also added a guard that rolls back and lowers the learning rate on a loss spike. It never had to fire, but I’d keep it. 3. Continued pretraining helped the small encoder, not the large one. Extra masked-language-model training on Indian-language text 7 gave mmBERT about +0.7 points in a pilot. On XLM-R, it cut that pretraining loss by 6–14% in every language, but the downstream result was a tie 0.861 against 0.862 . My guess is that XLM-R had already seen plenty of Indian-language text. I scored Yama and four popular open zero-shot classifiers with the same harness on the same test samples. The headline set contains only datasets Yama never saw any data from. That is 33 pick-one-label tasks: Zero-shot accuracy on 33 tasks from datasets Yama never trained on it did train on similar task types . On the 22 Indian-language tasks alone: Yama 0.813, bge-m3-zeroshot-v2.0 0.753. Chart by the author. The closest competitor, bge-m3-zeroshot-v2.0, is the same size class 568M . Yama beats it on 29 of the 33 tasks, loses on 3 AG News, Emotion and Marathi IndicCOPA and ties on 1. For a rough sense of scale I also asked an LLM the same questions zero-shot: Gemma 4, a 26B mixture-of-experts model with 4B active parameters 12 . On a slightly different 64-task set, with 200-sample subsets, it scored 0.850 against Yama’s 0.855. Treat that as “roughly level”, not a ranking. Benchmark tables are easy to tilt, so here is what to keep in mind: The model card lists every per-task number. How Yama scores a question simplified . Architecture and training method from THX-01 13 . Diagram by the author. Yama uses the THX-01 decision architecture, its training method and its open-source thx01 library 13 . It does not use THX-01's weights, and I'm not affiliated with its authors. The labels are part of the input, so a new label set needs no retraining. Yes/no questions get two markers “false” and “true” , and ratings get one marker per level. Training uses THX-01’s RLCD method 13 : Expected calibration error is 0.070 on the 67-task benchmark and 0.092 on the unseen tasks above. All training data is public and permissively licensed: CC0, CC-BY, Apache-2.0 or MIT, with no ShareAlike or NonCommercial sources. As far as I can verify, none of it is output from proprietary models. The one exception I couldn’t fully check is NVIDIA’s Nemotron Safety Guard v3 16 , which doesn’t name the models used to generate it. The training set has four parts: I built in one experiment. Telugu and Gujarati were left out of the cause/effect questions, and Hindi out of the yes/no questions. Any gain in those languages would then be transfer, not format memorisation. The result was mixed: All training text was deduplicated against every test set by normalised exact match. The new data is published as arunp0/yama-indic-gemma https://huggingface.co/datasets/arunp0/yama-indic-gemma . In Indian tradition Yama is also Dharmaraja , the judge who weighs what is placed before him and gives a verdict. A model that weighs the options you give it and returns a verdict with a confidence seemed a fair fit. This is a personal project. If you try it on your own data, tell me where it breaks. That’s the most useful feedback I can get. Thanks to AI4Bharat for Sangraha, to the THX-01 authors for the architecture, training method and thx01 library, and to the teams behind XLM-RoBERTa, Qwen and Gemma for releasing open weights. 1 W. Yin et al. Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. EMNLP 2019. https://arxiv.org/abs/1909.00161 https://arxiv.org/abs/1909.00161 2 M. Lewis et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL 2020. https://arxiv.org/abs/1910.13461 https://arxiv.org/abs/1910.13461 3 M. Laurer et al. Building Efficient Universal Classifiers with Natural Language Inference. 2023. https://arxiv.org/abs/2312.17543 https://arxiv.org/abs/2312.17543 4 J. Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. Findings of ACL 2024. https://arxiv.org/abs/2402.03216 https://arxiv.org/abs/2402.03216 5 M. Marone et al. mmBERT: A Modern Multilingual Encoder with Annealed Language Learning. 2025. https://arxiv.org/abs/2509.06888 https://arxiv.org/abs/2509.06888 6 A. Conneau et al. Unsupervised Cross-lingual Representation Learning at Scale. ACL 2020. https://arxiv.org/abs/1911.02116 https://arxiv.org/abs/1911.02116 7 S. Gururangan et al. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL 2020. https://arxiv.org/abs/2004.10964 https://arxiv.org/abs/2004.10964 8 D. I. Adelani et al. SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects. EACL 2024. https://arxiv.org/abs/2309.07445 https://arxiv.org/abs/2309.07445 9 S. Doddapaneni et al. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. ACL 2023. https://arxiv.org/abs/2212.05409 https://arxiv.org/abs/2212.05409 10 X. Zhang, J. Zhao and Y. LeCun. Character-level Convolutional Networks for Text Classification. NeurIPS 2015. https://arxiv.org/abs/1509.01626 https://arxiv.org/abs/1509.01626 11 E. Saravia et al. CARER: Contextualized Affect Representations for Emotion Recognition. EMNLP 2018. https://aclanthology.org/D18-1404/ https://aclanthology.org/D18-1404/ 12 Gemma 4 26B-A4B instruct, model card. https://huggingface.co/google/gemma-4-26B-A4B-it https://huggingface.co/google/gemma-4-26B-A4B-it 13 THX-01 decision model, RLCD training method and thx01 library. HAL-X AI. https://huggingface.co/doofz/THX-01 https://huggingface.co/doofz/THX-01 14 T. Gneiting and A. E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. JASA, 2007. 15 C. Guo et al. On Calibration of Modern Neural Networks. ICML 2017. https://arxiv.org/abs/1706.04599 https://arxiv.org/abs/1706.04599 16 NVIDIA Nemotron Safety Guard dataset v3. https://huggingface.co/datasets/nvidia/Nemotron-Safety-Guard-Dataset-v3 https://huggingface.co/datasets/nvidia/Nemotron-Safety-Guard-Dataset-v3 17 J. FitzGerald et al. MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages. ACL 2023. https://arxiv.org/abs/2204.08582 https://arxiv.org/abs/2204.08582 18 A. Yang et al. Qwen3 Technical Report. 2025. https://arxiv.org/abs/2505.09388 https://arxiv.org/abs/2505.09388 19 M. S. U. R. Khan et al. IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages. ACL 2024. https://arxiv.org/abs/2403.06350 https://arxiv.org/abs/2403.06350 20 Qwen3.6–35B-A3B, model card. https://huggingface.co/Qwen/Qwen3.6-35B-A3B https://huggingface.co/Qwen/Qwen3.6-35B-A3B Before You Invent a New Architecture, Try a Bigger Encoder https://pub.towardsai.net/before-you-invent-a-new-architecture-try-a-bigger-encoder-8f2b39fc50a8 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.