Training a Tiny Language Model to Follow Prompts Across 6 Low Resource Lanuages

BULaMU creator mwebazarick released a 47M-parameter language model fine-tuned to generate stories across six low-resource East African languages plus English, trained on a corpus expanded tenfold to 40 billion tokens. The expanded corpus adds Swahili, Lao, Amharic, Somali, Kinyarwanda and English to Luganda, with East African languages making up nearly 90% of the new training data. The fine-tuned model, BULaMU-3-42M-Story, is published on Hugging Face with a Streamlit demo, and the BULaMU family (20M, 47M and 110M parameters) is designed to run offline on a phone without a GPU or internet connection.

Hi Everybody, I am just writing to share some progress I have made on my project. I am the creator of BULaMU, a family of language models 20M, 47M, 110M trained completely from scratch for a low resource language: Luganda. The models are small and compute-efficient enough to run offline on a phone without requiring a GPU or internet connection. Over the last few months, I increased the corpus ten fold to 40 billion tokens by including text from 5 other East African Languages Swahili, Lao, Amharic, Somali, Kinyaranda and English. East African Languages make up nearly 90% of the new training corpus. I trained a 47M parameter model on this corpus and then fine-tuned it to be able to generate responses stories between the different languages. I have attached a link below if you would like to try a demo of it. I also made the models available on Huggingface if you would like to experiment with them on your own. Demo: https://tinyafricanstories.streamlit.app/ https://tinyafricanstories.streamlit.app/ Fine-tuned “Story-telling” model: mwebazarick/BULaMU-3-42M-Story · Hugging Face https://huggingface.co/mwebazarick/BULaMU-3-42M-Story