Hi Everybody,
I am just writing to share some progress I have made on my project. I am the creator of BULaMU, a family of language models (20M, 47M, 110M) trained completely from scratch for a low resource language: Luganda. The models are small and compute-efficient enough to run offline on a phone without requiring a GPU or internet connection.
Over the last few months, I increased the corpus ten fold (to 40 billion tokens) by including text from 5 other East African Languages (Swahili, Lao, Amharic, Somali, Kinyaranda) and English. East African Languages make up nearly 90% of the new training corpus.
I trained a 47M parameter model on this corpus and then fine-tuned it to be able to generate responses (stories) between the different languages. I have attached a link below if you would like to try a demo of it. I also made the models available on Huggingface if you would like to experiment with them on your own.
Demo: [https://tinyafricanstories.streamlit.app/](https://tinyafricanstories.streamlit.app/)
Fine-tuned (“Story-telling”) model: [mwebazarick/BULaMU-3-42M-Story · Hugging Face](https://huggingface.co/mwebazarick/BULaMU-3-42M-Story)