How I Built a 150 GB Multilingual & Code Dataset for Central Asian AI (And Fought Out-of-Memory Errors for 10 Hours)

A developer built the Multilingual Code and Language Dataset, an open-source collection of nearly 150 GB of uncompressed text and source code covering Central Asian languages including Kyrgyz, Kazakh, Uzbek, and Tajik, now published on Hugging Face under a CC BY 4.0 license. Compressing the corpus with .7z took roughly 10 hours and repeatedly failed with out-of-memory errors before the data was reduced to about 27.7 GB. The project aims to address the scarcity of technical and code-related corpora for low-resource Central Asian languages.

Hi Dev.to While tech giants are competing to train LLMs on trillions of English tokens, there is a severe shortage of high-quality open-source datasets for Central Asian languages Kyrgyz, Kazakh, Uzbek, Tajik . Technical corpora for these regions are scarce, and code-related datasets are practically non-existent. As a developer, I decided to fix this myself. I spent weeks collecting, parsing, and cleaning data to build the Multilingual Code and Language Dataset — an open-source project that packs nearly 150 GB of uncompressed data , now available on Hugging Face. But gathering the data was only half the battle. The real nightmare started when I tried to compress and upload it. When you have 150 gigabytes of raw text and source code, you can't just upload it directly. To save bandwidth for the community, compression was a must. I chose .7z with a maximum compression profile. And that's when my PC started begging for mercy. Compressing text at this scale requires massive amounts of RAM for dictionary sizes. The process took 10 agonizing hours. The compression tool kept crashing, my OS froze constantly, and the console kept throwing the dreaded Failed message due to Out-of-Memory OOM errors. I had to tweak parameters, reboot, catch the perfect timings, and pray my RAM sticks wouldn't melt. In the end, I won. 150 GB of text was successfully tamed and compressed down to ~27.7 GB , making it easily downloadable. Once you download and uncompress the archive, you will find a highly structured dataset stored in clean .txt and .jsonl files: The core feature of this dataset is bridging the gap between programming source code and low-resource languages: The project is fully open-source and distributed under the CC BY 4.0 license. Feel free to use it for research, benchmarking, or commercial applications. 👉 Get it here on Hugging Face: https://huggingface.co/datasets/fortik11/Multilingual dataset https://huggingface.co/datasets/fortik11/Multilingual dataset I would love to hear your feedback If you have any questions about data filtering, parsing pipelines, or want to collaborate on training models using this data, let's chat in the comments