Hi Dev.to! While tech giants are competing to train LLMs on trillions of English tokens, there is a severe shortage of high-quality open-source datasets for Central Asian languages (Kyrgyz, Kazakh, Uzbek, Tajik). Technical corpora for these regions are scarce, and code-related datasets are practically non-existent.
As a developer, I decided to fix this myself. I spent weeks collecting, parsing, and cleaning data to build the Multilingual Code and Language Dataset — an open-source project that packs nearly 150 GB of uncompressed data, now available on Hugging Face.
But gathering the data was only half the battle. The real nightmare started when I tried to compress and upload it.
When you have 150 gigabytes of raw text and source code, you can't just upload it directly. To save bandwidth for the community, compression was a must. I chose .7z with a maximum compression profile.
And that's when my PC started begging for mercy.
Compressing text at this scale requires massive amounts of RAM for dictionary sizes. The process took 10 agonizing hours. The compression tool kept crashing, my OS froze constantly, and the console kept throwing the dreaded Failed message due to Out-of-Memory (OOM) errors. I had to tweak parameters, reboot, catch the perfect timings, and pray my RAM sticks wouldn't melt.
In the end, I won. 150 GB of text was successfully tamed and compressed down to ~27.7 GB, making it easily downloadable.
Once you download and uncompress the archive, you will find a highly structured dataset stored in clean .txt and .jsonl files:
The core feature of this dataset is bridging the gap between programming source code and low-resource languages:
The project is fully open-source and distributed under the CC BY 4.0 license. Feel free to use it for research, benchmarking, or commercial applications.
👉 Get it here on Hugging Face: https://huggingface.co/datasets/fortik11/Multilingual_dataset
I would love to hear your feedback! If you have any questions about data filtering, parsing pipelines, or want to collaborate on training models using this data, let's chat in the comments!