{"slug": "i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres", "title": "🧠 I Trained a Massive Word2Vec Model on 13 Billion Russian Fiction Words — Here’s What Happened", "summary": "A developer has released a large Word2Vec model trained on 13 billion words of Russian fiction, available on Hugging Face under the MIT license. The lemma-based CBOW model with 300-dimensional vectors captures literary semantics, as demonstrated by nearest-neighbor examples for words like 'слово' and 'язык'. The model is intended for literary analysis and can be loaded with Gensim.", "body_md": "**TL;DR:** I built a lemma‑based Word2Vec model (CBOW, 300d) on a huge corpus of Russian fiction (13B tokens → 7.3B after cleaning). You can load it with Gensim and explore semantic neighborhoods of words like *слово*, *язык*, *речь*. The model captures literary semantics without stop words or grammar tags. Check it out on [Hugging Face](https://huggingface.co/nevmenandr/w2v-russian-fiction).\n\nMost pre‑trained Russian word2vec models are trained on web crawls, news, or mixed corpora. That’s fine for general NLP, but **fiction has its own semantic rules**. Poetic metaphors, archaic vocabulary, and author‑specific styles shift vector spaces.\n\nI wanted a model that:\n\nSo I built one. And I’m sharing it under the MIT license.\n\n| Metric | Value | \n|---|---|\n| Raw words before preprocessing | **13.98 B** | \n| After lemmatization & stop‑word removal | **7.36 B** | \n| Sentences (after cleaning short ones) | **1.36 B** | \n| Paragraphs processed | ~539 M | \n\n**Stop words:** removed using [this list](https://github.com/nevmenandr/DigitalHumanitiesMinorFeatures/blob/master/stop_ru.txt).\n\n**Lemmatizer:** Yandex Mystem.\n\n**Sentence splitter:** `razdel.sentenize`.\n\nAll lemmas are **lowercase**, dictionary form, no part‑of‑speech tags (saves time & space).\n\n``` python\nimport gensim\n\ndata = gensim.models.word2vec.LineSentence('splitted_lemmed_lines.txt')\nmodel = gensim.models.Word2Vec(\n    data,\n    vector_size=300,\n    window=10,\n    min_count=2,\n    sg=0          # CBOW (faster, good for large corpora)\n)\nmodel.save('cbow_300_10.model')\n```\n\nThe training script is plain Gensim – no weird dependencies. You can retrain or fine‑tune if you have more data.\n\nWhen you clone from Hugging Face, you get:\n\n`cbow_300_10.model` (197 MB) – the main Gensim model object\n`cbow_300_10.model.syn1neg.npy` (5.8 GB) – negative sampling weights\n`cbow_300_10.model.wv.vectors.npy` (5.8 GB) – the actual word vectors\n`README.md`, `tst.py` – docs and a test script\nYes, the two `.npy` files are large. But you can load the model without loading both if you only need similarities (Gensim does lazy loading). Or use `model.wv` directly.\n\n``` python\nimport gensim\n\nmodel = gensim.models.Word2Vec.load(\"cbow_300_10.model\")\n\n# Look at closest neighbours of \"слово\" (word)\nfor word, score in model.wv.most_similar(\"слово\", topn=10):\n    print(f\"{word}: {score:.4f}\")\n```\n\nOutput:\n\n```\nфраза: 0.7941\nсловечко: 0.6602\nслог: 0.6322\nреплика: 0.6015\nсловосочетание: 0.5928\nизречение: 0.5818\nвысказывание: 0.5800\nглагол: 0.5735\nэпитет: 0.5615\nсентенция: 0.5556\n```\n\nNotice how *epithet* and *verb* pop up – the model clearly learned linguistic meta‑concepts from fiction.\n\nNow try `язык` (language / tongue):\n\n```\nнаречие (adverb) 0.6684\nдиалект 0.6095\nлатынь 0.5892\nязычок (little tongue) 0.5698\nалфавит 0.5039\nграмматика 0.5027\nсуахили 0.4977\nидиома 0.4952\nиврит 0.4950\nпроизношение 0.4927\n```\n\nAnd `речь` (speech):\n\n```\nмонолог 0.6400\nспич 0.5914\nтирада 0.5900\nфраза 0.5652\nдиалог 0.5477\nпроповедь 0.5334\nслово 0.5217\nразглагольствование 0.5193\nфилиппика 0.5184\nдекламация 0.5147\n```\n\n`стекло` could be noun “glass” or past tense of “to flow”). That’s a conscious trade‑off for performance.`слова`, `слову`, `словом`? You won’t find them; use the lemma `слово`.` KeyedVectors.load()` to save RAM.\n| Model | Corpus | Size | POS | Availability | \n|---|---|---|---|---|\n| This one | Fiction, 13B words | 300d | No | MIT, HF | \n| [w2v-russian-tolstoy](https://huggingface.co/nevmenandr/w2v-russian-tolstoy) | Only Tolstoy | 300d | No | MIT, HF | \n| [w2v-russian-19c-fiction-lemmas](https://huggingface.co/dhcloud/w2v-russian-19c-fiction-lemmas) | 19th century prose | 300d | No | HF | \n| RusVectores (web+news) | Mixed, ~20B | 300d | Yes (tags) | CC BY‑SA | \n\nIf you need a **general‑purpose** model with grammatical info, go for RusVectores. If you work with **literary analysis**, this one is your friend.\n\n`pip install gensim` and `model = gensim.models.Word2Vec.load(...)`\n`.npy` files are optional.\n**Hugging Face:** [nevmenandr/w2v-russian-fiction](https://huggingface.co/nevmenandr/w2v-russian-fiction)\n\n**License:** MIT  \n\nClone with `git lfs` or download via `huggingface_hub`:\n\n``` python\nfrom huggingface_hub import snapshot_download\nsnapshot_download(repo_id=\"nevmenandr/w2v-russian-fiction\", local_dir=\"./w2v_model\")\n```\n\n`model.most_similar(positive=['царь', 'женщина'], negative=['мужчина'])` and share the result. (Hint: it’s not “царица” – fiction is weird.)\n\n``` python\nimport gensim\n\nmodel = gensim.models.Word2Vec.load(\"cbow_300_10.model\")\n\n# Find words similar to \"поэт\" (poet)\nprint(\"Neighbors of поэт:\")\nfor w, s in model.wv.most_similar(\"поэт\", topn=5):\n    print(f\"  {w}: {s:.4f}\")\n\n# Analogy: Москва : Россия = Париж : ?\nresult = model.wv.most_similar(positive=[\"Франция\", \"Москва\"], negative=[\"Россия\"], topn=1)\nprint(f\"\\nМосква / Россия ≈ Париж / {result[0][0]} (score {result[0][1]:.4f})\")\n```\n\nRun it. Play with it. Break it. Then tell me in the comments what you found.\n\nHappy vector hunting! 🧙♂️\n\n*P.S. The model is called `cbow_300_10.model` – old‑school name, but it works like a charm.*", "url": "https://wpnews.pro/news/i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres", "canonical_source": "https://dev.to/nevmenandr/i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres-what-happened-3m63", "published_at": "2026-09-07 06:14:00+00:00", "updated_at": "2026-09-07 06:27:06.233376+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "large-language-models"], "entities": ["Hugging Face", "Gensim", "Yandex Mystem", "razdel"], "alternates": {"html": "https://wpnews.pro/news/i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres", "markdown": "https://wpnews.pro/news/i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres.md", "text": "https://wpnews.pro/news/i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres.txt", "jsonld": "https://wpnews.pro/news/i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres.jsonld"}}