🧠 I Trained a Massive Word2Vec Model on 13 Billion Russian Fiction Words — Here’s What Happened

A developer has released a large Word2Vec model trained on 13 billion words of Russian fiction, available on Hugging Face under the MIT license. The lemma-based CBOW model with 300-dimensional vectors captures literary semantics, as demonstrated by nearest-neighbor examples for words like 'слово' and 'язык'. The model is intended for literary analysis and can be loaded with Gensim.

TL;DR: I built a lemma‑based Word2Vec model CBOW, 300d on a huge corpus of Russian fiction 13B tokens → 7.3B after cleaning . You can load it with Gensim and explore semantic neighborhoods of words like слово , язык , речь . The model captures literary semantics without stop words or grammar tags. Check it out on Hugging Face https://huggingface.co/nevmenandr/w2v-russian-fiction . Most pre‑trained Russian word2vec models are trained on web crawls, news, or mixed corpora. That’s fine for general NLP, but fiction has its own semantic rules . Poetic metaphors, archaic vocabulary, and author‑specific styles shift vector spaces. I wanted a model that: So I built one. And I’m sharing it under the MIT license. | Metric | Value | |---|---| | Raw words before preprocessing | 13.98 B | | After lemmatization & stop‑word removal | 7.36 B | | Sentences after cleaning short ones | 1.36 B | | Paragraphs processed | ~539 M | Stop words: removed using this list https://github.com/nevmenandr/DigitalHumanitiesMinorFeatures/blob/master/stop ru.txt . Lemmatizer: Yandex Mystem. Sentence splitter: razdel.sentenize . All lemmas are lowercase , dictionary form, no part‑of‑speech tags saves time & space . python import gensim data = gensim.models.word2vec.LineSentence 'splitted lemmed lines.txt' model = gensim.models.Word2Vec data, vector size=300, window=10, min count=2, sg=0 CBOW faster, good for large corpora model.save 'cbow 300 10.model' The training script is plain Gensim – no weird dependencies. You can retrain or fine‑tune if you have more data. When you clone from Hugging Face, you get: cbow 300 10.model 197 MB – the main Gensim model object cbow 300 10.model.syn1neg.npy 5.8 GB – negative sampling weights cbow 300 10.model.wv.vectors.npy 5.8 GB – the actual word vectors README.md , tst.py – docs and a test script Yes, the two .npy files are large. But you can load the model without loading both if you only need similarities Gensim does lazy loading . Or use model.wv directly. python import gensim model = gensim.models.Word2Vec.load "cbow 300 10.model" Look at closest neighbours of "слово" word for word, score in model.wv.most similar "слово", topn=10 : print f"{word}: {score:.4f}" Output: фраза: 0.7941 словечко: 0.6602 слог: 0.6322 реплика: 0.6015 словосочетание: 0.5928 изречение: 0.5818 высказывание: 0.5800 глагол: 0.5735 эпитет: 0.5615 сентенция: 0.5556 Notice how epithet and verb pop up – the model clearly learned linguistic meta‑concepts from fiction. Now try язык language / tongue : наречие adverb 0.6684 диалект 0.6095 латынь 0.5892 язычок little tongue 0.5698 алфавит 0.5039 грамматика 0.5027 суахили 0.4977 идиома 0.4952 иврит 0.4950 произношение 0.4927 And речь speech : монолог 0.6400 спич 0.5914 тирада 0.5900 фраза 0.5652 диалог 0.5477 проповедь 0.5334 слово 0.5217 разглагольствование 0.5193 филиппика 0.5184 декламация 0.5147 стекло could be noun “glass” or past tense of “to flow” . That’s a conscious trade‑off for performance. слова , слову , словом ? You won’t find them; use the lemma слово . KeyedVectors.load to save RAM. | Model | Corpus | Size | POS | Availability | |---|---|---|---|---| | This one | Fiction, 13B words | 300d | No | MIT, HF | | w2v-russian-tolstoy https://huggingface.co/nevmenandr/w2v-russian-tolstoy | Only Tolstoy | 300d | No | MIT, HF | | w2v-russian-19c-fiction-lemmas https://huggingface.co/dhcloud/w2v-russian-19c-fiction-lemmas | 19th century prose | 300d | No | HF | | RusVectores web+news | Mixed, ~20B | 300d | Yes tags | CC BY‑SA | If you need a general‑purpose model with grammatical info, go for RusVectores. If you work with literary analysis , this one is your friend. pip install gensim and model = gensim.models.Word2Vec.load ... .npy files are optional. Hugging Face: nevmenandr/w2v-russian-fiction https://huggingface.co/nevmenandr/w2v-russian-fiction License: MIT Clone with git lfs or download via huggingface hub : python from huggingface hub import snapshot download snapshot download repo id="nevmenandr/w2v-russian-fiction", local dir="./w2v model" model.most similar positive= 'царь', 'женщина' , negative= 'мужчина' and share the result. Hint: it’s not “царица” – fiction is weird. python import gensim model = gensim.models.Word2Vec.load "cbow 300 10.model" Find words similar to "поэт" poet print "Neighbors of поэт:" for w, s in model.wv.most similar "поэт", topn=5 : print f" {w}: {s:.4f}" Analogy: Москва : Россия = Париж : ? result = model.wv.most similar positive= "Франция", "Москва" , negative= "Россия" , topn=1 print f"\nМосква / Россия ≈ Париж / {result 0 0 } score {result 0 1 :.4f} " Run it. Play with it. Break it. Then tell me in the comments what you found. Happy vector hunting 🧙♂️ P.S. The model is called cbow 300 10.model – old‑school name, but it works like a charm.