# 🧠 I Trained a Massive Word2Vec Model on 13 Billion Russian Fiction Words — Here’s What Happened

> Source: <https://dev.to/nevmenandr/i-trained-a-massive-word2vec-model-on-13-billion-russian-fiction-words-heres-what-happened-3m63>
> Published: 2026-09-07 06:14:00+00:00

**TL;DR:** I built a lemma‑based Word2Vec model (CBOW, 300d) on a huge corpus of Russian fiction (13B tokens → 7.3B after cleaning). You can load it with Gensim and explore semantic neighborhoods of words like *слово*, *язык*, *речь*. The model captures literary semantics without stop words or grammar tags. Check it out on [Hugging Face](https://huggingface.co/nevmenandr/w2v-russian-fiction).

Most pre‑trained Russian word2vec models are trained on web crawls, news, or mixed corpora. That’s fine for general NLP, but **fiction has its own semantic rules**. Poetic metaphors, archaic vocabulary, and author‑specific styles shift vector spaces.

I wanted a model that:

So I built one. And I’m sharing it under the MIT license.

| Metric | Value | 
|---|---|
| Raw words before preprocessing | **13.98 B** | 
| After lemmatization & stop‑word removal | **7.36 B** | 
| Sentences (after cleaning short ones) | **1.36 B** | 
| Paragraphs processed | ~539 M | 

**Stop words:** removed using [this list](https://github.com/nevmenandr/DigitalHumanitiesMinorFeatures/blob/master/stop_ru.txt).

**Lemmatizer:** Yandex Mystem.

**Sentence splitter:** `razdel.sentenize`.

All lemmas are **lowercase**, dictionary form, no part‑of‑speech tags (saves time & space).

``` python
import gensim

data = gensim.models.word2vec.LineSentence('splitted_lemmed_lines.txt')
model = gensim.models.Word2Vec(
    data,
    vector_size=300,
    window=10,
    min_count=2,
    sg=0          # CBOW (faster, good for large corpora)
)
model.save('cbow_300_10.model')
```

The training script is plain Gensim – no weird dependencies. You can retrain or fine‑tune if you have more data.

When you clone from Hugging Face, you get:

`cbow_300_10.model` (197 MB) – the main Gensim model object
`cbow_300_10.model.syn1neg.npy` (5.8 GB) – negative sampling weights
`cbow_300_10.model.wv.vectors.npy` (5.8 GB) – the actual word vectors
`README.md`, `tst.py` – docs and a test script
Yes, the two `.npy` files are large. But you can load the model without loading both if you only need similarities (Gensim does lazy loading). Or use `model.wv` directly.

``` python
import gensim

model = gensim.models.Word2Vec.load("cbow_300_10.model")

# Look at closest neighbours of "слово" (word)
for word, score in model.wv.most_similar("слово", topn=10):
    print(f"{word}: {score:.4f}")
```

Output:

```
фраза: 0.7941
словечко: 0.6602
слог: 0.6322
реплика: 0.6015
словосочетание: 0.5928
изречение: 0.5818
высказывание: 0.5800
глагол: 0.5735
эпитет: 0.5615
сентенция: 0.5556
```

Notice how *epithet* and *verb* pop up – the model clearly learned linguistic meta‑concepts from fiction.

Now try `язык` (language / tongue):

```
наречие (adverb) 0.6684
диалект 0.6095
латынь 0.5892
язычок (little tongue) 0.5698
алфавит 0.5039
грамматика 0.5027
суахили 0.4977
идиома 0.4952
иврит 0.4950
произношение 0.4927
```

And `речь` (speech):

```
монолог 0.6400
спич 0.5914
тирада 0.5900
фраза 0.5652
диалог 0.5477
проповедь 0.5334
слово 0.5217
разглагольствование 0.5193
филиппика 0.5184
декламация 0.5147
```

`стекло` could be noun “glass” or past tense of “to flow”). That’s a conscious trade‑off for performance.`слова`, `слову`, `словом`? You won’t find them; use the lemma `слово`.` KeyedVectors.load()` to save RAM.
| Model | Corpus | Size | POS | Availability | 
|---|---|---|---|---|
| This one | Fiction, 13B words | 300d | No | MIT, HF | 
| [w2v-russian-tolstoy](https://huggingface.co/nevmenandr/w2v-russian-tolstoy) | Only Tolstoy | 300d | No | MIT, HF | 
| [w2v-russian-19c-fiction-lemmas](https://huggingface.co/dhcloud/w2v-russian-19c-fiction-lemmas) | 19th century prose | 300d | No | HF | 
| RusVectores (web+news) | Mixed, ~20B | 300d | Yes (tags) | CC BY‑SA | 

If you need a **general‑purpose** model with grammatical info, go for RusVectores. If you work with **literary analysis**, this one is your friend.

`pip install gensim` and `model = gensim.models.Word2Vec.load(...)`
`.npy` files are optional.
**Hugging Face:** [nevmenandr/w2v-russian-fiction](https://huggingface.co/nevmenandr/w2v-russian-fiction)

**License:** MIT  

Clone with `git lfs` or download via `huggingface_hub`:

``` python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="nevmenandr/w2v-russian-fiction", local_dir="./w2v_model")
```

`model.most_similar(positive=['царь', 'женщина'], negative=['мужчина'])` and share the result. (Hint: it’s not “царица” – fiction is weird.)

``` python
import gensim

model = gensim.models.Word2Vec.load("cbow_300_10.model")

# Find words similar to "поэт" (poet)
print("Neighbors of поэт:")
for w, s in model.wv.most_similar("поэт", topn=5):
    print(f"  {w}: {s:.4f}")

# Analogy: Москва : Россия = Париж : ?
result = model.wv.most_similar(positive=["Франция", "Москва"], negative=["Россия"], topn=1)
print(f"\nМосква / Россия ≈ Париж / {result[0][0]} (score {result[0][1]:.4f})")
```

Run it. Play with it. Break it. Then tell me in the comments what you found.

Happy vector hunting! 🧙♂️

*P.S. The model is called `cbow_300_10.model` – old‑school name, but it works like a charm.*