# I Built a Tiny Japanese GPT from Scratch in PyTorch and Tried a Toy 'RSI' Loop: Here's What It Does and Doesn't Show

> Source: <https://dev.to/koheip/i-built-a-tiny-japanese-gpt-from-scratch-in-pytorch-and-tried-a-toy-rsi-loop-heres-what-it-does-415j>
> Published: 2026-11-20 12:34:26+00:00

**Recommendation:** When running self-training loops, treat self-assigned scores as format and confidence signals rather than evidence of improvement, and validate gains against held-out data.
More in the playbook "Fine-tuning and data": <https://wpnews.pro/playbooks/training-data>

I'm a second-year student at ZEN University studying intelligent informatics (online), and I build AI agents as a personal project. I have no work experience in machine learning.

I use LLMs through APIs and agents every day, but I wanted to see what happens inside one with my own hands. So I wrote a small Japanese GPT using only basic PyTorch operations: no pretrained weights, no high-level model libraries. Then I tried a toy version of "RSI" (recursive self-improvement): the model generates several answers, grades them itself, and retrains on the best ones.

Up front: this is a **tiny educational model trained on a few KB of text**. Don't expect it to be useful, and please don't read the RSI numbers as "the model got smarter." I'll explain why below. This is not AGI or anything close to it.

A character-level, decoder-only Transformer with about 0.9M parameters (the pretraining version has 898,425, which I also checked by hand from the layer sizes). It has three stages:

`mini_gpt.py`: character tokenizer + Transformer, pretrained on short excerpts of modern Japanese literature to continue text.`chat_gpt.py`: the same architecture, fine-tuned on `ユーザー: ...\nAI: ...◁` dialogues so it behaves like a chatbot.`rsi_gpt.py`: a best-of-N self-training loop.
There is also a local web UI (`web_ui.py`, standard-library `http.server` only) that shows the top-5 probabilities for the first generated character and lets you edit the dialogue data and retrain in the browser.

Hyperparameters in `mini_gpt.py`:

```
BLOCK_SIZE = 64       # max context length
N_EMBD = 128          # embedding dimension
N_HEAD = 4            # attention heads
N_LAYER = 4           # Transformer blocks
```

Causal self-attention masks out future characters with a lower-triangular mask before the softmax:

```
att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(head_dim))
att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float("-inf"))
att = F.softmax(att, dim=-1)
```

Each block is Pre-LayerNorm with two residual connections:

``` python
def forward(self, x):
    x = x + self.attn(self.ln1(x))  # residual + attention
    x = x + self.mlp(self.ln2(x))   # residual + feed-forward
    return x
```

The tokenizer just numbers the characters that appear in the training text:

```
chars = sorted(list(set(text)))
stoi = {ch: i for i, ch in enumerate(chars)}
```

I didn't change the architecture. Two things changed:

`"ユーザー: {question}\nAI: "`.`◁`, and generation stops when it is sampled.

```
if token_id == stop_token_id:
    break  # stop when the end marker appears
```

Without the stop character the model keeps writing the next `ユーザー: ...` turn by itself. I call that role leakage below.

This is a small-scale best-of-N rejection sampling loop, in the spirit of STaR / ReST:

The score is the sum of four parts (`evaluate_candidate`):

| Part | Points | Meaning | 
|---|---|---|
| Completion | 25 | ended naturally with `◁` | 
| Format | 25 | no "ユーザー"/"AI:" leakage, no newline, no 4+ repeated characters, at least 12 characters | 
| Relevance | up to 30 | contains keywords I hand-picked per prompt (15 points each) | 
| Self-confidence | up to 20 | the model's own average probability of the characters it sampled, times 20 | 

```
matched_kw = [kw for kw in prompt_info["keywords"] if kw in reply]
kw_score = min(30.0, len(matched_kw) * 15.0)
...
conf_score = round(candidate["avg_confidence"] * 20.0, 1)
```

The exploration temperature is lowered each generation: `max(0.45, 0.85 - (gen - 1) * 0.12)`, i.e. 0.85, 0.73, 0.61.

Sources: `PAPER.md` and `rsi_history.json`. I recomputed the RSI table from `rsi_history.json`.

**Pretraining** (1,678 characters of Japanese literature excerpts, 377 distinct characters, 800 steps): loss went from 6.1098 at step 1 to 0.0683 at step 800. Prompted with "吾輩は", it produces the opening of *I Am a Cat*.

**Dialogue fine-tuning** (27 Q&A pairs, 1,575 characters, 1,000 steps): loss went from 5.5833 at step 1 to 0.0387 at step 1000.

**RSI** (7 prompts, N=6, threshold 75, 350 steps per generation):

| Generation | Temperature | Avg best score | Avg worst score | Prompts with worst < 75 | Loss after retraining | 
|---|---|---|---|---|---|
| Gen 1 | 0.85 | 97.56 | 80.77 | 4/7 | 0.0635 | 
| Gen 2 | 0.73 | 97.84 | 97.80 | 0/7 | 0.0490 | 
| Gen 3 | 0.61 | 97.84 | 97.80 | 0/7 | 0.0639 | 

Gen 1 failures looked like this:

The average worst-case score went from 80.77 to 97.80, and prompts with a failing candidate went from 4/7 to 0/7. But please don't read this as the model getting smarter. Here is why.

`chat_data.txt`). Gen 2 and Gen 3 are being asked questions whose answers they saw in training. Higher scores are expected and say nothing about unseen questions.
So what actually happened is: a tiny model picked its best-scoring answers to 7 questions and re-learned them.

```
pip install -r requirements.txt   # torch>=2.0.0

python mini_gpt.py                # pretrain and generate
python chat_gpt.py                # train on dialogues (saves chat_model.pt)
python chat_gpt.py --chat         # chat with the saved model
python web_ui.py                  # local web UI (http://localhost:8765)
python rsi_gpt.py                 # run 3 RSI generations
```

Note that `rsi_gpt.py` rewrites `chat_data.txt`, so back it up first. If you only want to poke at it, the demo Space below is easier.
