I'm a second-year student at ZEN University studying intelligent informatics (online), and I build AI agents as a personal project. I have no work experience in machine learning.
I use LLMs through APIs and agents every day, but I wanted to see what happens inside one with my own hands. So I wrote a small Japanese GPT using only basic PyTorch operations: no pretrained weights, no high-level model libraries. Then I tried a toy version of "RSI" (recursive self-improvement): the model generates several answers, grades them itself, and retrains on the best ones.
Up front: this is a tiny educational model trained on a few KB of text. Don't expect it to be useful, and please don't read the RSI numbers as "the model got smarter." I'll explain why below. This is not AGI or anything close to it.
A character-level, decoder-only Transformer with about 0.9M parameters (the pretraining version has 898,425, which I also checked by hand from the layer sizes). It has three stages:
mini_gpt.py: character tokenizer + Transformer, pretrained on short excerpts of modern Japanese literature to continue text.chat_gpt.py: the same architecture, fine-tuned on ユーザー: ...\nAI: ...◁ dialogues so it behaves like a chatbot.rsi_gpt.py: a best-of-N self-training loop.
There is also a local web UI (web_ui.py, standard-library http.server only) that shows the top-5 probabilities for the first generated character and lets you edit the dialogue data and retrain in the browser.
Hyperparameters in mini_gpt.py:
BLOCK_SIZE = 64 # max context length
N_EMBD = 128 # embedding dimension
N_HEAD = 4 # attention heads
N_LAYER = 4 # Transformer blocks
Causal self-attention masks out future characters with a lower-triangular mask before the softmax:
att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(head_dim))
att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float("-inf"))
att = F.softmax(att, dim=-1)
Each block is Pre-LayerNorm with two residual connections:
def forward(self, x):
x = x + self.attn(self.ln1(x)) # residual + attention
x = x + self.mlp(self.ln2(x)) # residual + feed-forward
return x
The tokenizer just numbers the characters that appear in the training text:
chars = sorted(list(set(text)))
stoi = {ch: i for i, ch in enumerate(chars)}
I didn't change the architecture. Two things changed:
"ユーザー: {question}\nAI: ".◁, and generation stops when it is sampled.
if token_id == stop_token_id:
break # stop when the end marker appears
Without the stop character the model keeps writing the next ユーザー: ... turn by itself. I call that role leakage below.
This is a small-scale best-of-N rejection sampling loop, in the spirit of STaR / ReST:
The score is the sum of four parts (evaluate_candidate):
| Part | Points | Meaning |
|---|---|---|
| Completion | 25 | ended naturally with ◁ |
| Format | 25 | no "ユーザー"/"AI:" leakage, no newline, no 4+ repeated characters, at least 12 characters |
| Relevance | up to 30 | contains keywords I hand-picked per prompt (15 points each) |
| Self-confidence | up to 20 | the model's own average probability of the characters it sampled, times 20 |
matched_kw = [kw for kw in prompt_info["keywords"] if kw in reply]
kw_score = min(30.0, len(matched_kw) * 15.0)
...
conf_score = round(candidate["avg_confidence"] * 20.0, 1)
The exploration temperature is lowered each generation: max(0.45, 0.85 - (gen - 1) * 0.12), i.e. 0.85, 0.73, 0.61.
Sources: PAPER.md and rsi_history.json. I recomputed the RSI table from rsi_history.json.
Pretraining (1,678 characters of Japanese literature excerpts, 377 distinct characters, 800 steps): loss went from 6.1098 at step 1 to 0.0683 at step 800. Prompted with "吾輩は", it produces the opening of I Am a Cat.
Dialogue fine-tuning (27 Q&A pairs, 1,575 characters, 1,000 steps): loss went from 5.5833 at step 1 to 0.0387 at step 1000.
RSI (7 prompts, N=6, threshold 75, 350 steps per generation):
| Generation | Temperature | Avg best score | Avg worst score | Prompts with worst < 75 | Loss after retraining |
|---|---|---|---|---|---|
| Gen 1 | 0.85 | 97.56 | 80.77 | 4/7 | 0.0635 |
| Gen 2 | 0.73 | 97.84 | 97.80 | 0/7 | 0.0490 |
| Gen 3 | 0.61 | 97.84 | 97.80 | 0/7 | 0.0639 |
Gen 1 failures looked like this:
The average worst-case score went from 80.77 to 97.80, and prompts with a failing candidate went from 4/7 to 0/7. But please don't read this as the model getting smarter. Here is why.
chat_data.txt). Gen 2 and Gen 3 are being asked questions whose answers they saw in training. Higher scores are expected and say nothing about unseen questions.
So what actually happened is: a tiny model picked its best-scoring answers to 7 questions and re-learned them.
pip install -r requirements.txt # torch>=2.0.0
python mini_gpt.py # pretrain and generate
python chat_gpt.py # train on dialogues (saves chat_model.pt)
python chat_gpt.py --chat # chat with the saved model
python web_ui.py # local web UI (http://localhost:8765)
python rsi_gpt.py # run 3 RSI generations
Note that rsi_gpt.py rewrites chat_data.txt, so back it up first. If you only want to poke at it, the demo Space below is easier.