I built a tiny GPT from scratch to show my kids how chatbots really work

A developer with over 30 years of coding experience built PyLLM, a from-scratch GPT implementation in pure Python and numpy that includes its own autograd engine, tokenizer, attention layers, optimizer, training loop and sampling code, with no PyTorch or TensorFlow. The package ships with a pre-trained two-block, 59,739-parameter model trained on 149 Pokémon names, and offers a three-rung ladder (bigram, MLP, GPT) each trained for 2,000 steps, with the bigram model reaching a final loss of 2.3501 in about a second. The project is aimed at showing how chatbots work, with numerical gradient checks validating every operation and layer.

What actually happens inside a GPT? Everyone talks about "attention" and "transformers" and "training", but if you had to build one with nothing but arrays of numbers, where would you even start? I've been coding for over 30 years and I couldn't have told you. So I built a small one in Python and called it PyLLM. There's no PyTorch and no TensorFlow, just numpy for the array maths. The autograd engine, the layers, attention, the tokenizer, the optimiser, the training loop and the sampling are all plain Python you can read. pip install fromscratch-pyllm PyLLM ships with a tiny model that's already trained, so it starts writing straight away: bash $ pyllm --seed 0 --max-new-tokens 40 machop machoke machamp bellsprout weepin It wrote that one letter at a time: it guessed m , then a , then c , and fed each guess back in as the start of the next one. weepin gets cut off because I only gave it 40 characters. A GPT does exactly this, just with far more practice. A model can't read letters. First the text gets chopped into tokens, a bit like snapping a sentence into Lego bricks, and every brick gets a number. The model only ever sees the numbers: php $ pyllm tokenize --text "Pika" 'P' - I have not seen that character yet 'i' - 9 'k' - 11 'a' - 1 Here every character is its own brick. The alphabet was learned from lower-case Pokémon names, so a capital P simply isn't in it. There's also a BPE tokenizer that learns bigger bricks like "ing" , which is closer to what the big models use. Learning means asking every number in the model: if I nudge you up a little, does the answer get better or worse, and by how much? That's a gradient, and working it out automatically is called autograd. The docs explain it with breadcrumbs. Every time PyLLM's Tensor does some maths, it drops a breadcrumb saying who its parents were and how to pass the blame back to them. Then backward follows the trail in reverse: python from pyllm.autograd import Tensor a = Tensor 2.0 b = Tensor 3.0 c = a b + a c = 2 3 + 2 = 8 c.backward print c.data, a.grad, b.grad 8. 4. 2. a was used twice, once multiplied by 3 and once on its own, so its blame adds up to 4. Each op is a few lines: work out the answer, then leave a small backward recipe behind. How do you know the breadcrumbs are right? You check them the slow, obvious way: nudge each number a tiny bit and see how far the output moves. PyLLM's numerical grad does exactly that, and the tests use it on every op and layer. Everything else in PyLLM sits on top of this one idea. When you read "The trophy didn't fit in the case because it was too big", you glance back to work out what "it" means. Attention is the model doing the same thing. Each token asks a question a query , every earlier token holds up a label a key , and the good matches hand over their information a value . A causal mask stops anyone peeking at letters that haven't been written yet. A few attention "heads" side by side, plus a bit of private thinking the feed-forward part , make one Transformer block. Stack a few blocks and you've got a GPT. The bundled Pokémon model has two blocks and 59,739 numbers in total. A real chatbot has billions. Training is a guessing game. Guess the next letter, find out how surprised you should have been the loss , follow the breadcrumbs to share out the blame, nudge every number a little, and repeat. PyLLM gives you a ladder of three models, each one fixing a weakness of the one below. I trained each for 2,000 steps on the bundled list of 149 Pokémon names: bash $ pyllm train --model bigram --steps 2000 --seed 0 --out bigram.npz $ pyllm train --model mlp --steps 2000 --seed 0 --out mlp.npz $ pyllm train --model gpt --steps 2000 --seed 0 --out gpt.npz | Model | What it looks at | Final loss | Time on my run | |---|---|---|---| | Bigram | just the last letter | 2.3501 | about 1 second | | MLP | the last 3 letters | 0.7192 | about 2 seconds | | GPT | the last 16 letters, with attention | 0.2996 | about 4 minutes | Then I asked each one for names with the same seed pyllm --checkpoint <model .npz --seed 3 --max-new-tokens 60 : | Bigram | MLP | GPT | |---|---|---| | chon | dodrio | choke | | le | meapodeon | machamp | | r | flearowth | bellsprout | | kangowarie | geom | weepinbell | | wakingo | vilectrete | victreebel | | picltararlasioraro | ponyte | tentacool | | sefaratarar | mandshrew | ten | This was the most interesting thing I learned. The bigram only sees one letter, so it babbles. The MLP makes up names that sound right, like "flearowth" and "mandshrew". The GPT has the lowest loss and mostly just recites real Pokémon in order. It starts with "choke" rather than "machoke" because it begins with nothing to read. With only 149 names to learn from, the strongest model basically memorised the list, and the bundled checkpoint does too. Over 20 seeds at the default setting, only 7 of the 223 complete names it wrote weren't real Pokémon. You can push it to be bolder with a dial called temperature. Think of the model's scores as a weighted dice where the letters it likes get bigger sides. At 0 it always picks the favourite. Turn it up and the dice flattens out, so long shots get a chance: bash $ pyllm --seed 0 --temperature 1.5 --max-new-tokens 60 ndbell victafble vulpix nizweepinbell victreebel tentacool t Counted the same way at 1.5, 66 of 217 names were new. "Nizweepinbell" is my favourite. The main demo is harder. Pebble is a small programming language from the same series, and there's no pile of Pebble code anywhere in the world to learn from. So PyLLM grows its own. The docs put it as planting your own garden instead of foraging for wild berries. A generator builds random programs by rolling dice at every choice "loop or print? add or multiply?" , but it only makes moves the grammar allows, so every program it writes is valid. Add the real snippets from Pebble's docs and you get the bundled corpus: 1,001 programs, about 354,000 characters. Because the series has Pebble's real parser too, we can mark the model's homework honestly. Does what it wrote actually parse? js $ pyllm pebble --seed 7 --temperature 0.5 --score let total = n 5 + n let a = 6 } let a = result 6 - result } } ... parse rate: 0.0% of 1 programs Output trimmed. It looks like Pebble, but the braces are all over the place. Over 20 seeds, only 4% of 27 programs parsed. So PyLLM brings in a referee. A football referee doesn't play and doesn't tell anyone where to kick. They know the rules, and when someone tries an illegal move they blow the whistle, the move doesn't count, and the player has to choose again. PyLLM's referee is a small hand-written parser that answers two questions about half-written code: python from pyllm.pebble import referee for text in "let total = 5 +", "let total = 5 +\n", "let total = 5 + 2", "let total = = 5" : print f"{text r:22} viable={referee.viable text s:5} complete={referee.complete text }" js 'let total = 5 +' viable=True complete=False 'let total = 5 +\n' viable=False complete=False 'let total = 5 + 2' viable=True complete=True 'let total = = 5' viable=False complete=False While the model writes, every character it picks gets checked. If the code could no longer turn out valid, the whistle goes: that character is crossed off the dice and the model rolls again. Same tiny model, but now: js $ pyllm pebble --seed 3 --constrained --score --max-new-tokens 300 let a = 10 let x = calc == 7 + 15 - n + 97 print step 1 - 89 - z, y - 8 parse rate: 100.0% of 2 programs Over the same 20 seeds, 100% of 37 programs parsed. There's a catch, though: parsing and working are different things. Save that program to a file and run it with Pebble itself the pebble command comes with the pebble extra : js $ pebble s3.pbl 3 | let x = calc == 7 + 15 - n + 97 ^ Undeclared function 'calc' The referee checks the grammar, not the meaning. To be fair to the model, the generated training programs call made-up functions like compute and combine too, so it learned to write exactly what it was shown. PyLLM is built to be read, so it keeps the simple version of a lot of things. The obvious one is scale: about 60,000 numbers here, billions in a real model, trained on a large chunk of the internet on racks of GPUs. It's the same loop at a very different size. The bundled models also read one character at a time, where real models use learned sub-word tokens the BPE tokenizer shows the idea , and the Pokémon model can only look back 16 characters, where real ones look back hundreds of thousands of tokens. There's no GPU and no caching of earlier attention work either, so every step is plain numpy. And real assistants get extra training to follow requests and be helpful, while PyLLM only ever learns to continue text. The referee is the one part that matches production systems. They use the same "veto illegal tokens" trick to force valid JSON or code, and PyLLM's version is small enough to read through. pip install fromscratch-pyllm add 'fromscratch-pyllm pebble ' for parse scores and the pebble command , then run pyllm PyLLM is part of From scratch in Python, a series of 18 projects that rebuild real systems: a database, an operating system, git, a search engine, a programming language and more. Get the whole series with pip install fromscratch-pystack , or start at https://robcox2910.github.io/ https://robcox2910.github.io/ How this was made: I designed PyLLM and directed every part of it, but most of the code was written with Claude Code, under strict test-driven development, and this article was drafted with Claude's help too. Every output shown above comes from actually running it.