A very small AI model that makes up town names in the style of real ones. It guesses the next letter, one at a time. It does not know which places exist.
What it is for: tinytowns makes up new town names in the style of real ones. It learned the letter patterns of 128,435 real place names: it was trained to guess the next letter of a place name as well as it can. It does not know which places exist, so most names it writes are new. To find real places, look them up in a list of places, or ask a large AI model such as Claude. How it compares with large models:
Watch it #
A 37-second video. It shows the model's guesses, a real name, an invented name, and what you could use it for. The video has no sound.
What it is #
tinytowns is a very small AI model. It works like the models behind chat apps, but it is much smaller. You can read every number in it.
The model reads a name one letter at a time. Then it gives a chance to each symbol that can come next. There are 28 symbols: the 26 letters, a space, and "end of name". The bars show the five symbols with the highest chances.
How to use it #
- Select a model. The larger models make better guesses.
- Type some letters, for example
san. - Look at the bars. To add a letter, tap its bar.
- To let the model write the rest of the name, tap Finish it for me .
- To end the name, tap end of name , type a full stop (.), or press Enter.
When the name is complete, the page tells you if it is real or invented.
Temperature #
Temperature changes how the model selects letters. At a low temperature ("safe"), the usual letters get almost all the chances, and the names look like real names. At a high temperature ("wild"), unusual letters get more chances, and the names are stranger. Move the slider and look at the bars.
Real or invented #
The model learned from a list of 142,706 real place names. When a name is complete, the page looks for it in this list. "Real" means that a place with this name exists. "Invented" means that the name is not in the list.
Most names are invented, and that is the model's job. Each model made 10,000 names as a test. For each model, more than 95% of the names were invented. A "real" name can be one that the model copied from the list it learned from.
Where it runs #
The model runs in your browser, on your device. The page downloads the model from Hugging Face one time. After that, your device calculates each guess. No server does this work.
Three sizes #
| Model | Numbers in the model | Download | Error score |
|---|---|---|---|
| nano | 4,572 | 20 KB | 2.47 |
| small | 105,052 | 424 KB | 2.20 |
| tinytowns | 1,001,500 | 4 MB | 2.15 |
The error score shows how bad the guesses are. A lower score is better. We measured it on 14,271 names that the model did not learn from. Random guesses get a score of 3.33.
A model with more numbers makes better guesses, but each step up helps less. From nano to small, the score goes down by 0.26. From small to tinytowns, it goes down by only 0.05.
What it can do #
These are ideas. We did not test them:
- Make names for towns in games and on maps.
- Make test data with place names that are not real.
- Tell if a word looks like a place name. For example, it could find
asdfghin a city field.
Technical details #
- Model: a decoder-only, character-level transformer, written by hand in plain PyTorch.
Vocabulary of 28 (index 0
.marks start and end of a name, then space,a–z), context of 21. Learned token and position embeddings, pre-LayerNorm blocks (causal multi-head self-attention, then a GELU MLP), a final LayerNorm and a separate unembedding. Biases on every linear layer, float32. - Sizes (d_model / layers × heads / MLP width): nano 16 / 1 × 1 / 64, small 64 / 2 × 4 / 256, tinytowns 128 / 5 × 4 / 512.
- Data: GeoNames
cities1000(places with 1,000+ people, 243 countries), lowercased, hyphens to spaces, apostrophes and names over 20 letters dropped: 142,706 names, split 90/10 with a fixed seed (128,435 train, 14,271 validation). - Training: next-symbol cross-entropy on
.+ name +., padding ignored. AdamW, weight decay 0.01, learning rate 3e-3 (nano, small) or 1e-3 (tinytowns), 6,000 steps, batch 64 on a CPU (nano) or 256 on a laptop RTX 3050 (small 135 s, tinytowns 550 s). The weights kept are from the step with the lowest validation loss. - Validation loss: uniform 3.3322, bigram 2.6732, nano 2.4676, small 2.2049, tinytowns 2.1504 (best at step 5,500; train loss 2.0085 there, and validation loss rises to 2.1543 by step 6,000).
- In the browser: a hand-written JavaScript forward pass, with no runtime library. It parses the
.safetensorsheader itself and recomputes the whole context for each guess (no KV cache). On all 14,271 validation names it matches the PyTorch loss to six decimal places for all three sizes. Measured in headless Chrome on a one-core cloud server: about 0.7 ms, 3 ms and 34 ms per guess.
More detail is in the model card on Hugging Face and the
step-by-step code on [GitHub](https://github.com/dave8172/tinytowns). The series
[How an LLM runs](https://quirkyagents.com/wizard/blog/series/how-an-llm-runs/) explains the same parts in a full-size model.
Place names: [GeoNames](https://www.geonames.org), licensed
[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
← [All work](https://quirkyagents.com/wizard/projects/#models)