the first thing my language model wrote that looked like a sentence was:
the little red fox ran home.o..ittle red fox ran home.o..ittle red fox r
i was happy about the first part. less happy about the rest.
there was no attention in this model. no transformer blocks. just a grid of cells, each holding a vector, repeatedly talking to its eight neighbours. i wanted to know if a next-character predictor could emerge from only that local conversation.
the fox suggested it could learn something about text. the loop suggested i needed a better question than “does the loss go down?”
making a language model out of a grid #
Conway’s Game of Life gives every cell the same rule: look at your neighbours, then update. my version replaces alive/dead with a learned vector and replaces the hand-written rule with a small neural network. so it’s closer to a neural cellular automaton than the actual Game of Life.
i put character embeddings along the top row of a 2D grid. every generation, the same 3×3 convolution runs at every cell. each cell changes its state a little. after several generations, i read one cell near the right edge and ask it for the next character.
input row: t h e l i t t
↓ ↓ ↓ ↓ ↓ ↓ ↓
[ cells updating with the same local rule ]
↓
readout cell → next character
the important constraint is that a cell cannot suddenly read the whole sentence. information from a character on the left has to pass through the cells in between. one generation gives it only one 3×3 hop. if the readout is sixteen columns away, there is no route in fewer than sixteen generations.
the implementation was intentionally small: PyTorch, 16 numbers per cell in the CPU experiments, a residual update, and one shared rule reused across the grid. the first text model used a 16×16 grid, eight characters of context, and eight generations. it trained on a tiny, repetitive fox corpus i wrote for the experiment. definitely not a serious language benchmark :)
one input window as it moves through the grid. the bright cells along the top are the characters i injected. the rest lighting up tells me the cells are changing, not yet that they remember the right character.
after 900 training steps its training cross-entropy was 0.310 and its held-out text cross-entropy was 0.359. it could generate recognisable bits of the training text. but it also fell into that repeating fox phrase.
so… was the grid actually passing character information around? or had the cells nearest the output just learned the easy local patterns?
i changed one character and watched where it went #
i froze the language-trained model. then i put one of two characters at a known location, surrounded it with random characters, and asked linear decoders at different cells to recover which character it was. the decoder saw held-out contexts it hadn’t been fitted on. chance was 50%.
near the source, the answer was visible. after more generations, weak information appeared a few cells farther away. but at the far readout, it wasn’t useful.
| distance from source | probe accuracy after 8 generations | after 32 generations |
|---|---|---|
| 1 | 75.5% | 74.0% |
| 4 | 58.9% | 59.5% |
| 8 | 57.6% | 56.6% |
| 16 | 50.0% | 50.0% |
these are averages over cells at the same Manhattan distance, not a claim that every cell at distance eight knows the character. and 32 generations here means running an eight-generation-trained rule longer than it was trained to run.
yellow means a linear decoder can recover the changed character on held-out inputs. the dark right-hand side is 50% chance. the 3×3 rule can also move diagonally, so a Manhattan distance of two can sometimes be reached in one generation.
there was a path. the useful signal just didn’t make it very far.
the repetition wasn’t the kind of collapse i expected #
i initially thought the fox loop meant the hidden grid was settling into one stable state. the model would generate the same output because, somehow, all the cells had stopped changing.
that wasn’t what happened.
as the phrase repeated, two successive grids stayed about 22 units apart in L2 distance. the output entropy didn’t suddenly fall to zero either. but compare grids at the same position in the 26-character phrase, and they were identical.
it was a cycle, not a fixed point. once the sliding context repeated, a deterministic model rebuilt the same grid, produced the same next character, and went round again. i had been using “the state collapsed” as shorthand for something the measurements didn’t support.
that distinction mattered later, because another experiment really did end in a single-character loop.
maybe the architecture just can’t carry things that far? #
this was the tempting conclusion. local rules sound nice until they have to move something sixteen cells away. so i took language out of the experiment entirely.
in a separate model with the same kind of local update, i placed a 0 or 1 in one source cell. i trained the cellular rule directly to make a target cell recover it. i tried targets 1, 2, 4, 8, and 16 columns away, and rollout lengths from 1 to 32 generations. the other cells had random background activity; paired test examples shared that activity and differed only in the source bit.
this time the grid learned to communicate.
a model trained for 32 generations recovered the bit 16 cells away with 100% accuracy on 128 balanced held-out examples. i got that result in two seeds. before enough generations had passed, the distant cell was at chance. by generation 32, the signal had made it. visualising the difference between the 0 grid and the 1 grid showed a widening band moving along the target row, with information still present nearer the source too.
same random background, opposite source bit. the bright region is where the two grids differ, not a map of prediction accuracy. by generation 32 that difference has spread toward the target on the right.
it wasn’t magic global access disguised as a convolution. when the objective said “get this bit over there,” the shared local rule found a way to do it.
there was a catch. some rules only knew how to carry a bit for a particular amount of time. one model got the far target right at generation 32 and lost its trained readout’s correct answer by generation 64. the bit could still affect the target cell; the code the head expected had changed. a rule trained across different rollout lengths was more stable than that short-horizon rule, though it wasn’t consistently better than a rule trained at 32 generations.
capacity to transport and a stable, readable representation are two separate problems. apparently i needed both.
telling the language model to remember #
now for the experiment i actually cared about. if direct supervision taught the cells to pass a bit, would a smaller memory objective make the language model do the same thing while predicting text?
i trained four new, matched character models. each had a 20-character context, a 20×20 grid, 32 generations, the same training examples, and the same normal next-character readout. the previous eight-character model couldn’t even contain a character sixteen positions back, so comparing its number directly would have been nonsense.
all four new models had three extra readout heads. each tried to identify a character from 4, 8, or 16 positions before the newest input. only the weight on their loss changed:
total loss = next-character loss + λ × old-character loss
λ = 0, 0.01, 0.1, or 0.5
the λ = 0 model was the matched baseline; its memory heads existed but didn’t train. half the training examples came from the little fox text. the other half were deliberately annoying copy examples: put a random r or b at an earlier position, mark which position matters, fill the rest with distractors, and require the normal next-character prediction to output that old bit. nearby character statistics alone cannot solve a balanced paired example.
after the same 250 steps plus 200 more steps for each model, here’s the part that made me stop staring at the language loss:
| memory weight | held-out fox-text cross-entropy | normal readout on copy tasks (4 / 8 / 16 back) |
|---|---|---|
| 0 | 0.358 | 50% / 50% / 50% |
| 0.01 | 0.322 | 50% / 50% / 50% |
| 0.1 | 0.365 | 50% / 50% / 50% |
| 0.5 | 0.339 | 50% / 50% / 50% |
the auxiliary heads got better at recovering old characters from the repetitive fox text. for example, the λ = 0.5 head reached about 29% accuracy on the character sixteen positions back, compared with an untrained head near 3%. at first that looked promising.
but on the unpredictable old bit, even the relevant auxiliary head stayed around chance. the normal next-character head was at chance at every tested offset, for every λ. when i froze the new language models and fitted fresh decoders along the grid, i could see the bit near where it entered. at the prediction cell sixteen columns away, all four decoders scored 50%. adding more inference generations didn’t rescue it: a fresh decoder at 40, 48, or 64 generations was still at chance. it wasn’t just that a good code had drifted and i needed a new decoder. i couldn’t measure a useful code there in the first place.
and to make sure the extra heads weren’t quietly doing the next-token work, i removed their weights from the checkpoint and loaded the rest into the plain language model. its next-token logits were exactly the same. the copy failure belonged to the normal readout.
the highest memory weight had one more surprise. its held-out fox-text loss looked slightly better than the baseline’s. greedy generation looked like this:
...the lun
h lettle blue rietbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb...
it generated b for 93 consecutive characters. unlike the original fox phrase, that final constant context rebuilt an identical grid each step. one validation number had missed a pretty visible failure mode.
top: changing only the old bit is easy to detect near where it entered, but probes are at chance at the readout on the far right. bottom: the normal next-character head’s entropy during greedy generation. the flat red line belongs to the model that keeps choosing b.
what i think happened #
the experiments don’t say cellular automata are bad language models. this was a tiny authored corpus, small grids, and a limited training budget. they also don’t say the memory loss can never work. i only tried three weights, one readout location, and this particular setup.
but they ruled out the explanation i started with: “the cells cannot communicate over sixteen positions.” they can. i watched them do it when the task was only to transport a bit.
in the text model, a repeating phrase is often predictable without recovering the exact character sixteen steps ago. an auxiliary head can get better on those phrases without teaching the grid to carry an independent bit all the way to the ordinary next-token readout. the direct transport task gives a clean, strong instruction. the mixed language objective doesn’t, at least not at these weights and this budget.
that’s the result i find interesting. a model can have the machinery to move information and still learn a shortcut when the task lets it. and if you only check whether it can finish the little red fox, you might never notice the difference.