Back when I was a teenager and first wanted to build an AI, I tried recreating the ELIZA chatbot and a word n-gram language model. Since then, I’ve often thought about how to upgrade ELIZA using modern technology while still adhering to its fundamental principles. The latest thing I’ve built is an autoencoder with a large memory layer in the middle, where the entries that would normally reside in ELIZA’s database are stored in a latent representation.
The advantages of this model are its speed, logarithmic memory scaling (log(n)), and strong resistance to catastrophic forgetting. This potentially allows for the storage of millions of entries; even though the model has a limited grasp of meaning, the sheer size of the memory makes it more likely to find a similar entry for any given case.
Architecture:
Similar to RAG, the encoder converts the text into an embedding.
Entries that would otherwise reside in an ELIZA-style database are stored latently in the memory layer, and the entries most similar to the embedding are activated.
Unlike ELIZA, which can only output the exact stored sequence of words, the decoder can slightly modify the response. However, because it does not grasp meaning particularly well, a similar entry must exist for every question.
One advantage of the model is that the neurons can effectively be ‚monosemantic‘; the entries represent concrete concepts that can be clearly stored within a few neurons, rather than abstract meanings that would be distributed like a cloud across many neurons. Only the top-k connections are activated, which incentivizes the model to generate compact concepts and also improves processing speed. A ‚cloud-like‘ representation might be more prone to catastrophic forgetting because neurons would tend to overlap. Contrastive learning is used to train the model so that similar concepts merge while dissimilar ones repel each other. This aims to further mitigate catastrophic forgetting, thereby allowing the memory to scale at a rate of log(n).
As an analogy, consider that many words overlap across documents; if these were stored in a binary search tree, they could—in the best-case scenario—be compressed to log(n) complexity, and this is the type of representation the model is intended to learn.
As a first test I trained the model on 10000 qa-pairs of the ‚naklecha/minecraft-question-answer-700k‘ dataset and was able to memorize all entries latently. The training took 3 hours and one problem is that I can’t scale much higher, so it’s not possible to test yet whether the compression with log(n) or the reduced catastrophic forgetting actually works.
Example:
Question:
What is a characteristic of the Birch Forest Hills biome in Minecraft that differentiates it from regular Birch Forests?
Preset answer:
In Birch Forest Hills, the terrain is hillier compared to regular Birch Forests, with steeper hills and rougher terrain.
t5-small response without memory layer: A forest that is surrounded by trees.
Response with trained memory:
In Birch Forest Hills, the terrain is hillier and more erratic, making it less suitable for shelter.