{"slug": "eliza-with-sparse-autoencoder", "title": "ELIZA with Sparse Autoencoder", "summary": "A developer has built an upgraded version of the classic ELIZA chatbot using a sparse autoencoder with a large latent memory layer, achieving logarithmic memory scaling and resistance to catastrophic forgetting. The model stores entries in a compressed latent representation, activating only the top-k most similar entries to generate responses, and was tested on 10,000 Minecraft QA pairs from the 'naklecha/minecraft-question-answer-700k' dataset. The approach aims to enable storage of millions of entries while maintaining speed and compact concept representation.", "body_md": "Back when I was a teenager and first wanted to build an AI, I tried recreating the ELIZA chatbot and a word n-gram language model. Since then, I’ve often thought about how to upgrade ELIZA using modern technology while still adhering to its fundamental principles. The latest thing I’ve built is an autoencoder with a large memory layer in the middle, where the entries that would normally reside in ELIZA’s database are stored in a latent representation.\n\nThe advantages of this model are its speed, logarithmic memory scaling (log(n)), and strong resistance to catastrophic forgetting. This potentially allows for the storage of millions of entries; even though the model has a limited grasp of meaning, the sheer size of the memory makes it more likely to find a similar entry for any given case.\n\n**Architecture:**\n\nSimilar to RAG, the encoder converts the text into an embedding.\n\nEntries that would otherwise reside in an ELIZA-style database are stored latently in the memory layer, and the entries most similar to the embedding are activated.\n\nUnlike ELIZA, which can only output the exact stored sequence of words, the decoder can slightly modify the response. However, because it does not grasp meaning particularly well, a similar entry must exist for every question.\n\nOne advantage of the model is that the neurons can effectively be ‚monosemantic‘; the entries represent concrete concepts that can be clearly stored within a few neurons, rather than abstract meanings that would be distributed like a cloud across many neurons. Only the top-k connections are activated, which incentivizes the model to generate compact concepts and also improves processing speed. A ‚cloud-like‘ representation might be more prone to catastrophic forgetting because neurons would tend to overlap. Contrastive learning is used to train the model so that similar concepts merge while dissimilar ones repel each other. This aims to further mitigate catastrophic forgetting, thereby allowing the memory to scale at a rate of log(n).\n\nAs an analogy, consider that many words overlap across documents; if these were stored in a binary search tree, they could—in the best-case scenario—be compressed to log(n) complexity, and this is the type of representation the model is intended to learn.\n\nAs a first test I trained the model on 10000 qa-pairs of the ‚naklecha/minecraft-question-answer-700k‘ dataset and was able to memorize all entries latently. The training took 3 hours and one problem is that I can’t scale much higher, so it’s not possible to test yet whether the compression with log(n) or the reduced catastrophic forgetting actually works.\n\n**Example:**\n\nQuestion:\n\nWhat is a characteristic of the Birch Forest Hills biome in Minecraft that differentiates it from regular Birch Forests?\n\nPreset answer:\n\nIn Birch Forest Hills, the terrain is hillier compared to regular Birch Forests, with steeper hills and rougher terrain.\n\nt5-small response without memory layer:\n\nA forest that is surrounded by trees.\n\nResponse with trained memory:\n\nIn Birch Forest Hills, the terrain is hillier and more erratic, making it less suitable for shelter.", "url": "https://wpnews.pro/news/eliza-with-sparse-autoencoder", "canonical_source": "https://discuss.huggingface.co/t/eliza-with-sparse-autoencoder/178156#post_1", "published_at": "2026-07-23 10:52:48+00:00", "updated_at": "2026-07-23 11:03:35.548137+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "neural-networks", "ai-research"], "entities": ["ELIZA", "t5-small", "naklecha/minecraft-question-answer-700k"], "alternates": {"html": "https://wpnews.pro/news/eliza-with-sparse-autoencoder", "markdown": "https://wpnews.pro/news/eliza-with-sparse-autoencoder.md", "text": "https://wpnews.pro/news/eliza-with-sparse-autoencoder.txt", "jsonld": "https://wpnews.pro/news/eliza-with-sparse-autoencoder.jsonld"}}