{"slug": "show-hn-benchmarks-of-continual-learning-model-trained-for-a-month-on-a-laptop", "title": "Show HN: Benchmarks of continual learning model trained for a month on a laptop", "summary": "An open-source project called mini-AGI has trained a byte-level, continually learning language model for a month on a single 8 GB laptop GPU (an Nvidia RTX 3070), reaching 1.14 bits/byte on enwik9 after reading only 2.8 GB of data at an estimated cost of about $56. The model pages 397 experts (1.26B total parameters) from an NVMe drive while keeping 32 resident (~109M active) in 5.1 GB of VRAM, and has consumed roughly 35% of its ~8 GB training corpus. In online mode, where the model specializes as it moves through the evaluation set, mini-AGI scored 1.447 bits/byte on PG19 versus 1.995 frozen, though both trail MambaByte-972M's 0.883, which was trained on 75 GB of PG19 books.", "body_md": "A month ago, I shared mini-AGI, an experiment in training a byte-level, continually learning language model from scratch on a single 8 GB laptop GPU (an RTX 3070). To recap the setup briefly: it completely decouples model size from VRAM by paging experts to and from an NVMe drive (currently 397 experts / 1.26B total parameters on disk, with only 32 resident / ~109M active in 5.1 GB of VRAM). It trains on a single stream at batch 1, with the attention/embedding trunk learning rate dampened to 0.1x to mitigate catastrophic forgetting.\n\nThe training is still going and went only through ~35% of the training corpus selected at the beginning of the run (about 8GB in total). Here is the current status:\n\n1. enwik9 (First 10^9 bytes of English Wikipedia):\n\n```\n    Model               Params   Data Read       Cost     enwik9 (bits/byte) ↓\n\n    Llama-3.2-1B        1.23B    36 TB (9T tok)  ~$900k   0.68\n    Chinchilla 1B       1.00B    5.6 TB          ~$68k    0.90\n    GPT-2 XL            1.56B    40 GB           ~$43k    1.07\n    GPT-2 Large         0.77B    40 GB           ~$21k    1.12\n    mini-AGI (online)   1.26B    2.8 GB          ~$56     1.14\n    GPT-2 Medium        0.35B    40 GB           ~$10k    1.17\n    mini-AGI (dynamic)  1.26B    2.8 GB          ~$56     1.23\n    mini-AGI (frozen)   1.26B    2.8 GB          ~$56     1.29\n    GPT-2 Small         0.12B    40 GB           ~$3k     1.33\n```\n\n2. PG19 (100 Project Gutenberg books, 41 MB test split):\n\n```\n    Model               Params   Trained On        Cost    PG19 (bits/byte) ↓\n\n    MambaByte-972M       0.97B   75 GB (PG19)      ~$4k    0.883\n    MegaByte-1B          1.02B   80 GB (PG19)      ~$1.6k  1.000\n    Transformer-320M     0.32B   80 GB (PG19)      ~$1.6k  1.057\n    mini-AGI (online)    1.26B   2.8 GB (0 PG19)   ~$56    1.447\n    mini-AGI (frozen)    1.26B   2.8 GB (0 PG19)   ~$56    1.995\n\n    The published models were trained on PG19's own training books, 75 to 80 GB of them, and are scored on more books like those. mini-AGI has never been trained on PG19 and has read no Victorian novels, so for it these books are completely out of its domain.\n```\n\nThere are more details in the repository. The most important part is a difference between \"frozen\" and \"online\" attempts where we allow the model to specialize on the task as it goes through the evaluation dataset. Clearly showcasing the power of continual learning regime simply unavailable for any other typical LLM.\nFull methodology, and replication code is in the repo:\n[https://github.com/volotat/mini-AGI](https://github.com/volotat/mini-AGI)\n\nIntermediate weights snapshot on Hugging Face:\n[https://huggingface.co/Volotat/mini-AGI-undertrained](https://huggingface.co/Volotat/mini-AGI-undertrained)\n\nComments URL: [https://news.ycombinator.com/item?id=50219762](https://news.ycombinator.com/item?id=50219762)\n\nPoints: 2\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-benchmarks-of-continual-learning-model-trained-for-a-month-on-a-laptop", "canonical_source": "https://news.ycombinator.com/item?id=50219762", "published_at": "2026-10-24 10:09:27+00:00", "updated_at": "2026-10-24 10:41:41.487633+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["mini-AGI", "Nvidia RTX 3070", "enwik9", "PG19", "MambaByte-972M", "Llama-3.2-1B", "GPT-2", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-benchmarks-of-continual-learning-model-trained-for-a-month-on-a-laptop", "markdown": "https://wpnews.pro/news/show-hn-benchmarks-of-continual-learning-model-trained-for-a-month-on-a-laptop.md", "text": "https://wpnews.pro/news/show-hn-benchmarks-of-continual-learning-model-trained-for-a-month-on-a-laptop.txt", "jsonld": "https://wpnews.pro/news/show-hn-benchmarks-of-continual-learning-model-trained-for-a-month-on-a-laptop.jsonld"}}