# I Trained a Language Model From Scratch on a 4 GB GPU > Source: > Published: 2026-10-12 08:07:10+00:00 *A 4 GB GPU, a small corpus and a lot of things that went wrong in instructive ways.* I trained a 22-million-parameter language model from scratch on a laptop with 4 GB of VRAM. It learned to imitate technical documentation, got stuck repeating fragments of language, and never produced a working Fibonacci function. Near the end, CUDA crashed. The result wasn't a useful chatbot. The failures were the interesting part. Pulling a pretrained model through Ollama makes modern AI feel deceptively simple. Training even a tiny one from random weights removes that illusion very quickly. That’s why I started AI Lab, a separate repository for research code, datasets, experiment definitions, models and results. AI.001 was my first experiment: training a small language model from random weights on my laptop. This is the story behind that run, including what the model actually generated. The complete methodology, corpus breakdown and training curves live in the [AI.001 experiment write-up](https://sandrabamboo.com/experiments/training-a-language-model-from-scratch-on-4gb-of-vram/). APIs and pretrained local models are wonderful, and they hide almost everything I wanted to see. What does the dataset actually look like? How much does cleaning matter? What happens between a document and a token? What is a checkpoint, really? What does loss tell me, and what does it quietly not tell me? What does generation look like before a model is useful at all? I couldn’t answer those by calling something that already worked. The only way I know to find out is to build the unglamorous version and watch it misbehave. The machine is a Lenovo IdeaPad Gaming laptop: an Intel i7-10750H, 16 GB of RAM and a GTX 1650 with 4 GB of VRAM, running PyTorch with CUDA under WSL2. It is not a good machine to train on, and I’m not presenting it as one. That’s the point. With a big GPU you can hide a lot of sloppy decisions behind headroom. With 4 GB you have to know why the context length is what it is, why the batch is split into microbatches, and what the optimizer state costs. I settled on a 256-token context and microbatches of 16 with gradient accumulation, so each update still sees a reasonably large batch. Peak memory for the main run was 1.54 GiB, which is comfortably under the card’s limit. I don’t read that as the card being generous. I read it as the result of designing the pipeline around the limit from the start rather than discovering it halfway through. Spanish and English Wikipedia, Python documentation and a handful of other technical documentation sets: collecting text is the easy part. Anyone can download a dump. The interesting part is deciding what is allowed into the training set and what that decision does later. I chose a controlled corpus over a large unfiltered web dataset, and judged sources on quality, licensing, language balance, duplication risk and how painful they were to preprocess. Cleaning and filtering is where the corpus stops being “everything I could download” and becomes something I made on purpose. I’m not going to dump corpus statistics here; the experiment page has them. What I’d rather say is that those choices came back later, in the evaluation, as behaviour I could point at. *The workflow behind AI.001. Checkpoints make recovery possible; held-out loss and generated samples answer different questions.* Validation loss started at 9.48. With a vocabulary of 12,000 tokens, uniformly guessing the next token would give a loss of about 9.39. After roughly one pass over the corpus, the model reached 4.27. That was progress, and it was satisfying to see. But it answered one narrow question — whether predictions on held-out text had improved — not whether the model could generate anything useful. Checkpoints stopped being an abstract idea at around the same time. I had tested resuming from a checkpoint on a short run before committing to the long one, and it printed exactly what I hoped to see: ``` Resumed from: results/EXP-001/resume-test/latest.pt Resume step: 10 ``` That turned out to be sensible. Shortly after step 3,540 of the main run, CUDA reported an unknown sticky error and training stopped. It wasn’t an out-of-memory failure and the loss had been behaving. I restarted from the step-3,500 checkpoint, with the optimizer, mixed-precision scaler and random state restored, and it ran to the end. The more instructive surprise was generation. With sampling, later checkpoints produced fragments that looked like technical documentation; an early one mixed WordPress, MDN and Docker material incoherently. With greedy decoding — always taking the most likely next token — the final model often fell into loops. Validation loss did not reveal that failure mode. During generation the model has to continue from its own output, not from the real text used for evaluation. Here is what a few prompts produced: Those examples were more revealing than the loss curve on its own. So I evaluated the final checkpoint by corpus component instead of relying only on an aggregate score. Technical documentation had lower loss than general prose, but there was a wide spread between sources: MDN and WordPress text was more predictable to this model than Docker, Python or Git documentation. [The source-by-source results in AI.001](https://sandrabamboo.com/experiments/training-a-language-model-from-scratch-on-4gb-of-vram/) show the actual measurements. I want to be careful with that ordering. It is an observation from one checkpoint of one run, and it is not evidence that the model is good. What it does is raise better questions than the aggregate loss did. Is a source easy because it is well-written, or because it is repetitive? How much of a low loss is the model knowing something and how much is a predictable format? Did I give some sources more room than they deserved? Is there overlap between domains that I haven’t noticed? Loss can’t separate those on its own. A 22-million-parameter model, trained for about one pass, has nowhere to hide. It learned how technical documentation looks far better than what it means. It writes convincing headings and parameter lists for APIs that don’t exist, drifts between domains within a paragraph, and never produced a working `def fibonacci(n):` at any checkpoint I tried. That is useful for learning precisely because it is bad. A strong model smooths over its mechanics. A weak one shows them: capacity, data mix, repetition, the gap between predicting text and knowing things. This is research, not product development, and I try to keep that straight in my own head. Not a competitive model, and not something I would use to answer questions. What I got was a research workflow: repeatable corpus tooling, training scripts that can pause and resume, and evaluations broken down by source. More importantly, I now know to look at both held-out loss and actual generations before deciding what a model has learned. That feels more useful than a demo that only looks convincing. I’m not promising a bigger model. The questions the first run left open are about corpus selection, better evaluation, and changing one decision at a time so I can tell what each change did. [The experiment page lists a more focused follow-up](https://sandrabamboo.com/experiments/training-a-language-model-from-scratch-on-4gb-of-vram/), with more source code in the mix and a fixed evaluation I can compare against this one. I’d rather make a small, controlled step that I understand than a large one that I can only describe. *Originally published on sandrabamboo.com: [https://sandrabamboo.com/training-my-first-language-model-on-a-gtx-1650/](https://sandrabamboo.com/training-my-first-language-model-on-a-gtx-1650/) — for model architecture, corpus sources, training curves and per-source evaluation, see the complete AI.001 experiment: [https://sandrabamboo.com/experiments/training-a-language-model-from-scratch-on-4gb-of-vram/](https://sandrabamboo.com/experiments/training-a-language-model-from-scratch-on-4gb-of-vram/)*