{"slug": "poppy-the-training-box-part-1-the-beginnings", "title": "poppy the training box, part 1: the beginnings", "summary": "A developer repurposed an old small-form-factor PC named 'poppy' into a dedicated machine for local LLM training, upgrading its case and power supply to accommodate future multi-GPU setups. The project aims to free up the daily driver PC 'perry' from lengthy training runs and enable longer experiments, with plans for custom water-cooling and multi-GPU testing.", "body_md": "For a while I've been planning to put together a separate machine for local LLM\ntraining. Until now, I've been using my desktop PC, `perry`\n\n. I have an RTX 3090\ninstalled, and can get useful training runs done (most recently,\n[a 163M-parameter GPT-2 small style LLM in JAX](https://www.gilesthomas.com/2026/07/llm-from-scratch-34b-building-and-training-gpt-2-small-in-jax)),\nbut there are a couple of problems.\n\n`perry`\n\nis my daily driver. If he's doing a training run, then everything is\njust a little bit sluggish as CPU and GPU alike are busy.And relatedly to all of those: the two-day limit to the training runs I've been doing is something I set\nbecause that's the maximum amount of time I'm willing to have `perry`\n\ntied up. It\nwould be really interesting to try longer training runs!\n\nI also have longer-term plans; a multi-GPU box would be interesting to put together --\nnot just to have more power locally, but so that I could test larger-scale cloud\nmulti-GPU\ntraining runs before starting to pay for expensive machines. US$15.92 an hour to rent a machine\nisn't a *lot* of money, but it adds up, especially if you're spending it while debugging\nparallelism issues.\n\nAnd finally, I've always been interested in putting together a custom water-cooling loop in a PC. I've been building my own machines since 1995 or so, but never got round to that side of things. It sounds fun!\n\nBut despite all of those future plans, this is a fairly normal machine-building post -- how I repurposed an old PC, plugged in a second-hand RTX 3090 from eBay, tested it all, accidentally trained an LLM for 11 days, and almost cooked a CPU.\n\nOver time, I expect to be posting more -- and more interesting -- build details. Let's think of this as establishing the baseline.\n\n`poppy`\n\nBack before I moved to Lisbon, we had a holiday home here. When we came over, I'd bring my laptop, but that was always somewhat unsatisfactory -- limited CPU power for work, limited GPU for my occasional gaming.\n\nDuring Covid, we started staying in the holiday home for longer periods -- and this became\ntoo big of an annoyance to ignore. So in 2020 I put together a small form-factor PC, which I\nnamed `poppy`\n\n. The constraints were:\n\n`poppy`\n\nthere normally, but\nmove her when we had guests for dinner.The build was a bit fiddly, like all SFF PCs. You can see the component list\nand build notes [here on PCPartPicker](https://pcpartpicker.com/b/J2Wbt6), but in short she had:\n\nShe looked like this:\n\n(Gosh, I'd forgotten how... vivid our wallpaper was in that dining room.)\n\nFor scale -- that case is slightly taller than two cans of coke stacked on top of each other. So, pretty small.\n\nWhen we moved to Lisbon full-time, I brought `perry`\n\nwith me from London, and while he's been upgraded\nseveral times since (including adding an RTX 3090 [in late 2023](https://www.gilesthomas.com/2024/02/llm-quantisation-weirdness)),\nhe's been my daily driver since. So `poppy`\n\nsat in the corner of my study, sad and unused :-(\n\nIt was time to bring her out again. Initial plan: get her up and running in a new, larger case, with a PSU that could potentially handle three graphics cards.\n\nInitially, I found that she wouldn't switch on: a quick check suggested that the problem was the PSU. I'd had problems with SFF PSUs in the past, and given that the plan was to give her a new one, I just got one, along with a new, larger case -- specifically:\n\n`perry`\n\nwas already in a North (not the XL variant) and I love the case; the XL\none looked like a good option if I was going to be cramming more GPUs in there,\nand had plenty of space for water-cooling.A few days later, the parts arrived. Here's a family photo:\n\n`perry`\n\nis to the left, `poppy`\n\ncentre, sitting on top of her new case, and Cornélia (wearing her Flower of Shame)\nis to the right. For scale, Cornélia is quite a large cat. (I appreciate that that is not\nimmensely helpful.)\n\nTime to put the old motherboard and the new PSU into the new case. Here's what it looked like:\n\nThe Mini-ITX motherboard in a case designed for full ATX looks comically like a postage stamp.\n\nI switched her on, and luckily enough, everything worked! Must have been a PSU issue.\n\nThe OS that she had was a more than three-year-old version of Arch, so I wiped the drives and installed the most recent version with my normal config, and it was time for a quick test.\n\nOne of the nice things about having done all of this LLM training stuff recently is\nthat you have a ready-made burn-in test for new hardware :-) I didn't have my\n[JAX training code](https://www.gilesthomas.com/2026/07/llm-from-scratch-34b-building-and-training-gpt-2-small-in-jax) yet,\nbut I did have the [PyTorch one](https://www.gilesthomas.com/2026/04/llm-from-scratch-32k-interventions-training-our-best-model-locally-gradient-accumulation).\n\nNow, with her GTX 1660 Super GPU, `poppy`\n\nwas clearly not going to be able to train\nan LLM of the size I could with `perry`\n\n's RTX 3090. I did some fiddling around with\nthe model and training run parameters, and found that I could fit in a cut-down version\nof GPT-2 small with this setup:\n\nI trained it with a microbatch size of 4, gradient accumulation over 16 steps, and\nall other hyperparameters the same as my normal training runs on `perry`\n\n. The number\nof training tokens went down -- the model had 76,933,120 parameters, so I needed to\ntrain for just over 20x that -- about 1.5B instead of the 3.2B I've been training my other\nmodels on.\n\nI kicked that off, and out of interest, I kicked off another training run on `perry`\n\nwith the same setup to see what happened.\n\nThe `perry`\n\ntraining run went normally -- GPU running at full blast, 368W, and it completed\nin about 9 hours. That's less than 1/4 of the time my normal training runs take, which\nmakes sense because time taken for this kind of thing scales roughly linearly with both the size of the model and\nthe number of tokens, and both of those were about half the normal size.\n\n`poppy`\n\nwas a bit more interesting. In `nvtop`\n\n, the GPU usage showed up as 100%, but\nwith an \"effective\" utilisation of 53%. The power draw matched the latter, being\n67W out of a total possible 125W. I'm not quite sure what was causing that -- clearly\nthere was a bottleneck somewhere. Not really worth digging into, though, given that I\nwas going to replace the card shortly.\n\nAnyway, that took 963,257 seconds to run. That's 267.57 hours, or just over 11 days.\n\nYikes.\n\nWhat's kind of interesting there is that this training run not only took much longer (which is only to be expected), but that it used more electricity. 67W over 267.57 hours is just short of 18kWh, whereas 368W over 9 hours is about 3.3kWh.\n\nBuy an RTX 3090, save the planet!\n\nI decided to run my normal evals to confirm that what had come out the other end\nwas sane. When asked to complete \"Every effort moves you\", `perry`\n\n's model said:\n\n```\nEvery effort moves you and your friends. The key is keeping you informed. But, you still need to be cautious about\n```\n\nAnd `poppy`\n\n's said:\n\n```\nEvery effort moves you forward. Every week, you make adjustments to a certain level—just as soon as you leave your\n```\n\nThose were actually rather good, I thought! And looking at my normal loss test\nconfirmed that the models really weren't that bad; `perry`\n\n's got 3.855702, and\n`poppy`\n\n's 3.855981. That was actually better than the 3.943522 I'd got on `perry`\n\nbefore [I went down my rabbit hole of optimising hyperparameters](https://www.gilesthomas.com/2026/02/llm-from-scratch-32a-interventions-baseline-model).\n\nSo, that was an interesting test -- I was talking to ChatGPT about it at the time and it called it \"maybe an art project\", which I thought was amusing if a bit arch.\n\nTime to do something a bit more useful.\n\nFinding an RTX 3090 for a decent price from a trustworthy-seeming vendor is kind of hard right now. But it's still the sweet spot for price-performance if you're looking to train models locally, so I set up an alert on eBay, and eventually one popped up in Bulgaria. I bought it, and a few days later, this turned up:\n\nIt's actually not as ugly as it looks in that photo -- it's considerably uglier. The stuff that looks a bit like crinkled aluminium foil is really white plastic with a kind of crystalline texture. Made me glad that I'd gone for the mesh-sided case rather than the glass one.\n\nWell, I hadn't bought it for the looks. I removed the old GTX 1660, and put in the new card, switched it on, and:\n\nWow, a disco in my PC. Lovely.\n\nIt was time to kick off another training run to see if it worked. This time, I did\nmy normal GPT-2 small sized train with optimised hyperparameters. It ran for about ten\nminutes, and then `poppy`\n\nswitched herself off.\n\nThat didn't look good.\n\nI spent some time digging around trying to work out why my new graphics card was broken, and then happened to be sending the video above to a friend, and spotted something.\n\nCheck out the Noctua fan -- the beige and brown one you can see behind the cooler mount, above the graphics card. It wasn't spinning. That's the CPU cooler fan and should always be spinning, even if slowly, when the machine is on.\n\nI log basic metrics for all of my PCs to a central InfluxDB instance, so I checked that out and:\n\nA CPU temperature spike up to about 115°C! Not good. Clearly an emergency thermal shutdown from the CPU.\n\nI initially thought that I must have knocked the fan cable loose while plugging in the new GPU -- plausible, though they were quite far apart -- but unplugging then reseating it, then powering up the machine still didn't start the fan spinning. And it was not visible in the BIOS.\n\nI then zoomed out a bit in Grafana; I only keep 30 days' worth of metrics, and it had been more than a month since I did my original burn-in test, so I didn't have anything for that. But I did have this:\n\n`poppy`\n\nhad been idle for all of that time, and was averaging CPU temps of over\n70°C. The dropoff prior to running the test was because she'd had a chance to\ncool down while I installed the GPU. Having spent ages setting up my InfluxDB\nmonitoring stuff so that I have metrics for everything, I should probably actually look at them every\nnow and then, because the fan had obviously not been doing anything for a month or so.\n\nWell, thank goodness for Amazon next-day delivery. I bought a new\n[Noctua NF-A9x14 PWM](https://www.amazon.es/-/pt/dp/B009NQM7V2) (praying that\nthe problem was the fan and not the header on the motherboard), and when it arrived,\nI put it in.\n\nThis time, when I powered her on, the fan was spinning. Phew. I left her running for an hour, and the CPU temperature stabilised at 35.5°C. Next, I kicked off a version of my standard LLM training run with the number of tokens reduced so that it would run for an hour.\n\nDuring that, the CPU temperature went up to a moderately-toasty 76°C -- not ideal, but remember that with the broken fan, she was running that hot at idle. It seemed a bit odd that it was that hot at 10% CPU usage, but given that one core was running at 100%, it didn't seem totally off. The heatsink and fan are designed for SFF PCs anyway, and those tend to run somewhat hot.\n\nThe GPU temperature also went up to 70°C and stabilised there, while power draw was stably about 368W out of 370W, and GPU utilisation at 100%. That was particularly pleasing because Nvidia cards throttle at 83°C or so by default, so if I was getting a lower temperature at full power, the fans clearly had some headroom for cooling.\n\nOnce that was completed, it was time for another full training run for a burn-in.\n\nI kicked off my normal run. CPU and GPU temperatures stabilised at the same level as they had with the one-hour test, which was promising, so it was just a question of waiting...\n\n...until I got this:\n\n```\nTraining complete in 144,531.113 seconds\nTokens seen: 3,260,252,160\nThroughput: 22,557 tokens/second\nFinal train loss: 3.530\n```\n\nAbout 40 hours, which is pretty much standard -- certainly the same as I'd expect\nfrom `perry`\n\n. The smoke test:\n\n```\nEvery effort moves you and your customers to the best possible extent.\nOur expertise and expertise have led to our ability to\n```\n\nDon't you just love it when your LLM tries to sell you something? 1\nBut anyway, loss on the test set was 3.548880, which is essentially the same as\nthe same training run on\n\n`perry`\n\ntoo.Success!\n\nSo, now `poppy`\n\nis a properly-configured training machine -- one RTX 3090, a CPU\nthat runs a bit hot but at least doesn't do emergency shutdowns, and a case and\na PSU with enough space for more GPUs.\n\nI think that the next step will be to move on to water cooling. In order to support more than one GPU, I'll need a new motherboard and probably a new CPU, so I don't think there's any point in watercooling the latter, despite its toastiness -- I'd just be buying a waterblock for it that I'd throw away in the not-too-distant future.\n\nInstead, I'll get the block for the GPU, and set up a loop to cool just that. Who knows -- maybe I can get rid of that horrendous RGB stuff at the same time!\n\nWe live in hope.\n\nAlso, that \"expertise and expertise\" tiny model smell. [↩](https://www.gilesthomas.com/feed/rss.xml#fnref-1)", "url": "https://wpnews.pro/news/poppy-the-training-box-part-1-the-beginnings", "canonical_source": "https://www.gilesthomas.com/2026/07/poppy-the-training-box-1-the-beginnings", "published_at": "2026-07-09 00:23:12.540097+00:00", "updated_at": "2026-07-09 00:23:14.548577+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-research", "developer-tools"], "entities": ["poppy", "perry", "RTX 3090", "eBay", "PCPartPicker", "Fractal Design North XL", "Corsair SF750"], "alternates": {"html": "https://wpnews.pro/news/poppy-the-training-box-part-1-the-beginnings", "markdown": "https://wpnews.pro/news/poppy-the-training-box-part-1-the-beginnings.md", "text": "https://wpnews.pro/news/poppy-the-training-box-part-1-the-beginnings.txt", "jsonld": "https://wpnews.pro/news/poppy-the-training-box-part-1-the-beginnings.jsonld"}}