All-in-One/In-the-box or DIY build for localized AI

A Level1Techs forum discussion advises that building a local AI rig with consumer hardware like an AMD 9700X and a Supermicro AM5 motherboard is a viable low-cost alternative to turnkey systems such as the NVIDIA DGX BasePod, which can cost around $1 million and take 12 months to deploy. One user recommends using an RTX 3090 or RTX 4090 to run local models for chat, image generation via Stable Diffusion, and speech, eliminating subscriptions to ChatGPT, ElevenLabs, and Midjourney, while another notes that RAM prices remain high, comparing them to printer ink, and predicts an AI bubble collapse within 3 to 4 years.

thr3e https://forum.level1techs.com/u/thr3e 22 There is no downside to trying it on the hardware you have. My sacrificial test bench was a supermicro am5 matx board and 9700x that cost like 300 bucks with no ecc anywhere in the equation. Something like this is a great little homelab if you are just dipping your toe in the water: 2 Likes Right now, we are suffering through an AI bubble that will collapse within the next 3 to 4 years. Just hold out until RAM prices normalize, or buy second-hand parts to save some money. 1 Like Are you running any sort of virtualization, proxmox etc…, on your AI rig or are you running a straight OS? I would like to think that having multiple vm’s would give options for experimentation and different AI paths, but having not done it yet I have no clue. And may I ask what your experience with NVME vs SSD vs HDD NAS? My initial thinking was NVME for OS, and either SSD or NAS for models/scratch/other shit. My current network is all 10GB SFP+, but is that fast enough with HDD 7200 NAS? thr3e https://forum.level1techs.com/u/thr3e 25 Are you on discord or something because I could explain this all way easier interactively. I guess DM me or something if you want to talk in greater depth. I have a custom storage solution for tier 1 ssd with multi hundred gig networking and RDMA. I have a custom compute solution on epyc running proxmox 9. its doing GPU passthrough on ADA generation hardware unrelated to the actual rig. The primary AI rig is bare metal for a whole lot of reasons. It has tier 0 nvme gen5 flash on it. yes the GPUs can and will suck 30+GB/s of data off the drives. If you are doing any kind of training, checkpointing with all reduce is going to absolutely trash your disks and pcie bandwidth. All of these things are engineered for their specific use cases in my environment, attached to months of lessons learned and multiple re-designs to get to where they are now. If someone is doing this professionally, do the right thing and buy a “turnkey” solution like dgx basepod I know they are not actually turnkey, every implementation where someone expects that its usually 12 months from initiation to production. I did this in no small part to understand all the WHY’s behind a million dollars per box with very particular design choices was needed for AI, and what could I do at 4% the cost. I also have a fairly competent understanding of everything from the chips all the way up the stack through compute storage networking virtualization security application - all of these were tested extensively in figuring out how to have Claude-at-home. 3 Likes Hey, If your aim is to reduce or completely move away from subscriptions like ChatGPT, ElevenLabs, and Midjourney, the most practical approach is to set up a local AI system on your own computer using a powerful NVIDIA GPU such as an RTX 3090 or RTX 4090. With a setup like this, you can run AI models directly on your machine instead of relying on cloud services. This means you can use it for chatting and learning programming or Linux, generating images similar to Midjourney using tools like Stable Diffusion, and even creating voice outputs with local speech models. The main benefit is that your data stays with you, and you are not dependent on monthly subscriptions or usage limits. At the same time, it is useful to understand how cloud-based AI systems work. Some platforms operate as all-in-one AI services that provide multiple AI capabilities like chat and other functions through APIs, which can then be connected to different tools or workflows. For example, this all-in-one AI platform https://geekflare.com/ai/chat/ can be explored here, to understand how such API-based AI systems are structured. 1 Like I have a 5090 that I am now experimenting with. If ram wasn t the same price-per-gram as printer-ink, I would get more ram to run larger models lol.