Help me spend $45k on local AI

A local-LLM practitioner's forum post argues that a $45,000 local AI build should skip server-grade CPUs, DDR5 and ECC memory, because full-inference workloads on a 12700KF/64GB/RTX Pro 6000 system show only 1-3% CPU and RAM utilization, with pressure limited to model load time. The post contends that scaling Strix Halo, Gorgon Halo, DGX Spark or Mac Mini clusters is largely a "look what I can do" exercise, citing 273GB/s memory bandwidth and 1PFLOP tensor compute for 8 Sparks versus 1,792GB/s and 4PFLOP for an RTX 6000 Pro, and recommends a single RTX 6000 Pro for turnkey performance on Qwen 3.8 27B or Flash-Next 120B.

I’d like to add that you’re not going to see any performance benefits from a current generation EPYC or Xeon build, DDR5, ECC, or 32GB of RAM. The reason being that a high concurrency full-inference workload has only 1-3% CPU and RAM utilization in my 12700KF 64GB RAM 6000 RTX Pro system. The only pressure on CPU/RAM comes at model load time, which only comes at boot time, which only happens very infrequently if you’re looking for system uptime as a goal. The only conceivable reason to go with a server board setup would be for long term endurance and uptime; in which case it would be better to grab a Zen2/Zen3 EPYC board and CPU ie. MZ32-AR0 that takes DDR4 and throw 32 or 64 GB in it. I, personally, feel like even that’s overkill because I’ve had no issues with my old hardware chugging along. If there’s any doubt about the hardware requirements outside of the GPU in an inference build I ran some of my own load tests to see if I could get CPU/RAM to move the needle here: So, like, how do I measure my local LLM environment and expand it? https://forum.level1techs.com/t/so-like-how-do-i-measure-my-local-llm-environment-and-expand-it/254682 The Strix Halo and Gorgon Halo platform is very similar to the Sparks in terms of memory bandwidth and VRAM offerings. The Sparks have much better compute so in prefill they’ll outperform the Halo range. Same with the Mac Mini clusters where Sparks will outperform on prefill so I can see why you’d start there because influencers like to talk about them as a great entry level bang for your buck. But the bottlenecks you get will turn the cost efficiency into an anchor very quickly in those builds. My experience has been that single units are very good entry level units to get familiar with local LLMs and if they fill your need/use case then the cost is great. You can sit there very nicely. Scaling them into clusters is almost entirely a “look what I can do” activity where you can load increasingly large models and set up a neat and complex cluster to make it all work. The actual usability… not so much. If you want to scale at reasonable cost the next step isn’t to add more boxes to a cluster mac mini, spark, halo platforms : it’s to switch to GPUs that can handle the memory bandwidth and tensor compute bottlenecks you’ll hit in those platforms. R9700s are popular here, as are other AMD offerings that are less commonly discussed in other places. From what you’ve said about ease of use/common deployments being important to giving you a wide base of knowledge to extract from, I’m not sure this is the route you’d want to go. My experience has been that if you want to easily run a software stack, or multiple ones, then AMD isn’t the way you want to do it. The hardware is there, but the software maturity continues to mean more tinkering is necessary to get what you want and get it running well. It becomes a question of tinkering and cost efficiency versus high cost and out-of-the-box turnkey performance. On the latter, you would have a hard time beating the RTX 6000 Pro. A single one will cut through Qwen 3.8 27B or Flash-Next 120B like butter. You’ll find kvcache overhead to handle a dozen agents at once is there, and compute holds together fine. If you want a larger model you can grab a second one and throw in the GLM or Deepseek flash models. These GPUs will deliver very strong prefill, very interactive and usable levels of inference t/s at high concurrency for many users, and they’ll be relatively stupid-proof in terms of setup and use. They’ll just work. Any other GPU setup you see here is going to work just as well if you’re interested in spending more time on setting it up and spec’ing it out. My only caution here is to not get caught up in the VRAM headline spec - at 8 Sparks you see 1TB of total VRAM and it looks like the best route. What you’ll find, though, is that the other bottlenecks will get you. Memory bandwidth is 273GB/s vs 1,792GB/s and tensor compute is 1PFLOP vs 4PFLOPs at fp4 sparse. Total VRAM is going to mean very little if the loaded model has 1/4 the compute and 1/6 the bandwidth. I would consider it like this: I consider parallelism to only be about increasing concurrency - run more agents, get more kvcache. Don’t consider it in terms of larger usable models or faster inference performance. Moreso than anything else I’d argue that your use case is going to drive the build rather than headline numbers. Where you put your workloads will be defined by what your workloads are. That varies from person to person and company to company. For reference, I use my single RTX 6000 Pro to run only a handful of agents at a time. I prefer high t/s interactivity because I am fitting multiple pieces together for more complex problems to address and I prefer my expertise/experience to guide these processes. Mostly coding and devops. Heavy on planning and analysis. Very little image and video. Anything long horizon where I don’t care about timing like doing an assessment by crawling a code base I have no familiarity with gets sent off to a Strix Halo instance. But then once I’m working on it actively it comes back to the 6000.