Post / Show off your Ai Rig - Whatchya got in there?

A user detailed a home AI inference rig built on an AMD Epyc Rome 7532 CPU with 128GB DDR4, roughly 5TB of NVMe storage, and 4x AMD V620 GPUs on an open-air crypto-mining bench. The rig runs a LiteLLM frontend to vLLM serving Qwen3.8-flash-next at int4 quantization with the n-gram table offloaded to system memory, reaching about 70 tokens per second for single-request inference and roughly 90TPS peak across 2-3 simultaneous streams. The user reported that multi-token prediction (MTP) is not reliably performant with multiple simultaneous requests and said they are still investigating the issue.

Oh, you wanna see my m~~e~~ath lab? Epyc Rome 7532 128GB DDR4 Various NVMe totaling ~5TB of storage 4x AMD V620 All in an open air crypto-mining bench. No, this does not trip breakers, but it comes close. Current stack is a LiteLLM frontend to vLLM I’ll do a write-up on how I optimized vLLM soon , serving Qwen3.8-flash-next at int4 quant, with the n-gram table offloaded to system memory. Gets me ~70TPS of single-request inference and ~90TPS peak for 2-3 simultaneous streams. This is without MTP, because MTP is not reliably performant with multiple simultaneous requests being served. Still trying to figure that one out.