Sanity check on my math: DGX Spark vs 2x Radeon PRO R9700 vs Mac Studio (M3 Ultra) for multi-user office LLM serving, does reality match the numbers?

A hardware-sizing analysis by an unnamed engineer claims that two AMD Radeon PRO R9700 GPUs ($1,300 each) and a Mac Studio M3 Ultra (256GB, $6k) could outperform NVIDIA's DGX Spark (128GB, 273GB/s) for multi-user office LLM serving, with theoretical throughput of 746 tok/s (37 concurrent users) and 478 tok/s (23 users) respectively versus 159 tok/s (7 users) for the Spark, but the author flags uncertainty about real-world scaling and software maturity, especially for Apple Silicon continuous batching.

TL;DR : Pricing out small on-prem LLM deployments for office clients vLLM/continuous batching, ~20 tok/s/user target . My bandwidth-bound math says two of these options wildly outperform a DGX Spark for the same money, but I know the calc is idealized and real serving software doesn’t always hit it. Two things I specifically don’t trust: 1 dual-GPU PCIe scaling, 2 whether Mac Studio’s continuous-batching software is anywhere near mature enough to hit what the raw bandwidth numbers promise. Hoping someone’s actually run these and can tell me where I’m wrong. What I’m trying to do Sizing hardware for small-office multi-user assistants 10-30 concurrent people, everyday office work + some coding . Budget lands around $8-9k USD equivalent. Options on the table: - 1x NVIDIA DGX Spark 128GB unified memory, 273GB/s bandwidth - 2x AMD Radeon PRO R9700 32GB GDDR6 each, 640GB/s each, ~$1,300/card , via vLLM --pipeline-parallel-size 2 no NVLink between them - 1x Mac Studio, M3 Ultra, 256GB unified memory, 819GB/s bandwidth ~$6k config The math I’m running aggregate tok/s = bandwidth / active params bytes bytes per param concurrency = aggregate tok/s / target tok/s per user For a ~35B MoE model 3B active, ~20GB weights, 48 layers, 8 KV heads, 128 head dim : Spark: 273e9 / 3e9 0.57 ≈ 159 tok/s → ~7 concurrent users Dual R9700 : 1280e9 / 3e9 0.57 ≈ 746 tok/s → ~37 concurrent users Mac Studio: 819e9 / 3e9 0.57 ≈ 478 tok/s → ~23 concurrent users The dual R9700 number assumes a naive linear combination of the two cards’ bandwidth. I know pipeline parallelism doesn’t give clean linear scaling in practice. There’s supposed to be pipeline-bubble overhead and vLLM’s own docs recommend pipeline parallelism over tensor parallelism specifically when there’s no NVLink, for exactly this multi-user-throughput use case. But I can’t find a real benchmark for this pairing, especially on AMD/ROCm rather than NVIDIA. The number that really doesn’t sit right with me is the Mac Studio one at bigger model sizes. Running the same formula against a ~120B-class model 5.1B active params, ~65GB weights on the Mac Studio’s 819GB/s and 256GB memory gives: aggregate = 819e9 / 5.1e9 0.54 ≈ 296 tok/s → ~14 concurrent users 14 concurrent users running a genuinely large 120B-class model, on a single desktop box that isn’t even primarily marketed as an AI server? That’s the hardest number to believe out of this whole exercise. Why I’m suspicious specifically of the Mac number The formula assumes vLLM-grade continuous batching, i.e. that the serving software can actually keep the bandwidth saturated across many concurrent users. That’s true and battle-tested for vLLM on CUDA. On Apple Silicon I can’t find anything that assures clarity: the most advanced continuous-batching project I could find for MLX reports something like 4.3x aggregate throughput at 16 concurrent requests, nowhere near the ~16x linear scaling my formula assumes. If that gap is representative, the real Mac Studio number could be way below 14, not just “a bit lower.” What I actually want to know - Has anyone run vLLM or llama.cpp with --pipeline-parallel-size 2 across two PCIe-only GPUs for concurrent multi-user serving not single-stream ? What fraction of the naive linear-sum throughput did you actually get? - Specifically for AMD/ROCm multi-GPU, is the R9700 very new card actually stable for this yet, or is ROCm multi-GPU still rough around the edges compared to CUDA? - Has anyone run mlx-lm, llama.cpp Metal, --cont-batching , vLLM-metal, or vllm-mlx on an M3 Ultra Mac Studio under genuine multi-user concurrent load not single-stream benchmarks ? What tok/s per user did you actually get at, say, 10+ simultaneous requests? - Does anyone have real numbers running a 100B+ class model specifically on Mac not just “it loads and runs” single-stream demos, which I’ve seen plenty of ? - Is continuous batching on Mac any backend actually production-viable today for a multi-user office deployment, or is this still “works great for one person, falls over past a handful of concurrent users” territory? - Is my ~20 tok/s/user target reasonable for “feels responsive,” or am I over/under-shooting what real users actually need, in a way that would change any of this? Happy to share more of my numbers/assumptions if useful, trying to get this right before I put real money behind a recommendation to clients. Also very grateful that you took time out of your day to help me with my problem.