Hitting 55-90 tok/s on large models without even touching speculative decoding is a massive win for our current deployment. We've been testing a new inference stack specifically for DGX Spark clusters to handle multi-model agentic workflows, and the LlamaBench numbers are actually holding up under simulated traffic.
We also tested a scenario with one endpoint controlling three different model activations. The scheduler batches bursts and coordinates ranks, only swapping the resident model at safe boundaries. The results were:
This is a solid deep dive into how we're scaling our AI workflow internally. We're expecting another 20% bump in performance once we finish the next round of optimization.
The real-world problem we're solving at work is that enterprise agentic apps usually need a mix of specialized models of different sizes. Buying dedicated multi-GPU stacks for every single model in a workflow is way too expensive. This setup lets us run a private, lower-cost stack that handles multiple models across a 2 DGX Spark cluster.
Here is the raw performance data from the LlamaBenchy tests (no quantization, no spec decode):
DeepSeek V4 Flash C4: 55.99 decode-system-tok/sGemma 4 26B A4B C4: 63.75 decode-system-tok/sNemotron 3 Nano Omni 30B NVFP4 C4: 90.83 decode-system-tok/s We also tested a scenario with one endpoint controlling three different model activations. The scheduler batches bursts and coordinates ranks, only swapping the resident model at safe boundaries. The results were:
DeepSeek V4 Flash: 49.30 decode-tok/s (16s activation wait)Gemma 4 26B A4B: 64.67 decode-tok/s (6s activation wait)Nemotron 3 Nano Omni 30B NVFP4: 93.31 decode-tok/s (2s activation wait) This is a solid deep dive into how we're scaling our AI workflow internally. We're expecting another 20% bump in performance once we finish the next round of optimization.
For those wanting the full technical breakdown, the report is here:`https://woolyai.com/ai-compute-software/dgx-spark-inference-stack/`
[Next Framework Desktop Build: Ryzen AI Max+ Pro 495 & 192GB RAM →](/en/threads/2194/)
All Replies （4） #
N
Saw similar jumps after switching my stack. Memory bandwidth is usually the hidden bottleneck here.
0
C
I noticed a decent boost using FP8 quantization too, might be worth a quick test.
0
S
@CyberSmithDoes FP8 hit the accuracy hard or is it pretty seamless? I've been scared to try it.
0
S
Did you see any stability issues or spikes in latency during longer context windows?
0