# DGX Spark Inference: 90 tok/s on Large Models

> Source: <https://promptcube3.com/en/threads/2213/>
> Published: 2026-07-23 10:03:34+00:00

# DGX Spark Inference: 90 tok/s on Large Models

Hitting 55-90 tok/s on large models without even touching speculative decoding is a massive win for our current deployment. We've been testing a new inference stack specifically for DGX Spark clusters to handle multi-model agentic workflows, and the LlamaBench numbers are actually holding up under simulated traffic.

We also tested a scenario with one endpoint controlling three different model activations. The scheduler batches bursts and coordinates ranks, only swapping the resident model at safe boundaries. The results were:

This is a solid deep dive into how we're scaling our AI workflow internally. We're expecting another 20% bump in performance once we finish the next round of optimization.

The real-world problem we're solving at work is that enterprise agentic apps usually need a mix of specialized models of different sizes. Buying dedicated multi-GPU stacks for every single model in a workflow is way too expensive. This setup lets us run a private, lower-cost stack that handles multiple models across a 2 DGX Spark cluster.

Here is the raw performance data from the LlamaBenchy tests (no quantization, no spec decode):

**DeepSeek V4 Flash C4:** 55.99 decode-system-tok/s**Gemma 4 26B A4B C4:** 63.75 decode-system-tok/s**Nemotron 3 Nano Omni 30B NVFP4 C4:** 90.83 decode-system-tok/s

We also tested a scenario with one endpoint controlling three different model activations. The scheduler batches bursts and coordinates ranks, only swapping the resident model at safe boundaries. The results were:

**DeepSeek V4 Flash:** 49.30 decode-tok/s (16s activation wait)**Gemma 4 26B A4B:** 64.67 decode-tok/s (6s activation wait)**Nemotron 3 Nano Omni 30B NVFP4:** 93.31 decode-tok/s (2s activation wait)

This is a solid deep dive into how we're scaling our AI workflow internally. We're expecting another 20% bump in performance once we finish the next round of optimization.

For those wanting the full technical breakdown, the report is here:`https://woolyai.com/ai-compute-software/dgx-spark-inference-stack/`

[Next Framework Desktop Build: Ryzen AI Max+ Pro 495 & 192GB RAM →](/en/threads/2194/)

## All Replies （4）

N

Saw similar jumps after switching my stack. Memory bandwidth is usually the hidden bottleneck here.

0

C

I noticed a decent boost using FP8 quantization too, might be worth a quick test.

0

S

[@CyberSmith](/en/users/CyberSmith/)Does FP8 hit the accuracy hard or is it pretty seamless? I've been scared to try it.

0

S

Did you see any stability issues or spikes in latency during longer context windows?

0