Update: same files, same RTX 4080, a day later
Two changes since the post. Both are lossless: every output logit is bit-identical to before.
| Qwen3-30B-A3B BF16 (.shn5), RTX 4080 16 GB | VRAM | Decode | 700-token prompt |
|—|—|—|—|
| Leviathan, 16 expert slots a layer (in the post) | 10.3 GiB | 11.2 tok/s | 35 tok/s |
| Leviathan, 16 expert slots a layer (now) | 10.3 GiB | 14.6 tok/s | 66 tok/s |
| Leviathan, 8 expert slots a layer (in the post) | 6.9 GiB | 10.4 tok/s | 47 tok/s |
| Leviathan, 8 expert slots a layer (now) | 6.9 GiB | 12.9 tok/s | 64 tok/s |
| llama.cpp CUDA, –cpu-moe | not measured | 9.5–10.4 tok/s | 116 tok/s |
The prompt figures are single runs, as before.
What changed:
-
The GPU decoder. Profiling showed it spending about 70 instructions on each weight, mostly waiting on memory. A leaner loop decodes the handful of experts a token needs about 2.5× faster. It produces the same bytes, and every tile is still checked.
-
Prompts fetch each expert once for all the tokens that need it, and decode those experts in one batch instead of one at a time.
-
A damaged file is refused more safely. Before, the GPU decoder could read past a damaged expert’s data and crash. Now it counts the damage and refuses the file.
Prompt processing is still behind llama.cpp (66 vs 116 tok/s), and that’s next.
We also measured speculative decoding over the streamed experts. It would gain about 1.16× at best, and with a real draft model it measured +4% to −15%. Each drafted token pulls its own experts over PCIe, so it stays off for streamed MoE.
These changes are in the engine now and will be in the release.
Built with Claude Code by Michael Ricky Neal and Claude (Anthropic).