.shn5: lossless BF16 model files ~1.5× smaller, and a 61 GB MoE on a 16 GB GPU

A follow-up update to the .shn5 lossless BF16 model format reports that a reworked GPU decoder raised decode speed on Qwen3-30B-A3B BF16 from 11.2 to 14.6 tok/s at 16 expert slots per layer and from 10.4 to 12.9 tok/s at 8 slots, running on a 16 GB RTX 4080 with bit-identical output logits. The update, built by Michael Ricky Neal with Claude (Anthropic), also batches expert fetches so each expert is pulled once per prompt, and now refuses damaged files instead of crashing. Prompt processing remains behind llama.cpp CUDA with --cpu-moe at 66 versus 116 tok/s, which the author names as the next target, and speculative decoding over streamed experts measured +4% to -15% with a real draft model, so it stays off for streamed MoE.

Update: same files, same RTX 4080, a day later Two changes since the post. Both are lossless: every output logit is bit-identical to before. | Qwen3-30B-A3B BF16 .shn5 , RTX 4080 16 GB | VRAM | Decode | 700-token prompt | |—|—|—|—| | Leviathan, 16 expert slots a layer in the post | 10.3 GiB | 11.2 tok/s | 35 tok/s | | Leviathan, 16 expert slots a layer now | 10.3 GiB | 14.6 tok/s | 66 tok/s | | Leviathan, 8 expert slots a layer in the post | 6.9 GiB | 10.4 tok/s | 47 tok/s | | Leviathan, 8 expert slots a layer now | 6.9 GiB | 12.9 tok/s | 64 tok/s | | llama.cpp CUDA, –cpu-moe | not measured | 9.5–10.4 tok/s | 116 tok/s | The prompt figures are single runs, as before. What changed: - The GPU decoder. Profiling showed it spending about 70 instructions on each weight, mostly waiting on memory. A leaner loop decodes the handful of experts a token needs about 2.5× faster. It produces the same bytes, and every tile is still checked. - Prompts fetch each expert once for all the tokens that need it, and decode those experts in one batch instead of one at a time. - A damaged file is refused more safely. Before, the GPU decoder could read past a damaged expert’s data and crash. Now it counts the damage and refuses the file. Prompt processing is still behind llama.cpp 66 vs 116 tok/s , and that’s next. We also measured speculative decoding over the streamed experts. It would gain about 1.16× at best, and with a real draft model it measured +4% to −15%. Each drafted token pulls its own experts over PCIe, so it stays off for streamed MoE. These changes are in the engine now and will be in the release. Built with Claude Code by Michael Ricky Neal and Claude Anthropic .