489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding

SonicBoom, a C++/CUDA tensor runtime from the hardenedlinux project, reached 489.8 tok/s on code editing with Qwen2.5-7B-Instruct Q4_K_M on an RTX 3050 8 GB, 1.09× llama.cpp's fastest speculative mode (448.0 tok/s) and 12.7× its own plain decode. SonicBoom produced bit-identical output with speculative decoding on and off across 10 of 10 turns, while llama.cpp's speculative output differed from its non-speculative output on 5–8 of 10 turns. Without speculation SonicBoom trailed llama.cpp slightly, at 40.6 vs 43.2 tok/s at 512 context and 38.2 vs 41.4 tok/s at 4K context, and prefill was 1557 vs 1584 tok/s at 512 tokens and 1200 vs 1420 tok/s at 8K.

A native tensor runtime. No Python. One .so . On a desktop RTX 3050, SonicBoom edits code with Qwen2.5-7B at 490 tok/s, 12.7× its own plain decode . llama.cpp's fastest speculative mode reaches 448 tok/s, and its output then differs from its own non-speculative output on up to half of the turns. SonicBoom's output is identical, bit for bit, with speculation on or off. SonicBoom is written from the ground up in C++ and CUDA, exposed through a stable C ABI, with GNU Guile as its first-class language. No Python interpreter, pip or wheel tree is involved in inference. Qwen2.5-7B-Instruct Q4 K M on an RTX 3050 8 GB | Workload | SonicBoom | llama.cpp | Ratio | |---|---|---|---| | Code editing, speculative decoding | 489.8 tok/s | 448.0 tok/s ¹ | 1.09× | | Same output with speculation on and off | 10 of 10 turns | 5–8 of 10 turns ¹ | | | Decode, speculation off, context 512 | 40.6 tok/s | 43.2 tok/s | 0.94× | | Decode, speculation off, context 4K | 38.2 tok/s | 41.4 tok/s | 0.92× | | Prefill, 512-token prompt | 1557 tok/s | 1584 tok/s | 0.98× | | Prefill, 8K-token prompt | 1200 tok/s | 1420 tok/s | 0.85× | On coding chat, speculation makes SonicBoom 4.4× faster than its own plain decode 176.8 tok/s , with the same output. Both runtimes run the same GGUF on the same machine: RTX 3050 8 GB sm 86 , Ryzen 5 5500, llama.cpp 43fe9c64281 with -ngl 99 -fa on . Each figure is the median of several process runs. Prefill is wall-clock time for the whole prompt. ¹ The code-editing workload is ten turns, each asking for an edit of a 100–300-line source file; generation is greedy, as long as the file. llama.cpp runs llama-server -ngl 99 -fa on -np 1 --spec-type ngram-mod , its fastest of the eight speculative modes on this workload. In the same test, llama.cpp's non-speculative output also differed between repeated runs on 5 of 10 turns. Model files, checksums, command lines and the long-context tiers up to 32K are in docs/benchmarks.md https://github.com/hardenedlinux/SonicBoom/blob/master/docs/benchmarks.md . Models under 2B parameters are no longer benchmarked, because they have little practical use. Gemma 4 E4B, the DeepSeek-R1-Distill-Qwen family and the other Qwen2.5 sizes run as well. - Same prompt, same output. This holds with the same version, the same GPU and the same warm-up table. - Speculation doesn't change the result. Turning speculative decoding on or off gives bit-identical token sequences. It is on by default. - Seeded sampling. The same seed and configuration give the same output. The warm-up table is measured on your own GPU when a model loads. You can export it and import it elsewhere with sb-tune , which carries a run to another machine with the same GPU. Batched execution is not covered yet. - Speculative decoding is on by default and verifies up to 64 drafted tokens per step; a long-match drafter targets code and document edits. - Grammar- and JSON-constrained decoding computes only the LM-head rows the grammar allows. - Warm-up tables can be exported and imported sb-tune , and a canonical numerics mode is available SONICBOOM TUNE=canonical . Requirements: Linux x86-64, g++-13, CMake, Ninja, CUDA 12.0+, an NVIDIA GPU tested on sm 86 , GNU Guile 3.0 with guile-json https://github.com/aconchillo/guile-json 4.7.3, and SonicCross https://github.com/hardenedlinux/SonicCross v0.0.2 at build time. Full details are in docs/build.md https://github.com/hardenedlinux/SonicBoom/blob/master/docs/build.md . git clone --recursive https://github.com/hardenedlinux/SonicBoom cd SonicBoom tools/mlir/build-mlir.sh one-time: pinned LLVM/MLIR cmake -S . -B build -G Ninja -DCMAKE BUILD TYPE=Release -DSONICBOOM USE CUDA=ON cmake --build build mkdir -p models curl -L -o models/qwen2.5-7b-instruct-1m-q4 k m.gguf \ https://huggingface.co/ngxson/Qwen2.5-7B-Instruct-1M-Q4 K M-GGUF/resolve/main/qwen2.5-7b-instruct-1m-q4 k m.gguf LD LIBRARY PATH=build/core guile -L bindings/guile/scheme use-modules sonicboom inference define model load-model "models/qwen2.5-7b-instruct-1m-q4 k m.gguf" define session make-session model 4096 display generate model session "The three laws of thermodynamics are" 64 The same calls are available from C sb llm in capi.h https://github.com/hardenedlinux/SonicBoom/blob/master/capi/include/sonicboom/capi.h . The server is a GNU Artanis app; its dependencies are listed in docs/serve-openai.md https://github.com/hardenedlinux/SonicBoom/blob/master/docs/serve-openai.md 7-running . LD LIBRARY PATH=build/core guile -L bindings/guile/scheme \ bindings/guile/scheme/sonicboom/serve/app.scm \ --model models/qwen2.5-7b-instruct-1m-q4 k m.gguf --model-id qwen2.5-7b \ --context-length 16384 --max-sessions 1 --port 8000 curl -s http://127.0.0.1:8000/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"model": "qwen2.5-7b", "stream": true, "messages": {"role": "user", "content": "Write the opening paragraph of a mystery novel."} }' Any OpenAI client, or a writing tool that speaks the OpenAI API, can use http://127.0.0.1:8000/v1 as its base URL; no API key is checked. A conversation reuses its shared prefix automatically, so asking for a revision of a long chapter only prefills the new instruction. --max-sessions 1 keeps one KV cache on an 8 GB card; raise it on larger GPUs to serve several conversations at once. LLAMA BENCH=/path/to/llama-bench tools/bench.sh models/your-model.gguf --runs 3 --long This prints the decode and prefill rows for your GPU. The code-editing workload and its llama-server command line are in docs/benchmarks.md https://github.com/hardenedlinux/SonicBoom/blob/master/docs/benchmarks.md code-editing . Post your run in the v0.0.5 results issue https://github.com/hardenedlinux/SonicBoom/blob/master/%7B%7BISSUE URL%7D%7D ; earlier runs are in issues 1 https://github.com/hardenedlinux/SonicBoom/issues/1 , 2 https://github.com/hardenedlinux/SonicBoom/issues/2 and 3 https://github.com/hardenedlinux/SonicBoom/issues/3 . Community runs keep these numbers honest beyond one card. Thanks to: Results: - Runtime. Each decode step is one CUDA Graph replay, with an on-device KV cache and exact prefix reuse: a continued prompt gives the same tokens as recomputing it from scratch. - Kernels. SonicBoom's own attention kernels boom fattn for prefill, boom fdec for decode and verification and ggml-derived quantized matmuls. The ggml code is MIT-attributed in NOTICE https://github.com/hardenedlinux/SonicBoom/blob/master/NOTICE and is not linked as a dependency. - Compiler. An MLIR dialect, Boom IR, that lowers fused glue kernels straight to PTX. Every pass has an on/off switch. - Beyond LLMs. MOSS-Image text-to-image through the same C ABI, and a research route that traces Scheme models into graphs, with autograd and AdamW. Architecture, scope and roadmap: docs/architecture.md https://github.com/hardenedlinux/SonicBoom/blob/master/docs/architecture.md , docs/scope.md https://github.com/hardenedlinux/SonicBoom/blob/master/docs/scope.md , docs/roadmap.md https://github.com/hardenedlinux/SonicBoom/blob/master/docs/roadmap.md . - Human architectured - Agent: Claude Code - Worker: Deepseek-flash-v4.1 SonicBoom is developed by HardenedVault https://hardenedvault.com . We offer consulting on private or air-gapped LLM inference, performance tuning on your hardware, and hardened, auditable inference stacks. Contact consulting@hardenedvault.com mailto:consulting@hardenedvault.com . GPL-3.0-or-later see LICENSE https://github.com/hardenedlinux/SonicBoom/blob/master/LICENSE . ggml/llama.cpp-derived code keeps its MIT notice see NOTICE https://github.com/hardenedlinux/SonicBoom/blob/master/NOTICE .