# 489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding > Source: > Published: 2026-11-09 12:39:28+00:00 **A native tensor runtime. No Python. One `.so`.** On a desktop RTX 3050, SonicBoom edits code with Qwen2.5-7B at **490 tok/s, 12.7× its own plain decode**. llama.cpp's fastest speculative mode reaches 448 tok/s, and its output then differs from its own non-speculative output on up to half of the turns. SonicBoom's output is identical, bit for bit, with speculation on or off. SonicBoom is written from the ground up in C++ and CUDA, exposed through a stable C ABI, with GNU Guile as its first-class language. No Python interpreter, pip or wheel tree is involved in inference. **Qwen2.5-7B-Instruct Q4_K_M** on an RTX 3050 8 GB | Workload | SonicBoom | llama.cpp | Ratio | |---|---|---|---| | **Code editing, speculative decoding** | **489.8 tok/s** | 448.0 tok/s ¹ | **1.09×** | | **Same output with speculation on and off** | **10 of 10 turns** | 5–8 of 10 turns ¹ | | | Decode, speculation off, context 512 | 40.6 tok/s | 43.2 tok/s | 0.94× | | Decode, speculation off, context 4K | 38.2 tok/s | 41.4 tok/s | 0.92× | | Prefill, 512-token prompt | 1557 tok/s | 1584 tok/s | 0.98× | | Prefill, 8K-token prompt | 1200 tok/s | 1420 tok/s | 0.85× | On coding chat, speculation makes SonicBoom 4.4× faster than its own plain decode (176.8 tok/s), with the same output. Both runtimes run the same GGUF on the same machine: RTX 3050 8 GB (sm_86), Ryzen 5 5500, llama.cpp `43fe9c64281` with `-ngl 99 -fa on`. Each figure is the median of several process runs. Prefill is wall-clock time for the whole prompt. ¹ The code-editing workload is ten turns, each asking for an edit of a 100–300-line source file; generation is greedy, as long as the file. llama.cpp runs `llama-server -ngl 99 -fa on -np 1 --spec-type ngram-mod`, its fastest of the eight speculative modes on this workload. In the same test, llama.cpp's non-speculative output also differed between repeated runs on 5 of 10 turns. Model files, checksums, command lines and the long-context tiers (up to 32K) are in [docs/benchmarks.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/benchmarks.md). Models under 2B parameters are no longer benchmarked, because they have little practical use. Gemma 4 E4B, the DeepSeek-R1-Distill-Qwen family and the other Qwen2.5 sizes run as well. - **Same prompt, same output.** This holds with the same version, the same GPU and the same warm-up table. - **Speculation doesn't change the result.** Turning speculative decoding on or off gives bit-identical token sequences. It is on by default. - **Seeded sampling.** The same seed and configuration give the same output. The warm-up table is measured on your own GPU when a model loads. You can export it and import it elsewhere with `sb-tune`, which carries a run to another machine with the same GPU. Batched execution is not covered yet. - Speculative decoding is on by default and verifies up to 64 drafted tokens per step; a long-match drafter targets code and document edits. - Grammar- and JSON-constrained decoding computes only the LM-head rows the grammar allows. - Warm-up tables can be exported and imported (`sb-tune` ), and a canonical numerics mode is available (`SONICBOOM_TUNE=canonical` ). Requirements: Linux x86-64, g++-13, CMake, Ninja, CUDA 12.0+, an NVIDIA GPU (tested on sm_86), GNU Guile 3.0 with [guile-json](https://github.com/aconchillo/guile-json) 4.7.3, and [SonicCross](https://github.com/hardenedlinux/SonicCross) v0.0.2 at build time. Full details are in [docs/build.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/build.md). ``` git clone --recursive https://github.com/hardenedlinux/SonicBoom cd SonicBoom tools/mlir/build-mlir.sh # one-time: pinned LLVM/MLIR cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSONICBOOM_USE_CUDA=ON cmake --build build mkdir -p models curl -L -o models/qwen2.5-7b-instruct-1m-q4_k_m.gguf \ https://huggingface.co/ngxson/Qwen2.5-7B-Instruct-1M-Q4_K_M-GGUF/resolve/main/qwen2.5-7b-instruct-1m-q4_k_m.gguf LD_LIBRARY_PATH=build/core guile -L bindings/guile/scheme (use-modules (sonicboom inference)) (define model (load-model "models/qwen2.5-7b-instruct-1m-q4_k_m.gguf")) (define session (make-session model 4096)) (display (generate model session "The three laws of thermodynamics are" 64)) ``` The same calls are available from C (`sb_llm_*` in [`capi.h`](https://github.com/hardenedlinux/SonicBoom/blob/master/capi/include/sonicboom/capi.h)). The server is a GNU Artanis app; its dependencies are listed in [docs/serve-openai.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/serve-openai.md#7-running). ``` LD_LIBRARY_PATH=build/core guile -L bindings/guile/scheme \ bindings/guile/scheme/sonicboom/serve/app.scm \ --model models/qwen2.5-7b-instruct-1m-q4_k_m.gguf --model-id qwen2.5-7b \ --context-length 16384 --max-sessions 1 --port 8000 curl -s http://127.0.0.1:8000/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"model": "qwen2.5-7b", "stream": true, "messages": [{"role": "user", "content": "Write the opening paragraph of a mystery novel."}]}' ``` Any OpenAI client, or a writing tool that speaks the OpenAI API, can use `http://127.0.0.1:8000/v1` as its base URL; no API key is checked. A conversation reuses its shared prefix automatically, so asking for a revision of a long chapter only prefills the new instruction. `--max-sessions 1` keeps one KV cache on an 8 GB card; raise it on larger GPUs to serve several conversations at once. ``` LLAMA_BENCH=/path/to/llama-bench tools/bench.sh models/your-model.gguf --runs 3 --long ``` This prints the decode and prefill rows for your GPU. The code-editing workload and its llama-server command line are in [docs/benchmarks.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/benchmarks.md#code-editing). Post your run in the [v0.0.5 results issue](https://github.com/hardenedlinux/SonicBoom/blob/master/%7B%7BISSUE_URL%7D%7D); earlier runs are in issues [#1](https://github.com/hardenedlinux/SonicBoom/issues/1), [#2](https://github.com/hardenedlinux/SonicBoom/issues/2) and [#3](https://github.com/hardenedlinux/SonicBoom/issues/3). Community runs keep these numbers honest beyond one card. Thanks to: Results: - **Runtime.** Each decode step is one CUDA Graph replay, with an on-device KV cache and exact prefix reuse: a continued prompt gives the same tokens as recomputing it from scratch. - **Kernels.** SonicBoom's own attention kernels (`boom_fattn` for prefill,`boom_fdec` for decode and verification) and ggml-derived quantized matmuls. The ggml code is MIT-attributed in[NOTICE](https://github.com/hardenedlinux/SonicBoom/blob/master/NOTICE) and is not linked as a dependency. - **Compiler.** An MLIR dialect, Boom IR, that lowers fused glue kernels straight to PTX. Every pass has an on/off switch. - **Beyond LLMs.** MOSS-Image text-to-image through the same C ABI, and a research route that traces Scheme models into graphs, with autograd and AdamW. Architecture, scope and roadmap: [docs/architecture.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/architecture.md), [docs/scope.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/scope.md), [docs/roadmap.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/roadmap.md). - Human architectured - Agent: Claude Code - Worker: Deepseek-flash-v4.1 SonicBoom is developed by [HardenedVault](https://hardenedvault.com). We offer consulting on private or air-gapped LLM inference, performance tuning on your hardware, and hardened, auditable inference stacks. Contact [consulting@hardenedvault.com](mailto:consulting@hardenedvault.com). GPL-3.0-or-later (see [`LICENSE`](https://github.com/hardenedlinux/SonicBoom/blob/master/LICENSE)). ggml/llama.cpp-derived code keeps its MIT notice (see [`NOTICE`](https://github.com/hardenedlinux/SonicBoom/blob/master/NOTICE)).