489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding

A native tensor runtime. No Python. One .so.

On a desktop RTX 3050, SonicBoom edits code with Qwen2.5-7B at 490 tok/s, 12.7× its own plain decode. llama.cpp's fastest speculative mode reaches 448 tok/s, and its output then differs from its own non-speculative output on up to half of the turns. SonicBoom's output is identical, bit for bit, with speculation on or off.

SonicBoom is written from the ground up in C++ and CUDA, exposed through a stable C ABI, with GNU Guile as its first-class language. No Python interpreter, pip or wheel tree is involved in inference.

Qwen2.5-7B-Instruct Q4_K_M on an RTX 3050 8 GB

On coding chat, speculation makes SonicBoom 4.4× faster than its own plain decode (176.8 tok/s), with the same output.

Both runtimes run the same GGUF on the same machine: RTX 3050 8 GB (sm_86), Ryzen 5 5500, llama.cpp 43fe9c64281 with -ngl 99 -fa on. Each figure is the median of several process runs. Prefill is wall-clock time for the whole prompt.

¹ The code-editing workload is ten turns, each asking for an edit of a 100–300-line source file; generation is greedy, as long as the file. llama.cpp runs llama-server -ngl 99 -fa on -np 1 --spec-type ngram-mod, its fastest of the eight speculative modes on this workload. In the same test, llama.cpp's non-speculative output also differed between repeated runs on 5 of 10 turns.

Model files, checksums, command lines and the long-context tiers (up to 32K) are in docs/benchmarks.md. Models under 2B parameters are no longer benchmarked, because they have little practical use. Gemma 4 E4B, the DeepSeek-R1-Distill-Qwen family and the other Qwen2.5 sizes run as well.

• Same prompt, same output. This holds with the same version, the same GPU and the same warm-up table.
• Speculation doesn't change the result. Turning speculative decoding on or off gives bit-identical token sequences. It is on by default.
• Seeded sampling. The same seed and configuration give the same output.

The warm-up table is measured on your own GPU when a model loads. You can export it and import it elsewhere with sb-tune, which carries a run to another machine with the same GPU. Batched execution is not covered yet.

• Speculative decoding is on by default and verifies up to 64 drafted tokens per step; a long-match drafter targets code and document edits.
• Grammar- and JSON-constrained decoding computes only the LM-head rows the grammar allows.
• Warm-up tables can be exported and imported (sb-tune ), and a canonical numerics mode is available (SONICBOOM_TUNE=canonical ).

Requirements: Linux x86-64, g++-13, CMake, Ninja, CUDA 12.0+, an NVIDIA GPU (tested on sm_86), GNU Guile 3.0 with guile-json 4.7.3, and SonicCross v0.0.2 at build time. Full details are in docs/build.md.

git clone --recursive https://github.com/hardenedlinux/SonicBoom
cd SonicBoom
tools/mlir/build-mlir.sh                 # one-time: pinned LLVM/MLIR
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSONICBOOM_USE_CUDA=ON
cmake --build build

mkdir -p models
curl -L -o models/qwen2.5-7b-instruct-1m-q4_k_m.gguf \
  https://huggingface.co/ngxson/Qwen2.5-7B-Instruct-1M-Q4_K_M-GGUF/resolve/main/qwen2.5-7b-instruct-1m-q4_k_m.gguf
LD_LIBRARY_PATH=build/core guile -L bindings/guile/scheme
(use-modules (sonicboom inference))

(define model   (load-model "models/qwen2.5-7b-instruct-1m-q4_k_m.gguf"))
(define session (make-session model 4096))
(display (generate model session "The three laws of thermodynamics are" 64))

The same calls are available from C (sb_llm_* in capi.h).

The server is a GNU Artanis app; its dependencies are listed in docs/serve-openai.md.

LD_LIBRARY_PATH=build/core guile -L bindings/guile/scheme \
  bindings/guile/scheme/sonicboom/serve/app.scm \
  --model models/qwen2.5-7b-instruct-1m-q4_k_m.gguf --model-id qwen2.5-7b \
  --context-length 16384 --max-sessions 1 --port 8000
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model": "qwen2.5-7b", "stream": true,
       "messages": [{"role": "user", "content": "Write the opening paragraph of a mystery novel."}]}'

Any OpenAI client, or a writing tool that speaks the OpenAI API, can use http://127.0.0.1:8000/v1 as its base URL; no API key is checked. A conversation reuses its shared prefix automatically, so asking for a revision of a long chapter only prefills the new instruction. --max-sessions 1 keeps one KV cache on an 8 GB card; raise it on larger GPUs to serve several conversations at once.

LLAMA_BENCH=/path/to/llama-bench tools/bench.sh models/your-model.gguf --runs 3 --long

This prints the decode and prefill rows for your GPU. The code-editing workload and its llama-server command line are in docs/benchmarks.md. Post your run in the v0.0.5 results issue; earlier runs are in issues #1, #2 and #3.

Community runs keep these numbers honest beyond one card. Thanks to:

Results:

• Runtime. Each decode step is one CUDA Graph replay, with an on-device KV cache and exact prefix reuse: a continued prompt gives the same tokens as recomputing it from scratch.
• Kernels. SonicBoom's own attention kernels (boom_fattn for prefill,boom_fdec for decode and verification) and ggml-derived quantized matmuls. The ggml code is MIT-attributed inNOTICE and is not linked as a dependency.
• Compiler. An MLIR dialect, Boom IR, that lowers fused glue kernels straight to PTX. Every pass has an on/off switch.
• Beyond LLMs. MOSS-Image text-to-image through the same C ABI, and a research route that traces Scheme models into graphs, with autograd and AdamW.

Architecture, scope and roadmap: docs/architecture.md, docs/scope.md, docs/roadmap.md.

• Human architectured
• Agent: Claude Code
• Worker: Deepseek-flash-v4.1

SonicBoom is developed by HardenedVault. We offer consulting on private or air-gapped LLM inference, performance tuning on your hardware, and hardened, auditable inference stacks. Contact consulting@hardenedvault.com.

GPL-3.0-or-later (see LICENSE). ggml/llama.cpp-derived code keeps its MIT notice (see NOTICE).