{"slug": "489-8-tok-s-qwen2-5-7b-on-rtx3050-for-code-edit-reproducible-decoding", "title": "489.8 tok/s, QWen2.5 7B on RTX3050 for code edit, reproducible decoding", "summary": "SonicBoom, a C++/CUDA tensor runtime from the hardenedlinux project, reached 489.8 tok/s on code editing with Qwen2.5-7B-Instruct Q4_K_M on an RTX 3050 8 GB, 1.09× llama.cpp's fastest speculative mode (448.0 tok/s) and 12.7× its own plain decode. SonicBoom produced bit-identical output with speculative decoding on and off across 10 of 10 turns, while llama.cpp's speculative output differed from its non-speculative output on 5–8 of 10 turns. Without speculation SonicBoom trailed llama.cpp slightly, at 40.6 vs 43.2 tok/s at 512 context and 38.2 vs 41.4 tok/s at 4K context, and prefill was 1557 vs 1584 tok/s at 512 tokens and 1200 vs 1420 tok/s at 8K.", "body_md": "**A native tensor runtime. No Python. One `.so`.**\n\nOn a desktop RTX 3050, SonicBoom edits code with Qwen2.5-7B at\n**490 tok/s, 12.7× its own plain decode**. llama.cpp's fastest speculative\nmode reaches 448 tok/s, and its output then differs from its own\nnon-speculative output on up to half of the turns. SonicBoom's output is\nidentical, bit for bit, with speculation on or off.\n\nSonicBoom is written from the ground up in C++ and CUDA, exposed through a stable C ABI, with GNU Guile as its first-class language. No Python interpreter, pip or wheel tree is involved in inference.\n\n**Qwen2.5-7B-Instruct Q4_K_M** on an RTX 3050 8 GB\n\n| Workload | SonicBoom | llama.cpp | Ratio | \n|---|---|---|---|\n| **Code editing, speculative decoding** | **489.8 tok/s** | 448.0 tok/s ¹ | **1.09×** | \n| **Same output with speculation on and off** | **10 of 10 turns** | 5–8 of 10 turns ¹ |  | \n| Decode, speculation off, context 512 | 40.6 tok/s | 43.2 tok/s | 0.94× | \n| Decode, speculation off, context 4K | 38.2 tok/s | 41.4 tok/s | 0.92× | \n| Prefill, 512-token prompt | 1557 tok/s | 1584 tok/s | 0.98× | \n| Prefill, 8K-token prompt | 1200 tok/s | 1420 tok/s | 0.85× | \n\nOn coding chat, speculation makes SonicBoom 4.4× faster than its own plain decode (176.8 tok/s), with the same output.\n\nBoth runtimes run the same GGUF on the same machine: RTX 3050 8 GB (sm_86),\nRyzen 5 5500, llama.cpp `43fe9c64281` with `-ngl 99 -fa on`. Each figure is the\nmedian of several process runs. Prefill is wall-clock time for the whole\nprompt.\n\n¹ The code-editing workload is ten turns, each asking for an edit of a\n100–300-line source file; generation is greedy, as long as the file.\nllama.cpp runs `llama-server -ngl 99 -fa on -np 1 --spec-type ngram-mod`, its\nfastest of the eight speculative modes on this workload. In the same test,\nllama.cpp's non-speculative output also differed between repeated runs on 5 of\n10 turns.\n\nModel files, checksums, command lines and the long-context tiers (up to 32K)\nare in [docs/benchmarks.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/benchmarks.md). Models under 2B parameters are\nno longer benchmarked, because they have little practical use. Gemma 4 E4B,\nthe DeepSeek-R1-Distill-Qwen family and the other Qwen2.5 sizes run as well.\n\n- **Same prompt, same output.** This holds with the same version, the same GPU\nand the same warm-up table.\n- **Speculation doesn't change the result.** Turning speculative decoding on\nor off gives bit-identical token sequences. It is on by default.\n- **Seeded sampling.** The same seed and configuration give the same output.\n\nThe warm-up table is measured on your own GPU when a model loads. You can\nexport it and import it elsewhere with `sb-tune`, which carries a run to\nanother machine with the same GPU. Batched execution is not covered yet.\n\n- Speculative decoding is on by default and verifies up to 64 drafted tokens per step; a long-match drafter targets code and document edits.\n- Grammar- and JSON-constrained decoding computes only the LM-head rows the grammar allows.\n- Warm-up tables can be exported and imported (`sb-tune` ), and a canonical\nnumerics mode is available (`SONICBOOM_TUNE=canonical` ).\n\nRequirements: Linux x86-64, g++-13, CMake, Ninja, CUDA 12.0+, an NVIDIA GPU\n(tested on sm_86), GNU Guile 3.0 with\n[guile-json](https://github.com/aconchillo/guile-json) 4.7.3, and\n[SonicCross](https://github.com/hardenedlinux/SonicCross) v0.0.2 at build\ntime. Full details are in [docs/build.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/build.md).\n\n```\ngit clone --recursive https://github.com/hardenedlinux/SonicBoom\ncd SonicBoom\ntools/mlir/build-mlir.sh                 # one-time: pinned LLVM/MLIR\ncmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSONICBOOM_USE_CUDA=ON\ncmake --build build\n\nmkdir -p models\ncurl -L -o models/qwen2.5-7b-instruct-1m-q4_k_m.gguf \\\n  https://huggingface.co/ngxson/Qwen2.5-7B-Instruct-1M-Q4_K_M-GGUF/resolve/main/qwen2.5-7b-instruct-1m-q4_k_m.gguf\nLD_LIBRARY_PATH=build/core guile -L bindings/guile/scheme\n(use-modules (sonicboom inference))\n\n(define model   (load-model \"models/qwen2.5-7b-instruct-1m-q4_k_m.gguf\"))\n(define session (make-session model 4096))\n(display (generate model session \"The three laws of thermodynamics are\" 64))\n```\n\nThe same calls are available from C (`sb_llm_*` in\n[`capi.h`](https://github.com/hardenedlinux/SonicBoom/blob/master/capi/include/sonicboom/capi.h)).\n\nThe server is a GNU Artanis app; its dependencies are listed in\n[docs/serve-openai.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/serve-openai.md#7-running).\n\n```\nLD_LIBRARY_PATH=build/core guile -L bindings/guile/scheme \\\n  bindings/guile/scheme/sonicboom/serve/app.scm \\\n  --model models/qwen2.5-7b-instruct-1m-q4_k_m.gguf --model-id qwen2.5-7b \\\n  --context-length 16384 --max-sessions 1 --port 8000\ncurl -s http://127.0.0.1:8000/v1/chat/completions \\\n  -H 'content-type: application/json' \\\n  -d '{\"model\": \"qwen2.5-7b\", \"stream\": true,\n       \"messages\": [{\"role\": \"user\", \"content\": \"Write the opening paragraph of a mystery novel.\"}]}'\n```\n\nAny OpenAI client, or a writing tool that speaks the OpenAI API, can use\n`http://127.0.0.1:8000/v1` as its base URL; no API key is checked. A\nconversation reuses its shared prefix automatically, so asking for a revision\nof a long chapter only prefills the new instruction. `--max-sessions 1` keeps\none KV cache on an 8 GB card; raise it on larger GPUs to serve several\nconversations at once.\n\n```\nLLAMA_BENCH=/path/to/llama-bench tools/bench.sh models/your-model.gguf --runs 3 --long\n```\n\nThis prints the decode and prefill rows for your GPU. The code-editing\nworkload and its llama-server command line are in\n[docs/benchmarks.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/benchmarks.md#code-editing). Post your run in the\n[v0.0.5 results issue](https://github.com/hardenedlinux/SonicBoom/blob/master/%7B%7BISSUE_URL%7D%7D); earlier runs are in issues\n[#1](https://github.com/hardenedlinux/SonicBoom/issues/1),\n[#2](https://github.com/hardenedlinux/SonicBoom/issues/2) and\n[#3](https://github.com/hardenedlinux/SonicBoom/issues/3).\n\nCommunity runs keep these numbers honest beyond one card. Thanks to:\n\nResults:\n\n- **Runtime.** Each decode step is one CUDA Graph replay, with an on-device KV\ncache and exact prefix reuse: a continued prompt gives the same tokens as\nrecomputing it from scratch.\n- **Kernels.** SonicBoom's own attention kernels (`boom_fattn` for prefill,`boom_fdec` for decode and verification) and ggml-derived quantized\nmatmuls. The ggml code is MIT-attributed in[NOTICE](https://github.com/hardenedlinux/SonicBoom/blob/master/NOTICE) and is not\nlinked as a dependency.\n- **Compiler.** An MLIR dialect, Boom IR, that lowers fused glue kernels\nstraight to PTX. Every pass has an on/off switch.\n- **Beyond LLMs.** MOSS-Image text-to-image through the same C ABI, and a\nresearch route that traces Scheme models into graphs, with autograd and\nAdamW.\n\nArchitecture, scope and roadmap: [docs/architecture.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/architecture.md),\n[docs/scope.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/scope.md), [docs/roadmap.md](https://github.com/hardenedlinux/SonicBoom/blob/master/docs/roadmap.md).\n\n- Human architectured\n- Agent: Claude Code\n- Worker: Deepseek-flash-v4.1\n\nSonicBoom is developed by [HardenedVault](https://hardenedvault.com). We\noffer consulting on private or air-gapped LLM inference, performance tuning on\nyour hardware, and hardened, auditable inference stacks. Contact\n[consulting@hardenedvault.com](mailto:consulting@hardenedvault.com).\n\nGPL-3.0-or-later (see [`LICENSE`](https://github.com/hardenedlinux/SonicBoom/blob/master/LICENSE)). ggml/llama.cpp-derived code\nkeeps its MIT notice (see [`NOTICE`](https://github.com/hardenedlinux/SonicBoom/blob/master/NOTICE)).", "url": "https://wpnews.pro/news/489-8-tok-s-qwen2-5-7b-on-rtx3050-for-code-edit-reproducible-decoding", "canonical_source": "https://github.com/hardenedlinux/SonicBoom", "published_at": "2026-11-09 12:39:28+00:00", "updated_at": "2026-11-09 12:53:17.698402+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["SonicBoom", "hardenedlinux", "Qwen2.5-7B-Instruct", "llama.cpp", "RTX 3050", "NVIDIA", "GNU Guile", "SonicCross"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/489-8-tok-s-qwen2-5-7b-on-rtx3050-for-code-edit-reproducible-decoding", "markdown": "https://wpnews.pro/news/489-8-tok-s-qwen2-5-7b-on-rtx3050-for-code-edit-reproducible-decoding.md", "text": "https://wpnews.pro/news/489-8-tok-s-qwen2-5-7b-on-rtx3050-for-code-edit-reproducible-decoding.txt", "jsonld": "https://wpnews.pro/news/489-8-tok-s-qwen2-5-7b-on-rtx3050-for-code-edit-reproducible-decoding.jsonld"}}