{"slug": "axera-ax8850-llm-running-ggufs", "title": "Axera AX8850 LLM running ggufs", "summary": "A custom llama.cpp backend (ggml-axcl) now runs Qwen3-0.6B directly from GGUF files on an Axera AX8850 NPU accelerator card (M5Stack LLM-8850) hosted on a Raspberry Pi 5, achieving 1.3-2.7 tokens per second decode versus the vendor's 13.5-16.9 t/s with baked weights. The developer, woolcoxm, reports 13 of 14 automated tests pass, with the GGUF as the only model artifact and Q8_0 and Q4_K_M quantizations supported. The performance gap is attributed to the NPU idling about 80% of each token due to host-side glue work and per-op scheduler fragmentation.", "body_md": "A custom [llama.cpp](https://github.com/ggml-org/llama.cpp) backend (`ggml-axcl`\n\n) that runs\nQwen3-0.6B **directly from GGUF** on an Axera AX8850 NPU accelerator card\n(M5Stack LLM-8850: 24 TOPS INT8, 8 GB LPDDR4x) hosted on a Raspberry Pi 5.\n\n**The GGUF is the only model artifact** — weights stream from the GGUF into NPU engines\nat load time. Q8_0 and Q4_K_M quants both work from the same code path.\n\n- Current speed: ~1.3-2.7 t/s decode (see \"Architecture\" for why)\n- Vendor reference, same card + model: 13.5-16.9 t/s (baked weights, closed runtime)\n- Code: branch\n`Axera-8850-GGUF-support-PoC-qwen3-0.9b-Q4KM-Q8`\n\non`github.com/woolcoxm/llama.cpp`\n\n```\nLLMTest/\n├── README.md                   this file\n├── NOTES-DYNAMIC-WEIGHTS.md    research log: whole-layer engines (weight layout cracked)\n├── llama.cpp/                  llama.cpp fork with the ggml-axcl backend\n│   └── ggml/src/ggml-axcl/ggml-axcl.cpp   THE backend\n├── gemm/                       NPU engine lab (harnesses, layout research, test scripts)\n│   ├── layout_artifacts/       captured QuantAxModel builds\n│   ├── mk_code_marker.py       code-encoded checkpoints (layout extraction)\n│   └── e2e_test.sh             the E2E test matrix (run on the Pi)\n├── pulsar2/                    Axera compiler toolchain (x86_64)\n├── ax-llm-build/               vendor LLM-builder configs\n├── Qwen3-0.6B/                 HF checkpoint (ground truth for builds)\n└── vendor/                     vendor reference packages\n```\n\nOn the **Pi** (kram@10.0.0.81): `~/build-axcl/`\n\n(build), `~/models/`\n\n(qwen3-q8.gguf,\nqwen3-q4km.gguf), `/usr/local/share/ggml-axcl/`\n\n(compiled NPU engines),\n`/usr/lib/axcl/`\n\n(card runtime, `axcl-smi`\n\nat `/usr/bin/axcl/axcl-smi`\n\n).\n\nEach generated token executes ~120-140 NPU engine calls (7 matmuls × 28 layers + vocab\nhead). Between every call the Pi's CPU does glue work: RMSNorm, RoPE, softmax, masking,\nadds, GLU. Per call: ~0.6 ms NPU exec wrapped in ~2-3 ms of host staging + PCIe DMA +\nscheduler overhead — llama.cpp's scheduler splits the graph into per-op fragments, so\n**the NPU idles ~80% of every token waiting for the host** (visible as ~21% NPU\nutilization with CMM at 7 GB).\n\nThe vendor's engine fuses the *entire layer* into one NPU call (norm, qkv, rope,\nattention with on-card KV cache, FFN, GLU): 28 calls/token, no host round-trips,\n1.5 ms/layer. That's the 5-10× gap. Our path there is staged (see \"Offload roadmap\").\n\nThe weight path is already solved: Pulsar2's `AxQuantizedMatMul`\n\ncustom op accepts\n**int8 weights as runtime tensor inputs**, so GGUF weights are quantized once at load,\nuploaded, and bound per call — no conversion step, 4× less traffic than f32.\n\n13/14 automated tests PASS: prompt sizes 1 → 3000 tokens, unicode, emoji, shell\nmetacharacters, 2000-char word, single char, long generation. Fixed during the pass:\ntwo empty-prompt crashes in llama-simple (`n_batch=0`\n\nassert; zero-token batch → BOS\nfallback). Verified: SIGINT shutdown clean with full CMM release; 5 back-to-back runs\nleak-free; 2 concurrent runs correct; NPU activity confirmed live via axcl-smi.\n\nDevice-resident **chain mode** (`GGML_AXCL_CHAIN=1`\n\n): verified coherent including\n180-token prompts (the old corruption is fixed) — norm/add/glu run as NPU engines on\ndevice-resident activations.\n\nCross-fragment QKV fusion: implemented, engine output verified bit-exact vs CPU\nreference, 3 calls → 1 per layer — but default-off because the scheduler interleaves\nq_norm/rope CPU fragments before the fusion's writeback (corrupts KV cache). Enable\nfor experiments with `GGML_AXCL_QKV_X=1`\n\n; needs op-claiming/scheduler work to be safe.\n\nPi:\n\n```\ncmake -B build-axcl -S llama.cpp -DGGML_AXCL=ON\ncmake --build build-axcl -j4\n~/build-axcl/bin/llama-simple -m ~/models/qwen3-q8.gguf -n 48 \"Your prompt here\"\n```\n\nNOTE: llama-simple takes the prompt as a **positional argument** (not `-p`\n\n), and `-n`\n\nmust come *before* the prompt.\n\nEngines (dev machine, x86): compiled with Pulsar2 from `gemm/`\n\nONNX sources\n(`pulsar2/p7p/.../bin`\n\non PATH), installed to the Pi under\n`/usr/local/share/ggml-axcl/`\n\n.\n\n| Var | Effect |\n|---|---|\n`GGML_AXCL_CHAIN` |\ndevice-resident chain mode (norm/add/glu on NPU) |\n`GGML_AXCL_CHAIN_OPS` |\ngate chain routes (`norm,add,glu` ) |\n`GGML_AXCL_QKV_X` |\ncross-fragment QKV fusion (default off — see above) |\n`GGML_AXCL_QKV_SWAP` |\nswap k/v bindings (diagnostics) |\n`GGML_AXCL_WPOOL_MB` |\ndevice weight pool size (default 2560) |\n`GGML_AXCL_NO_OVERRIDE` |\ndisable activation-source override |\n`GGML_AXCL_NO_FUSION` |\ndisable all fusions |\n`GGML_AXCL_ASYNC` |\nasync engine execute + stream sync |\n`GGML_AXCL_ATTN_MODEL` |\nattention engine path override |\n`GGML_AXCL_DEBUG` |\nverbose engine load/bind logging |\n\n- NPU3-compiled engines measured\n**neutral** vs NPU1 for our shapes (DRAM-bound); the vendor ships NPU1-default builds for this model class. - Multi-core opportunity: VNPU partitioning (\n`axclrtEngineInit`\n\nVNPU kinds) + concurrent engine execution — untapped. - Full card utilization comes from the whole-layer engines, not multi-core compilation.\n\n**Chain mode everywhere**(done, default off): elementwise ops as NPU engines.** Matmul device-pipelining**: bind X from device-resident chain buffers (the`g_chain_x_override`\n\nmechanism) for all projections — removes H2D per call.**Attention engine for all context lengths**: device KV cache with watermark uploads (exists; activation gate needs fixing — currently requires seq > 128).** Whole-layer engines**: vendor-class 28 calls/token. Weight layout fully reverse-engineered and verified (int4 nibble pairs, complete position table —`gemm/layer_layout_v3.pkl`\n\n); remaining: scale-table mapping, weight-patching loader via`axclrtEngineLoadFromMem`\n\n, backend integration. Full log:`NOTES-DYNAMIC-WEIGHTS.md`\n\n.\n\n- Engines fail to load →\n`sudo chmod 777 /tmp/axcl`\n\n(runtime log sink), check driver. - Garbage output → unset\n`GGML_AXCL_CHAIN`\n\n/`GGML_AXCL_QKV_X`\n\nexperiment flags. - Card busy → check\n`axcl-smi`\n\nfor stale processes; CMM baseline is ~18 MiB. - Periodic\n`memory api ... return fail`\n\nlog lines from the card runtime are non-fatal.", "url": "https://wpnews.pro/news/axera-ax8850-llm-running-ggufs", "canonical_source": "https://github.com/woolcoxm/LLMTest", "published_at": "2026-08-26 11:26:44+00:00", "updated_at": "2026-08-26 11:44:54.606416+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["llama.cpp", "Axera AX8850", "M5Stack LLM-8850", "Raspberry Pi 5", "Qwen3-0.6B", "Pulsar2", "woolcoxm", "GGUF"], "alternates": {"html": "https://wpnews.pro/news/axera-ax8850-llm-running-ggufs", "markdown": "https://wpnews.pro/news/axera-ax8850-llm-running-ggufs.md", "text": "https://wpnews.pro/news/axera-ax8850-llm-running-ggufs.txt", "jsonld": "https://wpnews.pro/news/axera-ax8850-llm-running-ggufs.jsonld"}}