Axera AX8850 LLM running ggufs

A custom llama.cpp backend (ggml-axcl) now runs Qwen3-0.6B directly from GGUF files on an Axera AX8850 NPU accelerator card (M5Stack LLM-8850) hosted on a Raspberry Pi 5, achieving 1.3-2.7 tokens per second decode versus the vendor's 13.5-16.9 t/s with baked weights. The developer, woolcoxm, reports 13 of 14 automated tests pass, with the GGUF as the only model artifact and Q8_0 and Q4_K_M quantizations supported. The performance gap is attributed to the NPU idling about 80% of each token due to host-side glue work and per-op scheduler fragmentation.

A custom llama.cpp https://github.com/ggml-org/llama.cpp backend ggml-axcl that runs Qwen3-0.6B directly from GGUF on an Axera AX8850 NPU accelerator card M5Stack LLM-8850: 24 TOPS INT8, 8 GB LPDDR4x hosted on a Raspberry Pi 5. The GGUF is the only model artifact — weights stream from the GGUF into NPU engines at load time. Q8 0 and Q4 K M quants both work from the same code path. - Current speed: ~1.3-2.7 t/s decode see "Architecture" for why - Vendor reference, same card + model: 13.5-16.9 t/s baked weights, closed runtime - Code: branch Axera-8850-GGUF-support-PoC-qwen3-0.9b-Q4KM-Q8 on github.com/woolcoxm/llama.cpp LLMTest/ ├── README.md this file ├── NOTES-DYNAMIC-WEIGHTS.md research log: whole-layer engines weight layout cracked ├── llama.cpp/ llama.cpp fork with the ggml-axcl backend │ └── ggml/src/ggml-axcl/ggml-axcl.cpp THE backend ├── gemm/ NPU engine lab harnesses, layout research, test scripts │ ├── layout artifacts/ captured QuantAxModel builds │ ├── mk code marker.py code-encoded checkpoints layout extraction │ └── e2e test.sh the E2E test matrix run on the Pi ├── pulsar2/ Axera compiler toolchain x86 64 ├── ax-llm-build/ vendor LLM-builder configs ├── Qwen3-0.6B/ HF checkpoint ground truth for builds └── vendor/ vendor reference packages On the Pi kram@10.0.0.81 : ~/build-axcl/ build , ~/models/ qwen3-q8.gguf, qwen3-q4km.gguf , /usr/local/share/ggml-axcl/ compiled NPU engines , /usr/lib/axcl/ card runtime, axcl-smi at /usr/bin/axcl/axcl-smi . Each generated token executes ~120-140 NPU engine calls 7 matmuls × 28 layers + vocab head . Between every call the Pi's CPU does glue work: RMSNorm, RoPE, softmax, masking, adds, GLU. Per call: ~0.6 ms NPU exec wrapped in ~2-3 ms of host staging + PCIe DMA + scheduler overhead — llama.cpp's scheduler splits the graph into per-op fragments, so the NPU idles ~80% of every token waiting for the host visible as ~21% NPU utilization with CMM at 7 GB . The vendor's engine fuses the entire layer into one NPU call norm, qkv, rope, attention with on-card KV cache, FFN, GLU : 28 calls/token, no host round-trips, 1.5 ms/layer. That's the 5-10× gap. Our path there is staged see "Offload roadmap" . The weight path is already solved: Pulsar2's AxQuantizedMatMul custom op accepts int8 weights as runtime tensor inputs , so GGUF weights are quantized once at load, uploaded, and bound per call — no conversion step, 4× less traffic than f32. 13/14 automated tests PASS: prompt sizes 1 → 3000 tokens, unicode, emoji, shell metacharacters, 2000-char word, single char, long generation. Fixed during the pass: two empty-prompt crashes in llama-simple n batch=0 assert; zero-token batch → BOS fallback . Verified: SIGINT shutdown clean with full CMM release; 5 back-to-back runs leak-free; 2 concurrent runs correct; NPU activity confirmed live via axcl-smi. Device-resident chain mode GGML AXCL CHAIN=1 : verified coherent including 180-token prompts the old corruption is fixed — norm/add/glu run as NPU engines on device-resident activations. Cross-fragment QKV fusion: implemented, engine output verified bit-exact vs CPU reference, 3 calls → 1 per layer — but default-off because the scheduler interleaves q norm/rope CPU fragments before the fusion's writeback corrupts KV cache . Enable for experiments with GGML AXCL QKV X=1 ; needs op-claiming/scheduler work to be safe. Pi: cmake -B build-axcl -S llama.cpp -DGGML AXCL=ON cmake --build build-axcl -j4 ~/build-axcl/bin/llama-simple -m ~/models/qwen3-q8.gguf -n 48 "Your prompt here" NOTE: llama-simple takes the prompt as a positional argument not -p , and -n must come before the prompt. Engines dev machine, x86 : compiled with Pulsar2 from gemm/ ONNX sources pulsar2/p7p/.../bin on PATH , installed to the Pi under /usr/local/share/ggml-axcl/ . | Var | Effect | |---|---| GGML AXCL CHAIN | device-resident chain mode norm/add/glu on NPU | GGML AXCL CHAIN OPS | gate chain routes norm,add,glu | GGML AXCL QKV X | cross-fragment QKV fusion default off — see above | GGML AXCL QKV SWAP | swap k/v bindings diagnostics | GGML AXCL WPOOL MB | device weight pool size default 2560 | GGML AXCL NO OVERRIDE | disable activation-source override | GGML AXCL NO FUSION | disable all fusions | GGML AXCL ASYNC | async engine execute + stream sync | GGML AXCL ATTN MODEL | attention engine path override | GGML AXCL DEBUG | verbose engine load/bind logging | - NPU3-compiled engines measured neutral vs NPU1 for our shapes DRAM-bound ; the vendor ships NPU1-default builds for this model class. - Multi-core opportunity: VNPU partitioning axclrtEngineInit VNPU kinds + concurrent engine execution — untapped. - Full card utilization comes from the whole-layer engines, not multi-core compilation. Chain mode everywhere done, default off : elementwise ops as NPU engines. Matmul device-pipelining : bind X from device-resident chain buffers the g chain x override mechanism for all projections — removes H2D per call. Attention engine for all context lengths : device KV cache with watermark uploads exists; activation gate needs fixing — currently requires seq 128 . Whole-layer engines : vendor-class 28 calls/token. Weight layout fully reverse-engineered and verified int4 nibble pairs, complete position table — gemm/layer layout v3.pkl ; remaining: scale-table mapping, weight-patching loader via axclrtEngineLoadFromMem , backend integration. Full log: NOTES-DYNAMIC-WEIGHTS.md . - Engines fail to load → sudo chmod 777 /tmp/axcl runtime log sink , check driver. - Garbage output → unset GGML AXCL CHAIN / GGML AXCL QKV X experiment flags. - Card busy → check axcl-smi for stale processes; CMM baseline is ~18 MiB. - Periodic memory api ... return fail log lines from the card runtime are non-fatal.