Rockchip RK3588 for Local LLMs: The $150 NPU Board (Orange Pi 5, Radxa Rock 5B)

Rockchip's RK3588 single-board computers, starting at $75, can run local large language models but are limited by memory bandwidth rather than the advertised 6-TOPS NPU. Decode speeds range from 41.6 tokens per second for a 0.5B model to 5 tokens per second for a 6B model, with larger models becoming impractical for interactive use.

Every RK3588 board is sold with the same sticker: a "6 TOPS NPU" and a promise of on-device AI. The boards are genuinely cheap, an Orange Pi 5 https://www.amazon.com/s?k=Orange+Pi+5&tag=57eqvt-20&ref=vettedconsumer.com starts around $75, and they draw a few watts. So the obvious question for a local-AI builder: can a $100 single-board computer actually run a language model? The answer is yes, with a hard ceiling that has nothing to do with that 6-TOPS number and everything to do with memory. We have not benchmarked these boards ourselves. What follows synthesizes Rockchip's own RKLLM benchmarks, owner reports, and the memory-bandwidth math, all linked at the end. What the RK3588 is The RK3588 is Rockchip's flagship SoC: eight CPU cores four Cortex-A76 plus four A55 , a Mali-G610 GPU, and a 6-TOPS INT8 NPU, paired with up to 32GB of LPDDR4x or LPDDR5 memory. It shows up in a family of boards, the Orange Pi 5 series https://www.amazon.com/s?k=Orange+Pi+5+Plus&tag=57eqvt-20&ref=vettedconsumer.com , the Radxa Rock 5B https://www.amazon.com/s?k=Radxa+Rock+5B&tag=57eqvt-20&ref=vettedconsumer.com , and the Khadas Edge2, which mostly differ in memory, ports, and build quality, not in AI performance. There are three ways to run an LLM on one, and they matter: The NPU , via Rockchip's RKLLM toolkit https://github.com/airockchip/rknn-llm?ref=vettedconsumer.com . You convert a model to Rockchip's .rkllm format W8A8, 8-bit weights and activations and it runs on the 6-TOPS NPU, offloading the CPU. This is the "AI accelerator" path the marketing points at. The CPU , via llama.cpp or Ollama with a normal GGUF usually Q4 K M . Simplest to set up, uses the eight Arm cores. The Mali GPU , via MLC-LLM over OpenCL. The MLC team famously demoed a 4-bit Llama-2 7B on a $100 Orange Pi 5 this way. What it runs: the real numbers Rockchip publishes RKLLM decode speeds for the RK3588's NPU at W8A8. These are the numbers to plan around: | Model W8A8, NPU | Decode speed | |---|---| | Qwen2 0.5B | 41.6 tok/s | | TinyLlama 1.1B | 24.4 tok/s | | Qwen2.5 1.5B | 16.7 tok/s | | Gemma2 2B | 10.4 tok/s | | Phi-3 mini 3.8B | 7.5 tok/s | | ChatGLM3 6B | 5.0 tok/s | Rockchip RKLLM benchmarks airockchip/rknn-llm . NPU path, W8A8. Read the shape of that table, not just the rows. Speed roughly halves every time the model doubles in size. A sub-2B model is genuinely responsive 15 to 40 tokens per second, faster than most people read . By 6 to 8B you are at 3 to 5 tokens per second, usable for a background task or a patient chat, painful for anything interactive. Owners running 7B on the CPU report the same band, and the 32GB boards will technically load a 14B or even a 32B model at 1 to 4 tokens per second, which is a demo, not a daily driver. Why the 6-TOPS NPU isn't the ceiling Here is the part the spec sheet hides. Generating a token means reading the entire model out of memory once, so the speed limit is set by memory bandwidth , not by the NPU's compute. This is the roofline model in action Williams, Waterman, Patterson, 2009 https://doi.org/10.1145/1498765.1498785?ref=vettedconsumer.com , and it is confirmed everywhere from Microsoft's Splitwise work https://arxiv.org/abs/2311.18677?ref=vettedconsumer.com to Apple's own MLX team https://machinelearning.apple.com/research/exploring-llms-mlx-m5?ref=vettedconsumer.com : "generating subsequent tokens is bounded by memory bandwidth, rather than by compute ability." The RK3588's memory bus is narrow. On paper it tops out around 34 GB/s a 64-bit LPDDR4x interface ; owner measurements on Rock 5-class boards land far lower, around 8.8 to 22 GB/s in practice. Compare that to a used RTX 3090 https://vettedconsumer.com/the-used-rtx-3090-in-2026-why-a-five-year-old-gpu-is-still-local-ais-best-deal/ at 936 GB/s or a Strix Halo mini PC at 256 GB/s. An 8-bit 7B model is roughly 7GB of weights; dividing your real bandwidth by that gives you your tokens per second, and it lands right where owners measure: a few. The 6-TOPS NPU speeds up prompt processing and frees the CPU, but it cannot move data any faster than the bus allows, so it does not lift the decode ceiling. More on that split in our prompt processing vs generation https://vettedconsumer.com/prompt-processing-vs-generation-why-your-box-is-fast-at-one-and-slow-at-the-other/ explainer. Which board to buy | Board | Memory | Best for | |---|---|---| Orange Pi 5 base , ~$75+ | up to 16GB LPDDR4x RK3588S | Cheapest way in; small models only. | Orange Pi 5 Max / Pro | up to 16GB LPDDR5 | A little more bandwidth; good all-rounder. | Orange Pi 5 Plus, ~$110+ | up to 32GB LPDDR4x | Full-size RK3588, 32GB to hold bigger models. | Radxa Rock 5B+, ~$150+ | up to 32GB LPDDR5 | Best software support + the LPDDR5/32GB combo. The pick if you want one. | Khadas Edge2, ~$199+ | up to 16GB LPDDR4x | Fanless, polished, but same speed for more money. | Two rules cut through the family. First, decode speed is nearly identical across all of them, they share the SoC, so do not pay more expecting faster tokens. Second, the only spec that changes what you can run is memory: the 32GB LPDDR5 boards Rock 5B+, Orange Pi 5 Plus/Max are the ones that can hold a 14B model at all. For most people the Radxa Rock 5B+ https://www.amazon.com/s?k=Radxa+Rock+5B+Plus+32GB&tag=57eqvt-20&ref=vettedconsumer.com is the sensible pick for its LPDDR5 memory and the best Debian/Armbian support in the group, which matters a lot when you are setting up RKLLM. Who this is for An RK3588 board is a great fit for a specific buyer: someone who wants an always-on, low-power, flexible little Linux machine that can also run a small local model, a home-assistant brain, a document classifier, a 1 to 3B chat model on a $100 board sipping a few watts. It is a poor fit for anyone who wants interactive chat with a 7B+ model; for that, a Jetson Orin Nano https://vettedconsumer.com/nvidia-jetson-orin-nano-super-the-249-way-into-edge-ai/ real CUDA and unified memory or a used GPU is the better answer, and we sort the whole edge field in which edge chips can run an LLM https://vettedconsumer.com/which-edge-chips-can-run-an-llm/ . To see how any of these boards compares to the machine you already own, drop the numbers into our Can I run it? calculator https://vettedconsumer.com/can-i-run-it/ and use the quant picker https://vettedconsumer.com/quant-picker/ to choose the right file size. Sources and how we researched this We have not tested these boards first-hand. Decode figures are Rockchip's own RKLLM benchmarks https://github.com/airockchip/rknn-llm?ref=vettedconsumer.com plus owner reports single-run, single-owner in several cases, treat as directional . The memory-bandwidth ceiling is grounded in the roofline model Williams et al., 2009 https://doi.org/10.1145/1498765.1498785?ref=vettedconsumer.com and the memory-bound-decode literature Patel et al., 2023 https://arxiv.org/abs/2311.18677?ref=vettedconsumer.com ; Apple MLX https://machinelearning.apple.com/research/exploring-llms-mlx-m5?ref=vettedconsumer.com . Real RK3588 memory bandwidth is from Radxa community measurements. Board specs and prices are from vendor pages and retailer listings and will vary. Related guides Which edge chips can run an LLM and which can't ? https://vettedconsumer.com/which-edge-chips-can-run-an-llm/ , the full field sorted NVIDIA Jetson Orin Nano https://vettedconsumer.com/nvidia-jetson-orin-nano-super-the-249-way-into-edge-ai/ , the faster edge option Raspberry Pi 5 16GB https://vettedconsumer.com/raspberry-pi-5-16gb-buyers-guide-a-120-local-ai-and-self-hosting-machine/ , the CPU-only comparison Prompt processing vs generation https://vettedconsumer.com/prompt-processing-vs-generation-why-your-box-is-fast-at-one-and-slow-at-the-other/ , why decode is bandwidth-bound