{"slug": "rockchip-rk3588-for-local-llms-the-150-npu-board-orange-pi-5-radxa-rock-5b", "title": "Rockchip RK3588 for Local LLMs: The $150 NPU Board (Orange Pi 5, Radxa Rock 5B)", "summary": "Rockchip's RK3588 single-board computers, starting at $75, can run local large language models but are limited by memory bandwidth rather than the advertised 6-TOPS NPU. Decode speeds range from 41.6 tokens per second for a 0.5B model to 5 tokens per second for a 6B model, with larger models becoming impractical for interactive use.", "body_md": "Every RK3588 board is sold with the same sticker: a \"6 TOPS NPU\" and a promise of on-device AI. The boards are genuinely cheap, an [Orange Pi 5](https://www.amazon.com/s?k=Orange+Pi+5&tag=57eqvt-20&ref=vettedconsumer.com) starts around $75, and they draw a few watts. So the obvious question for a local-AI builder: can a $100 single-board computer actually run a language model? The answer is yes, with a hard ceiling that has nothing to do with that 6-TOPS number and everything to do with memory.\n\nWe have not benchmarked these boards ourselves. What follows synthesizes Rockchip's own RKLLM benchmarks, owner reports, and the memory-bandwidth math, all linked at the end.\n\n## What the RK3588 is\n\nThe RK3588 is Rockchip's flagship SoC: eight CPU cores (four Cortex-A76 plus four A55), a Mali-G610 GPU, and a 6-TOPS INT8 NPU, paired with up to 32GB of LPDDR4x or LPDDR5 memory. It shows up in a family of boards, the [Orange Pi 5 series](https://www.amazon.com/s?k=Orange+Pi+5+Plus&tag=57eqvt-20&ref=vettedconsumer.com), the [Radxa Rock 5B](https://www.amazon.com/s?k=Radxa+Rock+5B&tag=57eqvt-20&ref=vettedconsumer.com), and the Khadas Edge2, which mostly differ in memory, ports, and build quality, not in AI performance.\n\nThere are three ways to run an LLM on one, and they matter:\n\n**The NPU**, via Rockchip's[RKLLM toolkit](https://github.com/airockchip/rknn-llm?ref=vettedconsumer.com). You convert a model to Rockchip's`.rkllm`\n\nformat (W8A8, 8-bit weights and activations) and it runs on the 6-TOPS NPU, offloading the CPU. This is the \"AI accelerator\" path the marketing points at.**The CPU**, via llama.cpp or Ollama with a normal GGUF (usually Q4_K_M). Simplest to set up, uses the eight Arm cores.** The Mali GPU**, via MLC-LLM over OpenCL. The MLC team famously demoed a 4-bit Llama-2 7B on a $100 Orange Pi 5 this way.\n\n## What it runs: the real numbers\n\nRockchip publishes RKLLM decode speeds for the RK3588's NPU at W8A8. These are the numbers to plan around:\n\n| Model (W8A8, NPU) | Decode speed |\n|---|---|\n| Qwen2 0.5B | 41.6 tok/s |\n| TinyLlama 1.1B | 24.4 tok/s |\n| Qwen2.5 1.5B | 16.7 tok/s |\n| Gemma2 2B | 10.4 tok/s |\n| Phi-3 mini 3.8B | 7.5 tok/s |\n| ChatGLM3 6B | 5.0 tok/s |\n\nRockchip RKLLM benchmarks (airockchip/rknn-llm). NPU path, W8A8.\n\nRead the shape of that table, not just the rows. Speed roughly halves every time the model doubles in size. A sub-2B model is genuinely responsive (15 to 40 tokens per second, faster than most people read). By 6 to 8B you are at 3 to 5 tokens per second, usable for a background task or a patient chat, painful for anything interactive. Owners running 7B on the CPU report the same band, and the 32GB boards will technically load a 14B or even a 32B model at 1 to 4 tokens per second, which is a demo, not a daily driver.\n\n## Why the 6-TOPS NPU isn't the ceiling\n\nHere is the part the spec sheet hides. Generating a token means reading the entire model out of memory once, so the speed limit is set by **memory bandwidth**, not by the NPU's compute. This is the roofline model in action ([Williams, Waterman, Patterson, 2009](https://doi.org/10.1145/1498765.1498785?ref=vettedconsumer.com)), and it is confirmed everywhere from [Microsoft's Splitwise work](https://arxiv.org/abs/2311.18677?ref=vettedconsumer.com) to [Apple's own MLX team](https://machinelearning.apple.com/research/exploring-llms-mlx-m5?ref=vettedconsumer.com): \"generating subsequent tokens is bounded by memory bandwidth, rather than by compute ability.\"\n\nThe RK3588's memory bus is narrow. On paper it tops out around 34 GB/s (a 64-bit LPDDR4x interface); owner measurements on Rock 5-class boards land far lower, around 8.8 to 22 GB/s in practice. Compare that to a used [RTX 3090](https://vettedconsumer.com/the-used-rtx-3090-in-2026-why-a-five-year-old-gpu-is-still-local-ais-best-deal/) at 936 GB/s or a Strix Halo mini PC at 256 GB/s. An 8-bit 7B model is roughly 7GB of weights; dividing your real bandwidth by that gives you your tokens per second, and it lands right where owners measure: a few. The 6-TOPS NPU speeds up prompt processing and frees the CPU, but it cannot move data any faster than the bus allows, so it does not lift the decode ceiling. More on that split in our [prompt processing vs generation](https://vettedconsumer.com/prompt-processing-vs-generation-why-your-box-is-fast-at-one-and-slow-at-the-other/) explainer.\n\n## Which board to buy\n\n| Board | Memory | Best for |\n|---|---|---|\nOrange Pi 5 (base), ~$75+ | up to 16GB LPDDR4x (RK3588S) | Cheapest way in; small models only. |\nOrange Pi 5 Max / Pro | up to 16GB LPDDR5 | A little more bandwidth; good all-rounder. |\nOrange Pi 5 Plus, ~$110+ | up to 32GB LPDDR4x | Full-size RK3588, 32GB to hold bigger models. |\nRadxa Rock 5B+, ~$150+ | up to 32GB LPDDR5 | Best software support + the LPDDR5/32GB combo. The pick if you want one. |\nKhadas Edge2, ~$199+ | up to 16GB LPDDR4x | Fanless, polished, but same speed for more money. |\n\nTwo rules cut through the family. First, decode speed is nearly identical across all of them, they share the SoC, so do not pay more expecting faster tokens. Second, the only spec that changes what you can run is memory: the 32GB LPDDR5 boards (Rock 5B+, Orange Pi 5 Plus/Max) are the ones that can hold a 14B model at all. For most people the [Radxa Rock 5B+](https://www.amazon.com/s?k=Radxa+Rock+5B+Plus+32GB&tag=57eqvt-20&ref=vettedconsumer.com) is the sensible pick for its LPDDR5 memory and the best Debian/Armbian support in the group, which matters a lot when you are setting up RKLLM.\n\n## Who this is for\n\nAn RK3588 board is a great fit for a specific buyer: someone who wants an always-on, low-power, flexible little Linux machine that can also run a small local model, a home-assistant brain, a document classifier, a 1 to 3B chat model on a $100 board sipping a few watts. It is a poor fit for anyone who wants interactive chat with a 7B+ model; for that, a [Jetson Orin Nano](https://vettedconsumer.com/nvidia-jetson-orin-nano-super-the-249-way-into-edge-ai/) (real CUDA and unified memory) or a used GPU is the better answer, and we sort the whole edge field in [which edge chips can run an LLM](https://vettedconsumer.com/which-edge-chips-can-run-an-llm/).\n\nTo see how any of these boards compares to the machine you already own, drop the numbers into our [Can I run it? calculator](https://vettedconsumer.com/can-i-run-it/) and use the [quant picker](https://vettedconsumer.com/quant-picker/) to choose the right file size.\n\n## Sources and how we researched this\n\nWe have not tested these boards first-hand. Decode figures are Rockchip's own [RKLLM benchmarks](https://github.com/airockchip/rknn-llm?ref=vettedconsumer.com) plus owner reports (single-run, single-owner in several cases, treat as directional). The memory-bandwidth ceiling is grounded in the roofline model ([Williams et al., 2009](https://doi.org/10.1145/1498765.1498785?ref=vettedconsumer.com)) and the memory-bound-decode literature ([Patel et al., 2023](https://arxiv.org/abs/2311.18677?ref=vettedconsumer.com); [Apple MLX](https://machinelearning.apple.com/research/exploring-llms-mlx-m5?ref=vettedconsumer.com)). Real RK3588 memory bandwidth is from Radxa community measurements. Board specs and prices are from vendor pages and retailer listings and will vary.\n\n## Related guides\n\n[Which edge chips can run an LLM (and which can't)?](https://vettedconsumer.com/which-edge-chips-can-run-an-llm/), the full field sorted[NVIDIA Jetson Orin Nano](https://vettedconsumer.com/nvidia-jetson-orin-nano-super-the-249-way-into-edge-ai/), the faster edge option[Raspberry Pi 5 (16GB)](https://vettedconsumer.com/raspberry-pi-5-16gb-buyers-guide-a-120-local-ai-and-self-hosting-machine/), the CPU-only comparison[Prompt processing vs generation](https://vettedconsumer.com/prompt-processing-vs-generation-why-your-box-is-fast-at-one-and-slow-at-the-other/), why decode is bandwidth-bound", "url": "https://wpnews.pro/news/rockchip-rk3588-for-local-llms-the-150-npu-board-orange-pi-5-radxa-rock-5b", "canonical_source": "https://vettedconsumer.com/rockchip-rk3588-for-local-llms-orange-pi-5-radxa-rock-5b/", "published_at": "2026-07-15 15:55:04+00:00", "updated_at": "2026-07-15 16:02:46.445478+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure"], "entities": ["Rockchip", "Orange Pi 5", "Radxa Rock 5B", "Khadas Edge2", "RK3588", "RKLLM", "llama.cpp", "MLC-LLM"], "alternates": {"html": "https://wpnews.pro/news/rockchip-rk3588-for-local-llms-the-150-npu-board-orange-pi-5-radxa-rock-5b", "markdown": "https://wpnews.pro/news/rockchip-rk3588-for-local-llms-the-150-npu-board-orange-pi-5-radxa-rock-5b.md", "text": "https://wpnews.pro/news/rockchip-rk3588-for-local-llms-the-150-npu-board-orange-pi-5-radxa-rock-5b.txt", "jsonld": "https://wpnews.pro/news/rockchip-rk3588-for-local-llms-the-150-npu-board-orange-pi-5-radxa-rock-5b.jsonld"}}