Rockchip RK3588 for Local LLMs: The $150 NPU Board (Orange Pi 5, Radxa Rock 5B)

wpnews.pro

Every RK3588 board is sold with the same sticker: a "6 TOPS NPU" and a promise of on-device AI. The boards are genuinely cheap, an Orange Pi 5 starts around $75, and they draw a few watts. So the obvious question for a local-AI builder: can a $100 single-board computer actually run a language model? The answer is yes, with a hard ceiling that has nothing to do with that 6-TOPS number and everything to do with memory.

We have not benchmarked these boards ourselves. What follows synthesizes Rockchip's own RKLLM benchmarks, owner reports, and the memory-bandwidth math, all linked at the end.

What the RK3588 is #

The RK3588 is Rockchip's flagship SoC: eight CPU cores (four Cortex-A76 plus four A55), a Mali-G610 GPU, and a 6-TOPS INT8 NPU, paired with up to 32GB of LPDDR4x or LPDDR5 memory. It shows up in a family of boards, the Orange Pi 5 series, the Radxa Rock 5B, and the Khadas Edge2, which mostly differ in memory, ports, and build quality, not in AI performance.

There are three ways to run an LLM on one, and they matter:

The NPU, via Rockchip'sRKLLM toolkit. You convert a model to Rockchip's.rkllm

format (W8A8, 8-bit weights and activations) and it runs on the 6-TOPS NPU, off the CPU. This is the "AI accelerator" path the marketing points at.The CPU, via llama.cpp or Ollama with a normal GGUF (usually Q4_K_M). Simplest to set up, uses the eight Arm cores.** The Mali GPU**, via MLC-LLM over OpenCL. The MLC team famously demoed a 4-bit Llama-2 7B on a $100 Orange Pi 5 this way.

What it runs: the real numbers #

Rockchip publishes RKLLM decode speeds for the RK3588's NPU at W8A8. These are the numbers to plan around:

| Model (W8A8, NPU) | Decode speed |
|---|---|

Rockchip RKLLM benchmarks (airockchip/rknn-llm). NPU path, W8A8.

Read the shape of that table, not just the rows. Speed roughly halves every time the model doubles in size. A sub-2B model is genuinely responsive (15 to 40 tokens per second, faster than most people read). By 6 to 8B you are at 3 to 5 tokens per second, usable for a background task or a patient chat, painful for anything interactive. Owners running 7B on the CPU report the same band, and the 32GB boards will technically load a 14B or even a 32B model at 1 to 4 tokens per second, which is a demo, not a daily driver.

Why the 6-TOPS NPU isn't the ceiling #

Here is the part the spec sheet hides. Generating a token means reading the entire model out of memory once, so the speed limit is set by memory bandwidth, not by the NPU's compute. This is the roofline model in action (Williams, Waterman, Patterson, 2009), and it is confirmed everywhere from Microsoft's Splitwise work to Apple's own MLX team: "generating subsequent tokens is bounded by memory bandwidth, rather than by compute ability."

The RK3588's memory bus is narrow. On paper it tops out around 34 GB/s (a 64-bit LPDDR4x interface); owner measurements on Rock 5-class boards land far lower, around 8.8 to 22 GB/s in practice. Compare that to a used RTX 3090 at 936 GB/s or a Strix Halo mini PC at 256 GB/s. An 8-bit 7B model is roughly 7GB of weights; dividing your real bandwidth by that gives you your tokens per second, and it lands right where owners measure: a few. The 6-TOPS NPU speeds up prompt processing and frees the CPU, but it cannot move data any faster than the bus allows, so it does not lift the decode ceiling. More on that split in our prompt processing vs generation explainer.

Which board to buy #

| Board | Memory | Best for |

|---|---|---|
Orange Pi 5 (base), ~$75+ | up to 16GB LPDDR4x (RK3588S) | Cheapest way in; small models only. |

Two rules cut through the family. First, decode speed is nearly identical across all of them, they share the SoC, so do not pay more expecting faster tokens. Second, the only spec that changes what you can run is memory: the 32GB LPDDR5 boards (Rock 5B+, Orange Pi 5 Plus/Max) are the ones that can hold a 14B model at all. For most people the Radxa Rock 5B+ is the sensible pick for its LPDDR5 memory and the best Debian/Armbian support in the group, which matters a lot when you are setting up RKLLM.

Who this is for #

An RK3588 board is a great fit for a specific buyer: someone who wants an always-on, low-power, flexible little Linux machine that can also run a small local model, a home-assistant brain, a document classifier, a 1 to 3B chat model on a $100 board sipping a few watts. It is a poor fit for anyone who wants interactive chat with a 7B+ model; for that, a Jetson Orin Nano (real CUDA and unified memory) or a used GPU is the better answer, and we sort the whole edge field in which edge chips can run an LLM.

To see how any of these boards compares to the machine you already own, drop the numbers into our Can I run it? calculator and use the quant picker to choose the right file size.

Sources and how we researched this #

We have not tested these boards first-hand. Decode figures are Rockchip's own RKLLM benchmarks plus owner reports (single-run, single-owner in several cases, treat as directional). The memory-bandwidth ceiling is grounded in the roofline model (Williams et al., 2009) and the memory-bound-decode literature (Patel et al., 2023; Apple MLX). Real RK3588 memory bandwidth is from Radxa community measurements. Board specs and prices are from vendor pages and retailer listings and will vary.

Which edge chips can run an LLM (and which can't)?, the full field sortedNVIDIA Jetson Orin Nano, the faster edge optionRaspberry Pi 5 (16GB), the CPU-only comparisonPrompt processing vs generation, why decode is bandwidth-bound

source & further reading

vettedconsumer.com — original article Two Used RTX 3090s vs One RTX 5090 for Local LLMs: 48GB and a 70B, or 32GB and Raw Speed? A $1,000 Local-AI Server With 32GB of VRAM: Inside a Triple-3060 OptiPlex Build RTX 5090 vs RTX 4090 for Local LLMs: What 32GB and 78% More Bandwidth Really Buy