{"slug": "which-edge-chips-can-run-an-llm-and-which-can-t", "title": "Which Edge Chips Can Run an LLM (and Which Can't)?", "summary": "A new analysis of edge AI hardware reveals that TOPS (trillions of operations per second) is a poor predictor of large language model (LLM) performance, with memory capacity and bandwidth being the critical factors. NVIDIA Jetson devices are the only edge hardware that can run LLMs well, while Rockchip, Raspberry Pi, and Hailo-10 can only handle small models. The findings challenge marketing claims and guide buyers toward suitable hardware for local LLM inference.", "body_md": "Walk through the edge-AI aisle and every device advertises the same thing: TOPS. The Raspberry Pi AI Kit, the Google Coral, the Jetson, the Orange Pi, all wave a trillions-of-operations-per-second number at you. So which of them can actually run a language model you can chat with? The surprising answer: TOPS barely predicts it. A $250 Jetson runs an 8B model at a comfortable pace; a 26-TOPS Hailo chip with four times the compute cannot run one at all. Here is the whole edge field sorted by what genuinely runs an LLM, and the three specs that decide it.\n\nWe have not tested all of this hardware first-hand. This synthesizes vendor benchmarks, owner reports, and the inference literature, linked at the end.\n\n## The three things that decide it (TOPS isn't one)\n\nRunning a language model needs three things from hardware, and a vision accelerator's headline TOPS speaks to none of them:\n\n**Enough memory to hold the model.** A 7B model at 4-bit is roughly 4GB; an 8-bit one is ~7GB. The device needs that much RAM the accelerator can reach. This alone rules out the vision TPUs.**Memory bandwidth to read it.** Generating each token means streaming the whole model out of memory once, so decode speed is set by bandwidth, not compute (the roofline model,[Williams et al., 2009](https://doi.org/10.1145/1498765.1498785?ref=vettedconsumer.com); confirmed by[Splitwise](https://arxiv.org/abs/2311.18677?ref=vettedconsumer.com)and[Apple's MLX team](https://machinelearning.apple.com/research/exploring-llms-mlx-m5?ref=vettedconsumer.com): decode is \"bounded by memory bandwidth, rather than by compute ability\").**A runtime that speaks transformer.** The software has to support attention, a growing KV cache, and dynamic shapes. Vision chips run a fixed set of CNN operations and nothing else.\n\nTOPS is a compute number tuned for a completely different job (running convolutions over an image). It is why the sort below cuts across price and marketing.\n\n## Tier 1, runs LLMs well: NVIDIA Jetson\n\nThe Jetson family is the only edge hardware built the way an LLM needs: a CUDA GPU sharing real unified memory, with the most mature edge runtime (llama.cpp, Ollama, and NVIDIA's own containers all just work). Owner and NVIDIA benchmarks:\n\n| Jetson module | Memory | Llama 3.1 8B (4-bit) |\n|---|---|---|\n| Orin Nano 8GB (Super) | 8GB, ~102 GB/s | ~19 tok/s (3B ~43 tok/s) |\n| Orin NX 16GB | 16GB, ~102 GB/s | ~18 tok/s (7B ~20 tok/s) |\n| AGX Orin 64GB | 64GB, ~204 GB/s | ~52 tok/s |\n\nOwner-measured (llama.cpp / MLC, JetPack 6.2) + NVIDIA Jetson AI Lab.\n\nAn [Orin Nano at $249](https://vettedconsumer.com/nvidia-jetson-orin-nano-super-the-249-way-into-edge-ai/) gives you an 8B model at reading speed; the 64GB AGX Orin runs the same model three times faster and can hold much larger ones. This is the edge tier where \"run a real chatbot\" is a straight yes.\n\n## Tier 2, runs small LLMs: Rockchip, Raspberry Pi, and the new Hailo-10\n\nThese can run a model, within limits set by a narrow memory bus. They are fine for small models and background tasks, slow for anything 7B and up.\n\n**Rockchip RK3588 boards**(Orange Pi 5, Radxa Rock 5B). The 6-TOPS NPU via Rockchip's RKLLM runs a 0.5B model at ~40 tok/s and a 2B at ~10, but its ~34 GB/s (real: closer to 10 to 20) memory bus drops a 7B to 3 to 5 tok/s. Cheap, flexible, small-model-friendly. Full breakdown in our[RK3588 guide](https://vettedconsumer.com/rockchip-rk3588-for-local-llms-orange-pi-5-radxa-rock-5b/).**Raspberry Pi 5**(CPU only). No usable AI accelerator for LLMs, so it leans on four Arm cores. A 1B model is responsive; a 3B is low single digits; 7B is a novelty. Its real strength is being a flexible $80 Linux box that*also*does small AI, covered in our[Pi 5 guide](https://vettedconsumer.com/raspberry-pi-5-16gb-buyers-guide-a-120-local-ai-and-self-hosting-machine/).**Hailo-10H**(the new one). Unlike the Hailo-8 in the Pi AI Kit, the Hailo-10 added an external LPDDR interface, so it can finally run small LLMs: Hailo reports a 1.5B model at ~9 tok/s and Llama 3 8B at ~11 tok/s, all at a few watts. It now ships in the Raspberry Pi AI HAT+ 2. Low-power and real, but capped at ~8GB and one model at a time.\n\n## Tier 3, cannot run an LLM: Google Coral and Hailo-8\n\nThis is the tier that trips people up, because these are the chips most often marketed as \"AI accelerators\" and sold in the Raspberry Pi AI Kit. They are excellent at vision (object detection, pose, keyword spotting) and cannot run a language model on the accelerator at all.\n\n**Google Coral Edge TPU**(4 TOPS): runs only INT8 TensorFlow Lite CNN models compiled for it, caches weights in ~8MB of on-chip SRAM, and has no attention or transformer op support. A 1B model is ~1000x too big for that cache.**Hailo-8 / 8L**(26 / 13 TOPS, the Raspberry Pi AI Kit and AI HAT+): a vision dataflow chip with*no external memory at all*, so an LLM's weights have nowhere to live.\n\nThe compute is plenty; the memory architecture is the wall. We walk through exactly why, and why Hailo's own next chip proves the point, in [why your Coral and Hailo TPU can't run an LLM](https://vettedconsumer.com/your-coral-and-hailo-tpu-cant-run-an-llm-heres-why/).\n\n## The one-table decision\n\n| You want to... | Buy |\n|---|---|\n| Chat with a 7 to 8B model on the edge | Jetson Orin Nano ($249) or Orin NX |\n| Run a 1 to 3B model cheaply, low power, always-on | RK3588 board, or a Hailo-10 (AI HAT+ 2) for lowest power |\n| A flexible Linux box that also does small AI | Raspberry Pi 5 |\n| Object detection / vision at the edge | Coral or Hailo-8 (they are great at this) |\n| Run a big model fast | Not the edge, a used GPU or a unified-memory mini PC |\n\nThe through-line: for LLMs, ignore the TOPS on the box and look at memory, how much, how fast, and whether the software can address it. To match a specific model to your hardware, use our [Can I run it? calculator](https://vettedconsumer.com/can-i-run-it/) and the [hardware cheat-sheet](https://vettedconsumer.com/the-local-llm-hardware-cheat-sheet-which-box-runs-which-model/).\n\n## Sources and how we researched this\n\nWe have not tested all of this first-hand. Jetson figures are owner-measured (llama.cpp/MLC, JetPack 6.2) plus NVIDIA's Jetson AI Lab; RK3588 figures are Rockchip's RKLLM benchmarks and owner reports; Hailo-10 figures are Hailo's published numbers; Coral and Hailo-8 limits are from Google's and Hailo's own documentation. The memory-bound-decode framing is the roofline model ([Williams et al., 2009](https://doi.org/10.1145/1498765.1498785?ref=vettedconsumer.com)) plus [Splitwise (Patel et al., 2023)](https://arxiv.org/abs/2311.18677?ref=vettedconsumer.com) and [Apple's MLX write-up](https://machinelearning.apple.com/research/exploring-llms-mlx-m5?ref=vettedconsumer.com). Owner numbers are single measurements that vary by model, quant, and software version.", "url": "https://wpnews.pro/news/which-edge-chips-can-run-an-llm-and-which-can-t", "canonical_source": "https://vettedconsumer.com/which-edge-chips-can-run-an-llm/", "published_at": "2026-07-06 13:36:10+00:00", "updated_at": "2026-07-07 00:44:46.209452+00:00", "lang": "en", "topics": ["large-language-models"], "entities": ["NVIDIA Jetson", "Raspberry Pi", "Hailo", "Rockchip", "Orange Pi", "Google Coral", "llama.cpp", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/which-edge-chips-can-run-an-llm-and-which-can-t", "markdown": "https://wpnews.pro/news/which-edge-chips-can-run-an-llm-and-which-can-t.md", "text": "https://wpnews.pro/news/which-edge-chips-can-run-an-llm-and-which-can-t.txt", "jsonld": "https://wpnews.pro/news/which-edge-chips-can-run-an-llm-and-which-can-t.jsonld"}}