{"slug": "running-models-on-risc-v-with-iree", "title": "Running models on RISC-V with IREE", "summary": "IREE, an open-source compiler from the IREE project, now compiles machine learning models to native RISC-V CPU code with support for the RISC-V Vector extension (RVV), hand-written microkernels, and data-tiling. The project's blog post demonstrates the full flow—importing models from PyTorch, compiling for RISC-V, running under QEMU, and benchmarking—using torchvision models like MobileNetV2 and ResNet18, and also provides a ready-to-compile Qwen3-0.6B model. This enables efficient AI inference on RISC-V hardware, expanding deployment options for edge and embedded devices.", "body_md": "[CPU](../../tags/#tag:cpu)\n\n[RISC-V](../../tags/#tag:risc-v)\n\n# Running models on RISC-V with IREE[link](#running-models-on-risc-v-with-iree)\n\nIREE compiles machine learning models to native RISC-V CPU code, with support for the RISC-V Vector extension (RVV), hand-written microkernels, and data-tiling. This post walks through the full flow for a model: importing it from PyTorch, compiling it for a RISC-V target, running it, and benchmarking the result.\n\nAll commands below run under `qemu-riscv64`\n\n. The flow on real hardware is\nidentical — the QEMU invocation is simply replaced by running the tools natively\non the target.\n\n## Setup[link](#setup)\n\nIREE is a cross-compiler: the compiler is built on the host, the runtime is\ncross-compiled for the target, and the runtime is then copied to the target (or\nrun under QEMU). The\n[RISC-V cross-compilation guide](https://iree.dev/building-from-source/riscv/)\ncovers this in full. In brief:\n\n`./build_tools/riscv/riscv_bootstrap.sh`\n\ndownloads a prebuilt clang toolchain and QEMU into`~/riscv`\n\n.- Build and install the host compiler, then cross-build the runtime with the\n`build_tools/cmake/linux_riscv64.cmake`\n\ntoolchain file. - Point\n`QEMU_BIN`\n\nat`qemu-riscv64`\n\nand`RISCV_TOOLCHAIN_ROOT`\n\nat the toolchain.\n\nThe result is an `iree-compile`\n\non the host and `iree-run-module`\n\n/\n`iree-benchmark-module`\n\nbuilt for RISC-V.\n\n## Importing a model[link](#importing-a-model)\n\nThis post uses a few PyTorch models as a running example, but IREE supports\nmodels from other frameworks such as LiteRT (TensorFlow Lite) and ONNX just as well,\nonce they have been imported to MLIR. See the\n[ML frameworks guides](https://iree.dev/guides/ml-frameworks/) for the\nper-framework export/import steps — for example\n[PyTorch](https://iree.dev/guides/ml-frameworks/pytorch/),\n[LiteRT / TensorFlow Lite](https://iree.dev/guides/ml-frameworks/tflite/), and\n[ONNX](https://iree.dev/guides/ml-frameworks/onnx/). Also check out the\nIREE community meeting\n[presentation](https://youtu.be/UvH9rVe9_KA?si=Z6aYmsspb1SLs0ik) by Artem\nGindinson from Roofline.\n\nFor PyTorch, [iree-turbine](https://iree.dev/guides/ml-frameworks/pytorch/)'s\n`aot.export`\n\nproduces the MLIR. The following script exports two torchvision\nmodels, saving an input for each to feed later:\n\n``` python\n# export.py\nimport numpy as np, torch, torchvision as tv\nimport iree.turbine.aot as aot\n\ndef dump(name, model, example):\n    aot.export(model.eval(), example).save_mlir(f\"{name}.mlir\")\n    np.save(f\"{name}_input.npy\", example.numpy())\n\n# Vision models (torchvision), NCHW float input.\ndump(\"mobilenet\", tv.models.mobilenet_v2(weights=\"DEFAULT\"), torch.randn(1, 3, 224, 224))\ndump(\"resnet18\",  tv.models.resnet18(weights=\"DEFAULT\"),     torch.randn(1, 3, 224, 224))\n```\n\nThe exported entry point is `@main`\n\n, see the `--function=main`\n\nflag for the\n`iree-*-module`\n\ninvocations below.\n\nAlternatively, you can check the models in the [IREE test suites](https://github.com/iree-org/iree-test-suites/tree/main/torch_models).\nWe have some ready-to-compile `.mlir`\n\nfiles whose weights are kept in a\nseparate `.irpa`\n\n(IREE parameter archive) — which keeps the `.mlir`\n\nsmall and lets you swap weights\nwithout recompiling. For example, [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B):\n\n```\ncurl -L -o qwen3.mlir https://raw.githubusercontent.com/iree-org/iree-test-suites/main/torch_models/qwen3-600m/model.mlir\ncurl -L -o qwen3.irpa https://huggingface.co/roofline/iree-regression-models/resolve/main/qwen3-600m/real_weights.irpa\n```\n\nThe weights are supplied at run time with `--parameters=`\n\n(see below).\n\n## Compiling for RISC-V[link](#compiling-for-risc-v)\n\nThe base command to produce RISC-V vector code is:\n\n```\niree-compile mobilenet.mlir -o mobilenet_rv64.vmfb \\\n  --iree-hal-target-device=local \\\n  --iree-hal-local-target-device-backends=llvm-cpu \\\n  --iree-llvmcpu-target-triple=riscv64 \\\n  --iree-llvmcpu-target-abi=lp64d \\\n  --iree-llvmcpu-target-cpu-features=+m,+a,+f,+d,+c,+zvl512b,+v\n```\n\nThe flag that matters most on RISC-V is ** --iree-llvmcpu-target-cpu-features**,\nwhich specifies the ISA.\n\n`+m,+a,+f,+d,+c`\n\nis `rv64gc`\n\n, `+v`\n\nenables RVV 1.0, and\n`+zvl512b`\n\ndeclares the minimum vector register width (VLEN) — 512 bits here.\nThe `zvl`\n\nvalue should match the target hardware's actual VLEN — 512 for the QEMU\nconfiguration used below, 256 on a device such as a SpaceMiT X60 — since a\nmismatch leaves the vector units underutilized. VLEN is the key RISC-V knob: it\ndrives LLVM's vector codegen *and*the tile sizes IREE selects for data-tiling (more on that below).\n\nThe remaining flags are the optimization knobs. None of them are RISC-V-specific, but they are where the performance comes from, so they are layered on top of the base command.\n\n### Data-tiling[link](#data-tiling)\n\n`--iree-opt-data-tiling`\n\nrepacks matmul-shaped operations into a tiled `mmt4d`\n\nlayout that maps cleanly onto the vector unit. It is off by default; most models,\nespecially matmul-heavy models, benefit from this. On RISC-V the tile shape\ndepends on\nVLEN, so the `+zvl*b`\n\nvalue chosen above also determines the produced layout. The\n[data-tiling walkthrough](https://iree.dev/community/blog/2025-08-25-data-tiling-walkthrough/)\nand [mmt4d blogpost](https://iree.dev/community/blog/2021-10-13-matrix-multiplication-with-mmt4d/)\ncover the mechanism in detail.\n\n### im2col for convolutions[link](#im2col-for-convolutions)\n\n`--iree-global-opt-use-im2col-for-convs=true`\n\nrewrites convolutions as im2col plus\nmatmul, so that convolutions use the same optimized matmul, data-tiling, and\nmicrokernel path as everything else. It is also off by default, and is beneficial\nfor most of the convolution models above. Native data-tiling support for convolutions\nis still work-in-progress.\n\n### Microkernels[link](#microkernels)\n\n`--iree-llvmcpu-enable-ukernels=...`\n\nselects IREE's hand-written microkernels\ninstead of relying solely on the generic vectorizer:\n\n`mmt4d`\n\n,`pack`\n\n,`unpack`\n\n— enable specific microkernels (comma-separated)`all`\n\n— all of them`none`\n\n— none`default`\n\n— IREE's per-target default\n\nData-tiling together with the `mmt4d`\n\nmicrokernel is the recommended combination\non RISC-V. Enabling data-tiling while disabling microkernels makes the packed\n`mmt4d`\n\nfall back to generic vectorization; although this also generally produces\nefficient code, the microkernel path is currently the most stable one.\nFor background, see the\n[microkernels](https://iree.dev/community/blog/2024-01-22-microkernels/) and\n[mmt4d](https://iree.dev/community/blog/2021-10-13-mmt4d/) posts.\n\n### Static vs. scalable RVV[link](#static-vs-scalable-rvv)\n\nAt present, IREE's RISC-V vector path is **static / fixed-length**: the VLEN is\nfixed at compile time through `+zvl*b`\n\n, and both LLVM's vectorizer and IREE's\ntile-size selection specialize to that width. IREE derives its `mmt4d`\n\ntile shapes\nfrom the target's fixed-width vector register width, and the microkernels are compiled\nfor that same `+zvl*b`\n\ntarget, so the VLEN is the single value everything keys\noff of.\n\nThere is also preliminary support for **scalable, vector-length-agnostic** RVV\ncodegen — the `vscale`\n\n-style path that runs on any VLEN without recompiling, but\nit is still a work in progress. The scalable vectorization pipeline can be\nactivated with `--iree-llvmcpu-enable-scalable-vectorization=true`\n\n(which\ncurrently has to be combined with `--iree-experimental-vscale-value=VLEN/64`\n\nflag due to some ongoing work on the host compiler).\n\nCombined, a performance-oriented compilation for the mobilenet example is:\n\n```\niree-compile mobilenet.mlir -o mobilenet_rv64.vmfb \\\n  --iree-hal-target-device=local --iree-hal-local-target-device-backends=llvm-cpu \\\n  --iree-llvmcpu-target-triple=riscv64 --iree-llvmcpu-target-abi=lp64d \\\n  --iree-llvmcpu-target-cpu-features=+m,+a,+f,+d,+c,+zvl512b,+v \\\n  --iree-opt-data-tiling \\\n  --iree-global-opt-use-im2col-for-convs=true\n```\n\n## Running the module[link](#running-the-module)\n\nCopy the `.vmfb`\n\nand the cross-built `iree-run-module`\n\nto the target, or run under\nQEMU. Vector QEMU requires its `vlen`\n\nto match the `+zvl512b`\n\nused at compile time:\n\n```\n${QEMU_BIN} -cpu rv64,Zve64d=true,vlen=512,elen=64,vext_spec=v1.0 \\\n  -L ${RISCV_TOOLCHAIN_ROOT}/sysroot/ \\\n  ../iree-build-riscv/tools/iree-run-module \\\n  --device=local-task \\\n  --module=mobilenet_rv64.vmfb \\\n  --function=main \\\n  --input=@mobilenet_input.npy\n```\n\n`--device=local-task`\n\nselects the multithreaded runtime; `--device=local-sync`\n\nruns single-threaded and inline. Passing `--expected_output=@ref.npy`\n\ncompares the\nresult against a saved reference output as a correctness check. See\n`iree-run-module --help`\n\nfor the `--input`\n\n/ `--output`\n\nformats - inline literals,\nsplats such as `=0`\n\n, or `@file.npy`\n\n.\n\nFor a model whose weights live in a separate `.irpa`\n\n(like the Qwen3 above), pass\nthem with `--parameters=<scope>=<file>`\n\n; the scope is baked into the `.mlir`\n\n(here\n`model`\n\n):\n\n```\niree-run-module --device=local-task \\\n  --module=qwen3_rv64.vmfb --parameters=model=qwen3.irpa \\\n  --function=main --input=1x5xi64=1\n```\n\n## Benchmarking[link](#benchmarking)\n\n`iree-benchmark-module`\n\naccepts the same module, device, and input flags, and adds\nthe [Google Benchmark](https://github.com/google/benchmark) options on top:\n\n```\n${QEMU_BIN} -cpu rv64,Zve64d=true,vlen=512,elen=64,vext_spec=v1.0 \\\n  -L ${RISCV_TOOLCHAIN_ROOT}/sysroot/ \\\n  ../iree-build-riscv/tools/iree-benchmark-module \\\n  --device=local-task \\\n  --module=mobilenet_rv64.vmfb \\\n  --function=main \\\n  --input=@mobilenet_input.npy \\\n  --benchmark_repetitions=10\n```\n\nThe output looks like:\n\n```\nBenchmark                     Time             CPU   Iterations\nBM_main/real_time          12.3 ms         41.0 ms           57\nBM_main/real_time_mean     12.4 ms         41.2 ms           10\nBM_main/real_time_median   12.3 ms         41.0 ms           10\nBM_main/real_time_stddev    0.2 ms          0.7 ms           10\n```\n\ngreater than 1 produces the mean, median, and standard-deviation rows.`--benchmark_repetitions`\n\n- Other useful options:\n`--benchmark_min_time=1s`\n\n(or`100x`\n\nfor a fixed iteration count) and`--benchmark_format=json`\n\n.\n\nTo control threading, pin workers to specific cores with\n`--task_topology_cpu_ids=0,1,2,3`\n\n, or run single-threaded with `--device=local-sync`\n\n(preferably compiled with `--iree-llvmcpu-disable-distribution=true`\n\n). See\n`iree-run-module --help`\n\nfor the other `--task_topology_*`\n\noptions\n(worker/group counts, NUMA nodes, performance level).\n\nNote that under QEMU these are functional results rather than representative performance numbers. Representative timings require real hardware (or cycle-accurate simulators, which are hardly feasible to use for large programs that ML/AI models are).\n\n## Summary[link](#summary)\n\nThis post walks through building IREE, then importing, compiling, running, and\nbenchmarking a model on RISC-V. For more on the general flow and other CPU\ntargets, see IREE's [CPU deployment guide](https://iree.dev/guides/deployment-configurations/cpu/).", "url": "https://wpnews.pro/news/running-models-on-risc-v-with-iree", "canonical_source": "https://iree.dev/community/blog/2026-07-23-running-models-on-risc-v-with-iree/", "published_at": "2026-08-09 18:18:53+00:00", "updated_at": "2026-08-09 18:34:52.619056+00:00", "lang": "en", "topics": ["machine-learning"], "entities": ["IREE", "RISC-V", "PyTorch", "torchvision", "MobileNetV2", "ResNet18", "Qwen3-0.6B", "QEMU"], "alternates": {"html": "https://wpnews.pro/news/running-models-on-risc-v-with-iree", "markdown": "https://wpnews.pro/news/running-models-on-risc-v-with-iree.md", "text": "https://wpnews.pro/news/running-models-on-risc-v-with-iree.txt", "jsonld": "https://wpnews.pro/news/running-models-on-risc-v-with-iree.jsonld"}}