Look, I know Docker is an easy “push to start” for most things but for my environment where I have access to VMs and I’m running a hypervisor as-is rather than a docker-based server, it’s nice to actually get the bare metal setup.
For this I am using Fedora Server 44, and this comes with a few of its own quirks when building.
I basically spent the day because I really wanted that “double performance” for running a Radiance specific model and Quen 3.8 27B is a pretty decent model. While it’s not “amazing” I am curious now how far I can push it with RAG tools and all kinds of other things.
Recommended: Configure a static IP for your server
This guide assumes the following hardware configuration:
The reason is that the MXFP4 container fails on this build with either KV cache type:
fp8 → i8 → The IQR-V4 family (iq2r … iq8r ) is the int8-recipe line that the error itself points to. A single sentence — “Use an IQR-V4 container (iq2r–iq8r), not the MXFP4 one; the MXFP4 export’s GDN weights are not served by this build”
Without further ado, here we go.
Install git, and development packages to build from source.
sudo dnf install gcc gcc-c++ make cmake ninja-build git rocm-hip-devel rocm-smi hipcc
sudo dnf group install -y c-development
We want to do the following steps with this next shell segment
/opt/radiance folder to contain the application
cd /opt
git clone https://codeberg.org/StillDeadcode/radiance.git
cd /opt/radiance
cmake -S . -B build -G Ninja \
-DCMAKE_INSTALL_PREFIX=/opt/radiance \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DRAD_WITH_HIP=ON \
-DRAD_GPU_TARGETS='gfx1201' \
-DROCM_PATH=/usr
cmake --build build -j "$(nproc)"
sudo cmake --install build
We need to poke a hole in the system firewall to allow the new service to communicate on port 8080.
sudo firewall-cmd --permanent --new-service=radiance
sudo firewall-cmd --permanent --service=radiance --add-port=8080/tcp
sudo firewall-cmd --permanent --service=radiance --set-description="Radiance Local LLM Server"
sudo firewall-cmd --reload
Next we download the model we want (in my example here, it’s the IPQ6 for a 6-bit quant). I made an entry in the /srv directory
sudo mkdir -p /srv/models/qwen3.8-27b-iq6r
cd /srv/models/qwen3.8-27b-iq6r
curl -L -O https://huggingface.co/StillDeadcode/qwen3.8-27b-iqr-v4/resolve/main/qwen3.8-27b-iq6r-v4.rad?download=true
And now we can build the configuration yaml if you would like to test before we go ahead and make it a service.
sudo vi /opt/radiance.yaml
Here’s the file contents I used:
server:
host: 192.168.x.x # This is the listening IP, keep it local
port: 8080
memory:
gpu-memory-mib: 30000
models:
qwen3.8-27b-iq6r:
model: /srv/models/qwen3.8-27b-iq6r/qwen3.8-27b-iq6r-v4.rad
tp: 1
kv-cache-dtype: i8
max-model-len: 32768
max-num-seqs: 8
If you like, we can go ahead and run some checks against running the radiance service and hopefully catch any errors.
We check the yaml then confirm it can start up with everything.
/opt/radiance/bin/radiance --config /opt/radiance.yaml --check
/opt/radiance/bin/radiance --config /opt/radiance.yaml
As long as it starts without any errors, we are good to continue.
Now we can configure the service.
Create and edit the file /etc/systemd/system/radiance.service
[Unit]
Description=Radiance Local LLM Server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStartPre=/opt/radiance/bin/radiance --config /opt/radiance.yaml --check
ExecStart=/opt/radiance/bin/radiance --config /opt/radiance.yaml
Restart=on-failure
RestartSec=10
TimeoutStopSec=60
LimitNOFILE=65536
LimitMEMLOCK=infinity
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
Enable the service and turn it on.
sudo systemctl daemon-reload
sudo systemctl enable --now radiance.service
sudo systemctl start radiance.service
Now connect your favorite OpenAI API to the service and make sure it can read the models.
Optionally, you can curl that (my local server IP is 192.168.x.x). It’s important you don’t curl localhost because the server is only listening on the LAN Address of 192.168.x.x
curl -s http://192.168.x.x:8080/v1/models