Setting up LLama.cpp on a older GPU

A hobbyist guide published November 9, 2026 on darin755.xyz documents running llama.cpp's official container images on an 8 GB AMD Radeon RX 580 via the Vulkan backend, pulling ghcr.io/ggml-org/llama.cpp:server-vulkan with Podman and serving models on 127.0.0.1:8080. The author reports llama.cpp "absolutely destroys Ollama in terms of performance" and recommends GGUF models including ibm-granite/granite-4.1-3b-GGUF:Q4_K_M, google/gemma-4-E4B-it-qat-q4_0-gguf:IT, LiquidAI/LFM3-1.7B-GGUF:Q4_K_M and Qwen/Qwen3-8B-GGUF:Q4_K_M. The guide notes that the -hf flag downloads a model from Hugging Face but hides other models in the web UI unless llama.cpp runs in router mode, and that downloads show no progress until the server prints "listening on http://0.0.0.0:8080".

Originally posted here: Setting up LLama.cpp | Darin Rawson https://darin755.xyz/ai/llms/llamacpp/2026/11/09/running-llamacpp.html Note: I am generally not in favor of AI. AI should not be used for code creation and should generally be avoided when doing anything that has the ability to make you grow as a person.. Using AI all the time will harm you long term as it doesn’t make you think and gives the illusion of talking to a person. In reality LLMs are not alive and are simply prediction engines. I would strongly recommend that you work though problems without relying on a chatbot. Over the last few years, I’ve been playing with LLMs as a hobby since they are fun to tinker with. I started off with Ollama on a CPU but overtime I’ve moved into GPU acceration. As it turns out, you don’t need anything special to run many of the smaller models. I have a AMD 580, which is working just fine thanks to Vulkan. I think Ollama has many merits, especially since it provides a very polished feel. However, Ollama has become less of a community project recently with the addition of pain subscriptions. Although it is still open source, I am still concerned about its future. For this guide, I will use llama.cpp. Historically, it was limited on functionality, but these days it has a nice web UI as well as an API. More Importantly, it absolutely destroys Ollama in terms of performance. I played with some different install methods and figured out, though trial and error, that the easiest way to use llama.cpp is though the official container images. The documentation isn’t entirely clear but hopefully, I can help explain. The biggest resource I was able to find was this page: llama.cpp/docs/docker.md at master · ggml-org/llama.cpp · GitHub https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md On this page, you will see several different images. What you generally want is the image corresponding to the backend you are using. I am using Vulkan on a XFX RX 580 8 GB https://www.techpowerup.com/gpu-specs/xfx-gts-rx-580-xxx-8-gb.b4480 so in my case I choose the Vulkan image. If you want to use the web UI, you will need the server- image. To download it run the following: podman pull ghcr.io/ggml-org/llama.cpp:server-vulkan Next, we need to download some models. This is where I got completely lost, as llama.cpp is not all that polished yet. Within llama.cpp, there are effectively two “modes” you can be in. The first mode is when you use the -hf flag. This tells llama.cpp to download a model from Hugging Face if it isn’t already downloaded before starting the server. The problem as of writing this with the -hf flag is that it will not show any other models in the web UI. For additional models to show up you need to be in router mode. To download a model run the following: podman run --rm -p 127.0.0.1:8080:8080 --device /dev/dri --volume ~/models:/root/.cache:Z ghcr.io/ggml-org/llama.cpp:server-vulkan -hf <gguf model from Hugging Face In the command above, we are opening 8080 to localhost and passing in all GPUs on the system, as well as the directory we want to store our models. llama.cpp does not have a official downloads directory, so we instead map the cache directory to the “models” folder. When you run the command nothing will happen for a while since llama.cpp does not show download progress. You will know it completed when you see listening on http://0.0.0.0:8080 . Note: It may not work on localhost:8080 since we told Podman to listen on 127.0.0.1 not ::1 mostly due to the fact that llama.cpp doesn’t seem to have IPv6 support as far as I can tell. In my testing, I settled on the following models notice they are all GGUF formats : ibm-granite/granite-4.1-3b-GGUF:Q4 K M google/gemma-4-E4B-it-qat-q4 0-gguf:IT LiquidAI/LFM3-1.7B-GGUF:Q4 K M Qwen/Qwen3-8B-GGUF:Q4 K M To download all of them, you will need to run llama.cpp once for each with the -hf option. as seen in the above command To start llama.cpp in router mode run llama.cpp without any flags like so: podman run -d -p 127.0.0.1:8080:8080 --device /dev/dri --volume ~/models:/root/.cache:Z ghcr.io/ggml-org/llama.cpp:server-vulkan Router mode is what makes llama.cpp behave like Ollama. abet with a different API All of your models will then be available in the web UI when you run llama.cpp in router mode. In the web UI, you can control which models are loaded into VRAM. Once your VRAM is exhausted, llama.cpp will fall back to system RAM, which will destroy performance. To change models, first unload any existing models so that your VRAM is freed. If you don’t do this, or if you change them out of order, your performance will rapidly degrade.