Originally posted here: Setting up LLama.cpp | Darin Rawson
Note: I am generally not in favor of AI. AI should not be used for code creation and should generally be avoided when doing anything that has the ability to make you grow as a person.. Using AI all the time will harm you long term as it doesn’t make you think and gives the illusion of talking to a person. In reality LLMs are not alive and are simply prediction engines. I would strongly recommend that you work though problems without relying on a chatbot.
Over the last few years, I’ve been playing with LLMs as a hobby since they are fun to tinker with. I started off with Ollama on a CPU but overtime I’ve moved into GPU acceration. As it turns out, you don’t need anything special to run many of the smaller models. I have a AMD 580, which is working just fine thanks to Vulkan.
I think Ollama has many merits, especially since it provides a very polished feel. However, Ollama has become less of a community project recently with the addition of pain subscriptions. Although it is still open source, I am still concerned about its future.
For this guide, I will use llama.cpp. Historically, it was limited on functionality, but these days it has a nice web UI as well as an API. More Importantly, it absolutely destroys Ollama in terms of performance.
I played with some different install methods and figured out, though trial and error, that the easiest way to use llama.cpp is though the official container images. The documentation isn’t entirely clear but hopefully, I can help explain. The biggest resource I was able to find was this page: llama.cpp/docs/docker.md at master · ggml-org/llama.cpp · GitHub
On this page, you will see several different images. What you generally want is the image corresponding to the backend you are using. I am using Vulkan on a XFX RX 580 8 GB so in my case I choose the Vulkan image. If you want to use the web UI, you will need the server-* image. To download it run the following:
podman pull ghcr.io/ggml-org/llama.cpp:server-vulkan
Next, we need to download some models. This is where I got completely lost, as llama.cpp is not all that polished yet. Within llama.cpp, there are effectively two “modes” you can be in. The first mode is when you use the -hf flag. This tells llama.cpp to download a model from Hugging Face if it isn’t already downloaded before starting the server. The problem (as of writing this) with the -hf flag is that it will not show any other models in the web UI. For additional models to show up you need to be in router mode.
To download a model run the following:
podman run --rm -p 127.0.0.1:8080:8080 --device /dev/dri --volume ~/models:/root/.cache:Z ghcr.io/ggml-org/llama.cpp:server-vulkan -hf <gguf model from Hugging Face>
In the command above, we are opening 8080 to localhost and passing in all GPUs on the system, as well as the directory we want to store our models. llama.cpp does not have a official downloads directory, so we instead map the cache directory to the “models” folder. When you run the command nothing will happen for a while since llama.cpp does not show download progress. You will know it completed when you see listening on http://0.0.0.0:8080``. **Note:** It may not work on localhost:8080 since we told Podman to listen on127.0.0.1 not::1``` mostly due to the fact that llama.cpp doesn’t seem to have IPv6 support as far as I can tell.
In my testing, I settled on the following models (notice they are all GGUF formats):
ibm-granite/granite-4.1-3b-GGUF:Q4_K_M
google/gemma-4-E4B-it-qat-q4_0-gguf:IT
LiquidAI/LFM3-1.7B-GGUF:Q4_K_M
Qwen/Qwen3-8B-GGUF:Q4_K_M
To download all of them, you will need to run llama.cpp once for each with the -hf option. (as seen in the above command)
To start llama.cpp in router mode run llama.cpp without any flags like so:
podman run -d -p 127.0.0.1:8080:8080 --device /dev/dri --volume ~/models:/root/.cache:Z ghcr.io/ggml-org/llama.cpp:server-vulkan
Router mode is what makes llama.cpp behave like Ollama. (abet with a different API) All of your models will then be available in the web UI when you run llama.cpp in router mode.
In the web UI, you can control which models are loaded into VRAM. Once your VRAM is exhausted, llama.cpp will fall back to system RAM, which will destroy performance. To change models, first unload any existing models so that your VRAM is freed. If you don’t do this, or if you change them out of order, your performance will rapidly degrade.