Hello!
I’m in a similar boat to yours. While my rant might not be helpful, I hope it atleast makes you feel less isolated in the chaos of running Local LLMs on an Intel Arc GPU. For context, I’m running Ubuntu 24.04.4 LTS on an Intel b580, Ryzen 5 7600 & 32Gib RAM.
I’ve got PTSD with Jira tickets so all work struggles have been logged in markdown files in this repo: Github - spatel333/ollama-on-intel-arc …as well as some sources for how to prep the GPU overall.
My biggest impetus for improving the stack came from discussions around OpenVINO have superior performance to Vulkan.
sources:
While a basic ollama deployment worked on my machine & even an OpenUI GUI could be spun up, the idea of subpar performance just didn’t sit right. Why invest in this hardware if we’re not getting its full potential? So I set out to learn the stack & where OpenVINO would fit. Similar to you, I’ve leveraged Claude to help demystify how the software meets the hardware. The general stack it mapped out is this:
| Concept Layers | Description | Column 3 | Column 4 |
|---|---|---|---|
| Framework | Here is where OpenVINO, PyTorch, oneAPI/SYCL, and other toolkits sit. There is no direct communication with the GPU, these simply call down to the API level. | ||
| Userspace API Implementation | Here is where Vulkan, OpenGL, VA-API, Level Zero, and other libraries sit. This is where libraries run API calls to leverage GPU compute. They don’t technically speak with the GPU directly, they speak with the Kernel driver which in turn speaks to the hardware registers. |
Different stacks leverage different libraries & compilers (I’ll touch on the libigc1/libigc2 struggle later in this post) | | | |
| DRM / libdrm | The Kernel driver gets exposed to the userspace as device nodes (e.g. /dev/dri/card0). Everything above this is in userspace. | | |
| Kernel Driver (KMD) | This is the only piece that talks directly to hardware registers. It receives system calls [such as ioctl()] and works on the registers. | | |
| Hardware | The GPU itself | | |
A key note was in the Userspace API Implementation section: “different stacks leverage different libraries & compilers”. This would explain why OpenVINO installation steps recommended Intel Drivers (Source) which leveraged libigc1, while Ubuntu 24.04 LTS ships with libigc2. A blocker which highlighted the gap in Intel Documentation.
In working on an ollama deployment, I’ve noticed the IPEX-LLM repository is archived (link to repo) and having an OpenVINO backend does not seem to be a feature of ollama. The path forward may be another product such as LM Studio, but I still have much to learn.
My end state is being able to locally serve an LLM for chatbot tutoring, VS Code integration, and ideally work with Qiskit.
If you’ve made progress, do share & if any of my details are misguided or wrong please correct me. This has all been a learning process.