Building an AI Runtime Operating System for Commodity Hardware (UGR)

A developer has open-sourced UGR, an AI runtime operating system designed to orchestrate inference across commodity hardware by treating models as collections of resources that can move between memory tiers. The project sits above engines like llama.cpp and includes a simulation engine to predict execution feasibility before loading models.

For the last few months I've been working on something that started as a simple question: Why do we still treat AI inference as "load an entire model into GPU memory and hope it fits"? My development machine certainly doesn't make life easy. Like many developers experimenting with open-source LLMs, I quickly ran into the same wall everyone eventually hits: CUDA Out of Memory At first I assumed the answer was simply "buy a bigger GPU." But after digging through projects like llama.cpp, Ollama, TensorRT-LLM, Colibri, and various memory streaming experiments, I started wondering whether we were solving the wrong problem. Most inference engines think about execution like this: Application ↓ Inference Engine ↓ GPU If the model fits, great. If it doesn't... You're usually done. Modern runtimes have become incredibly efficient, but they're still primarily focused on executing tensors as quickly as possible. What if we instead treated inference more like an operating system treats processes and memory? That idea eventually became UGR. Instead of replacing inference engines like llama.cpp, UGR sits above them as an orchestration layer. Application │ OpenAI-Compatible API │ Planner │ Runtime Intelligence Engine │ Scheduler │ Virtual Resource Manager │ Adaptive Memory Fabric │ Driver Abstraction Layer │ llama.cpp / TensorRT / Vulkan The execution backend becomes interchangeable. The runtime becomes responsible for deciding how execution should happen. Rather than managing a single GGUF file, UGR treats AI workloads as collections of resources. That includes things like: These resources can move between different memory tiers depending on runtime conditions. VRAM ↓ RAM ↓ NVMe ↓ Remote Storage ↓ Archive Instead of asking: Can this model fit into VRAM? The runtime asks: How should these resources be orchestrated across available hardware? One thing I kept running into was that modern AI software often assumes modern hardware. Reality is different. There are countless developers, researchers, universities, and small teams running inference on older GPUs or mixed hardware environments. Rather than optimizing for a single GPU architecture, I wanted UGR to detect hardware capabilities and adapt its execution strategy accordingly. One feature I'm particularly excited about is the simulation engine. Instead of loading a model and hoping for the best: ugr simulate gemma-4-31b.gguf The runtime predicts: ...before execution begins. UGR is still very much a research project. Currently implemented: Still under active development: I decided to open source the project because I don't believe architectures improve in isolation. I also don't have access to high-end hardware. Most of my development has been on a machine with a 4 GB Quadro GPU, so I would love feedback from people running larger GPUs or heterogeneous systems. If nothing else, I hope some of the ideas around resource virtualization and runtime orchestration spark interesting discussions. ⭐ UGR AI Runtime Operating System https://github.com/UGEM-io/UGR https://github.com/UGEM-io/UGR I'd genuinely appreciate feedback, bug reports, architectural criticism, or ideas for improving the runtime. When I started this project, I thought I was trying to solve a memory problem. Today I think I'm trying to solve a runtime orchestration problem. Whether UGR ultimately becomes a useful research platform or something more, I'm excited to see where the community takes it.