{"slug": "lowvram-a-cli-tool-to-run-large-ai-models-in-low-vram-gpus", "title": "Lowvram: A CLI tool to run large AI models in low VRAM GPUs", "summary": "A developer released lowvram, an open-source CLI tool that intercepts CUDA calls via LD_PRELOAD and redirects cuMemAlloc to cuMemAllocManaged so applications can run on GPUs with less VRAM than they require. On an NVIDIA GeForce RTX 3060 with 6 GB of VRAM, the tool ran YuE2 music generation at 43.1 tokens per second (range 41.0–46.0), about 95% of an oracle setup's 45.3 tokens per second and 23× faster than Hugging Face Accelerate's 1.9 tokens per second, while unmodified PyTorch failed with a CUDA out-of-memory error. The tool also ran Llama-TTS with the Qwen3-TTS-12Hz-1.7B-Base-Q8_0 model that otherwise failed with cudaMalloc out of memory, trading speed for the ability to execute workloads on constrained hardware.", "body_md": "Sometimes the cutting-edge AI tools require more GPU memory than consumer hardware can provide. When standard optimization fails, custom solutions are needed. Here is how I tackled the memory wall using a simple intercept layer.\n\nI recently wanted to explore [YuE2](https://github.com/multimodal-art-projection/YuE): frontier music generation. The requirements were clear: *Linux, Python 3.12, an NVIDIA GPU supporting BF16, and critically, 24 GB of VRAM*. My setup, featuring an *NVIDIA GeForce RTX 3060 with only 6 GB of VRAM*, was fundamentally incompatible right out of the box.\n\nMy first attempt was to adapt the existing code, leveraging PyTorch's CPU offloading capabilities. This required deep diving into the YuE2 source code, which was time-consuming and fragile—any update to the tool broke my customized solution. I needed a solution that was agnostic, something that could run without touching the underlying application code.\n\nThe goal was to intercept the GPU memory requests at the driver level. This led to the creation of `lowvram`, a tool designed to wrap CUDA programs and allow them to run even when VRAM is insufficient.\n\nIn short, `lowvram` intercepts CUDA calls in the library layer, trickng the application into believing that system RAM is available for use by the GPU kernel.\n\nYou can find the source code and documentation for the tool here: [lowvram on GitHub](https://github.com/henriquegogo/lowvram).\n\n`cuMemAlloc` calls (which usually only use VRAM) with `cuMemAllocManaged`.` LD_PRELOAD` or the wrapper, the tool ensures its hooks are active before the target application begins execution.\nThe effectiveness of this approach was demonstrated across several use cases where memory constraints usually halt execution.\n\nWhile the specific setup for YuE2 is complex, the philosophy holds: by using `lowvram`, the model attempts to load and run on a 6GB card, trading raw speed for the ability to execute the workload that would otherwise fail with a \"CUDA Out of Memory\" error.\n\nUnmodified PyTorch cannot load YuE2 on this GPU. Under lowvram the same code generates 43.1 tokens per second (range over runs 41.0–46.0): 95% of the 45.3 tokens per second of the oracle, which runs the AR half from ordinary device memory after we deleted the NAR half by hand. Accelerate, the standard way to run an oversized Hugging Face model, reaches 1.9 tokens per second-lowvram is 23× faster-and the CPU 6.8.\n\nA clear, practical demonstration involved using Llama-TTS with the Qwen3-TTS model. Standard execution resulted in an immediate `cudaMalloc failed: out of memory` error.\n\nBy simply prepending the tool:\n\n``` bash\n$ lowvram llama-tts -m Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf -mm mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf --tts-lang en -p \"Hello, world\"\n```\n\nThe program ran successfully.\n\nWhile `lowvram` provides a powerful workaround, it is not a silver bullet. It involves fundamental changes to how the GPU interacts with system resources, and these changes come with performance trade-offs. It is crucial to be aware of these limitations:\n\nIn the realm of local AI deployment, the barrier between a perfect setup and a functional setup can often be just one line of code or one tool. If a powerful CUDA application simply won't fit your GPU, a highly practical and clever solution like `lowvram` offers a pathway to run it. It prioritizes functionality over peak performance, proving that sometimes, the most practical approach is the simplest one.", "url": "https://wpnews.pro/news/lowvram-a-cli-tool-to-run-large-ai-models-in-low-vram-gpus", "canonical_source": "https://soarhen.com/how-run-ai-in-low-vram/", "published_at": "2026-11-06 12:29:42+00:00", "updated_at": "2026-11-06 12:50:16.075447+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "large-language-models", "generative-ai", "developer-tools"], "entities": ["lowvram", "YuE2", "NVIDIA GeForce RTX 3060", "PyTorch", "Hugging Face Accelerate", "Llama-TTS", "Qwen3-TTS-12Hz-1.7B-Base-Q8_0", "CUDA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/lowvram-a-cli-tool-to-run-large-ai-models-in-low-vram-gpus", "markdown": "https://wpnews.pro/news/lowvram-a-cli-tool-to-run-large-ai-models-in-low-vram-gpus.md", "text": "https://wpnews.pro/news/lowvram-a-cli-tool-to-run-large-ai-models-in-low-vram-gpus.txt", "jsonld": "https://wpnews.pro/news/lowvram-a-cli-tool-to-run-large-ai-models-in-low-vram-gpus.jsonld"}}