{"slug": "running-three-ai-models-on-one-local-server-when-your-vram-doesn-t-cover-all-of", "title": "Running three AI models on one local server when your VRAM doesn't cover all of them", "summary": "A developer detailed a sequential model-loading approach to run Whisper, bge-m3, and Gemma on a single local workstation with limited VRAM, avoiding out-of-memory errors by loading and unloading models as needed. The setup, which prioritizes accuracy for Korean/English mixed documents and local image analysis, has been stable for weeks, with ongoing work to integrate the models into a unified pipeline.", "body_md": "The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it OOM’d immediately. I’d known this was going to happen, but I tried anyway to see where the ceiling actually was.\n\nThe machine is a workstation I already had, enough VRAM for any single model but not three simultaneously. I had some options: get more hardware, split across multiple machines, or figure out a sequential loading pattern. I went with sequential because adding hardware or managing separate boxes felt like problems I didn't want to solve yet.\n\nSequential loading means: load whatever model you need, use it, unload before moving on. Nothing runs in parallel. This was fine for my use case because the workloads don't actually overlap in time — I'vm not running embedding lookups while transcribing a recording. The load time overhead adds a few seconds per task. For Whisper, that's ~4 seconds on top of ~90 seconds of transcription for a 30-minute call, which I don't notice in practice.\n\n**The bge-m3 decision took the most deliberation**\n\nI started with `all-MiniLM-L6-v2`\n\nfor embeddings. It's small and fast. The issue was Korean/English mixed documents. I'm processing meeting summaries and reference docs that switch between the two, and MiniLM's cross-lingual similarity scores were unreliable enough that I was getting wrong nearest-neighbor results. bge-m3 handles cross-lingual matching better. It costs more VRAM and I had to drop the batch size from the default to stabilize it, but the accuracy difference on my actual data was clear.\n\nOne thing I didn't fully account for: when bge-m3 and Whisper are both unloaded, the load time for whichever comes next varies. bge-m3 seems to be slower on first load than subsequent loads in the same session, probably something to do with model weights caching at the OS level. I haven't investigated this properly.\n\n**Whisper was the easy pick**\n\nmedium.en. I ran large once to compare on a client call recording and the accuracy improvement wasn't worth the extra VRAM cost for my use case. Transcription time for a 30-minute recording is under two minutes with medium. That's fast enough that I'm not sitting watching it.\n\n**Gemma was a coin flip between a few candidates**\n\nI needed local image analysis for screenshots and scanned documents. I tested gemma, a couple of LLaVA variants, and MiniCPM-V on maybe 20 images from my actual use case. Counted how many each model described correctly. Gemma came out ahead on document-heavy images. That test was informal enough that a different set of images might have given different results, but I had to pick something.\n\nOne thing I noticed later: gemma's processing time varies a lot by input image resolution. Standard screenshots are fast. A high-res photo from a phone camera takes noticeably longer. I still haven't profiled exactly what's happening there. If you're planning around latency, test with the actual image sizes you'll be using, not benchmarks from smaller inputs.\n\n**Where things stand**\n\nThe sequential loading setup is stable. I’ve been running it for a few weeks without the OOM errors from the first attempt. The models work individually and I have code that chains them in sequence for a given input set.\n\nWhat I’m still building is the layer that makes use of this: a pipeline that processes call recordings, emails, and messages together and produces something more useful than three separate outputs. The individual pieces are working. Connecting them into a coherent pipeline is the current work.\n\n*Originally posted at hannune.ai*", "url": "https://wpnews.pro/news/running-three-ai-models-on-one-local-server-when-your-vram-doesn-t-cover-all-of", "canonical_source": "https://dev.to/hannune/running-three-ai-models-on-one-local-server-when-your-vram-doesnt-cover-all-of-them-b7g", "published_at": "2026-08-17 02:49:37+00:00", "updated_at": "2026-08-17 03:11:51.314528+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["Whisper", "bge-m3", "Gemma", "MiniLM", "LLaVA", "MiniCPM-V"], "alternates": {"html": "https://wpnews.pro/news/running-three-ai-models-on-one-local-server-when-your-vram-doesn-t-cover-all-of", "markdown": "https://wpnews.pro/news/running-three-ai-models-on-one-local-server-when-your-vram-doesn-t-cover-all-of.md", "text": "https://wpnews.pro/news/running-three-ai-models-on-one-local-server-when-your-vram-doesn-t-cover-all-of.txt", "jsonld": "https://wpnews.pro/news/running-three-ai-models-on-one-local-server-when-your-vram-doesn-t-cover-all-of.jsonld"}}