# Running three AI models on one local server when your VRAM doesn't cover all of them > Source: > Published: 2026-08-17 02:49:37+00:00 The first time I tried loading Whisper, bge-m3, and gemma at the same time on my local box, it OOM’d immediately. I’d known this was going to happen, but I tried anyway to see where the ceiling actually was. The machine is a workstation I already had, enough VRAM for any single model but not three simultaneously. I had some options: get more hardware, split across multiple machines, or figure out a sequential loading pattern. I went with sequential because adding hardware or managing separate boxes felt like problems I didn't want to solve yet. Sequential loading means: load whatever model you need, use it, unload before moving on. Nothing runs in parallel. This was fine for my use case because the workloads don't actually overlap in time — I'vm not running embedding lookups while transcribing a recording. The load time overhead adds a few seconds per task. For Whisper, that's ~4 seconds on top of ~90 seconds of transcription for a 30-minute call, which I don't notice in practice. **The bge-m3 decision took the most deliberation** I started with `all-MiniLM-L6-v2` for embeddings. It's small and fast. The issue was Korean/English mixed documents. I'm processing meeting summaries and reference docs that switch between the two, and MiniLM's cross-lingual similarity scores were unreliable enough that I was getting wrong nearest-neighbor results. bge-m3 handles cross-lingual matching better. It costs more VRAM and I had to drop the batch size from the default to stabilize it, but the accuracy difference on my actual data was clear. One thing I didn't fully account for: when bge-m3 and Whisper are both unloaded, the load time for whichever comes next varies. bge-m3 seems to be slower on first load than subsequent loads in the same session, probably something to do with model weights caching at the OS level. I haven't investigated this properly. **Whisper was the easy pick** medium.en. I ran large once to compare on a client call recording and the accuracy improvement wasn't worth the extra VRAM cost for my use case. Transcription time for a 30-minute recording is under two minutes with medium. That's fast enough that I'm not sitting watching it. **Gemma was a coin flip between a few candidates** I needed local image analysis for screenshots and scanned documents. I tested gemma, a couple of LLaVA variants, and MiniCPM-V on maybe 20 images from my actual use case. Counted how many each model described correctly. Gemma came out ahead on document-heavy images. That test was informal enough that a different set of images might have given different results, but I had to pick something. One thing I noticed later: gemma's processing time varies a lot by input image resolution. Standard screenshots are fast. A high-res photo from a phone camera takes noticeably longer. I still haven't profiled exactly what's happening there. If you're planning around latency, test with the actual image sizes you'll be using, not benchmarks from smaller inputs. **Where things stand** The sequential loading setup is stable. I’ve been running it for a few weeks without the OOM errors from the first attempt. The models work individually and I have code that chains them in sequence for a given input set. What I’m still building is the layer that makes use of this: a pipeline that processes call recordings, emails, and messages together and produces something more useful than three separate outputs. The individual pieces are working. Connecting them into a coherent pipeline is the current work. *Originally posted at hannune.ai*