When I talk about "quick answers," I am specifically looking for low time-to-first-token (TTFT) and high tokens-per-second. On an M4 Pro, you aren't just limited to tiny 3B parameter models; you can actually run some heavy hitters that still feel snappy.
The Vision vs. Speed Tradeoff #
The biggest headache in local deployment is that vision models (LMMs) are inherently heavier because they have to process the image embedding alongside the text tokens. If you want a seamless experience where you can drop a screenshot and get an immediate explanation, here is how I’ve been categorizing the current landscape:
The Speed Demon (Llama 3.2 Vision): If you need raw speed, the 11B version of Llama 3.2 is the current gold standard for a "fast" vision experience. It’s lightweight enough that the M4 Pro's GPU cores will absolutely shred through the inference. It's great for OCR tasks or describing UI elements quickly.The Intelligent All-Rounder (Moondream2): This is a tiny model, but don't sleep on it. It is incredibly fast—almost suspiciously so—but its reasoning capabilities are much more limited compared to the larger families. It's perfect if you just need to know "is there a cat in this photo?" rather than "analyze the architectural style of this building."The Heavyweight Champ (Qwen2-VL): If you find that Llama is hallucinating details in complex diagrams, Qwen2-VL is the move. It handles high-resolution inputs much better than most, though you will notice a slight dip in tokens-per-second compared to the 11B Llama models.
My Recommended Deployment Strategy #
Since you have 64GB of RAM, you should stop worrying about quantization levels too much and focus on maximizing the context window. Most people make the mistake of running highly compressed 4-bit models when they have the hardware to run 8-bit or even FP16 for specific tasks.
For a practical tutorial on getting this running via Ollama, I suggest this specific workflow:
-
Install Ollama (the easiest way to manage these local weights).
-
Pull the vision-capable models:
ollama run llama3.2-vision
ollama run moondream
3.Monitor your memory pressure: Use
asitop
in your terminal to see exactly how much of that 64GB your GPU is pulling during an image inference task.The real magic happens when you integrate these into an AI workflow where the vision model acts as a "pre-processor." For example, you can have a fast vision model describe an image, and then pass that text description to a much larger, text-only model (like a Llama 3.1 70B if you want to push that 64GB to the limit) for deep reasoning. That combo provides a much better experience than trying to force one single model to be both a genius and a speedster.
Next My ANN_SEARCH queries are absolutely nuking my OLTP performance →
these real-world AI monetization case studies, with plenty of directly applicable cases.