How to Run Mistral Large 4 Locally: Hardware, GGUF Sizes and vLLM (Oct 2026)

Mistral released the weights of Mistral Large 4 on 29 Oct 2026, a mixture-of-experts model with 49 billion active parameters out of about 1 trillion, a 1M token context window and text-plus-image input, and running it locally requires roughly 256 GB of memory or more. Unsloth's 1-bit UD-IQ1_S GGUF is 233 GB and "should fit on 256GB RAM including a Mac Studio or 2x NVIDIA DGX Sparks," while its 2-bit UD-Q2_K_XL file is 378 GB and keeps 95.6% of top-1 accuracy against the original; the 4-bit file is 605 GB, the 8-bit Q8_0 is 1.12 TB, and Mistral recommends serving the full checkpoint with vLLM across 8 GPUs using --tensor-parallel-size 8.

How to Run Mistral Large 4 Locally: Hardware, GGUF Sizes and vLLM Oct 2026 Mistral Large 4 weights are out. Which file fits your memory, the llama.cpp and vLLM commands, the GPUs Mistral validated and the licence, checked 29 Oct 2026. You can run Mistral Large 4 yourself from today, but only with about 256 GB of memory or more: a 1 bit GGUF on a large Mac Studio, a 2 to 4 bit GGUF on a workstation, or the full checkpoint on 8 data center GPUs with vLLM. Mistral released the weights on 29 Oct 2026, checked the same day. If you want a team of agents on your own work, Munder Difflin https://harnessmd.com/download is free and open source: a desktop app that runs a team of AI agents for all kinds of knowledge work, coding included, on your own computer. For what the model is and its scores, see Mistral Large 4 https://munderdiffl.in/blog/mistral-large-4/ . This guide sits in our Guides hub https://munderdiffl.in/blog/topics/guides/ . What do you need to run Mistral Large 4? what-do-you-need-to-run-mistral-large-4 Total memory decides it: RAM plus VRAM, or unified memory, has to be larger than the file you load. The model card https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B , read 29 Oct 2026, describes a mixture of experts model with “49 billion active parameters per token” out of about 1 trillion, a 1M token context window and text plus image input. Only the active part runs for each token, but all of it has to sit in memory. That is why a model with 49B active parameters still needs hundreds of gigabytes. | File checked 29 Oct 2026 | Size | Source | |---|---|---| | UD-IQ1 S, 1 bit | 233 GB | Unsloth | | UD-Q2 K XL, 2 bit | 378 GB | Unsloth | | UD-Q4 K XL, 4 bit | 605 GB | Unsloth | | Q8 0, 8 bit | 1.12 TB | ggml-org | | NVFP4 checkpoint | about 610 GB | vLLM recipe | | FP8 checkpoint | about 1.06 TB | vLLM recipe | Unsloth’s guide https://unsloth.ai/docs/models/mistral-large-4 says the 1 bit file “should fit on 256GB RAM including a Mac Studio or 2x NVIDIA DGX Sparks”, and that its 2 bit file keeps 95.6% of top 1 accuracy against the original. Longer contexts need more memory on top of the file. How to run it on one machine with llama.cpp how-to-run-it-on-one-machine-with-llamacpp Use a GGUF file and a llama.cpp build with Mistral Large 4 support. ggml-org publishes GGUF files https://huggingface.co/ggml-org/Mistral-Large-4-1T-A52B-GGUF from 298 GB up, and Unsloth publishes its own. Unsloth’s example serves the 2 bit file with a 32K context: ./llama.cpp/build/bin/llama-server \ -hf unsloth/Mistral-Large-4-1T-A52B-GGUF:UD-Q2 K XL \ --ctx-size 32768 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \ --chat-template-kwargs '{"reasoning effort":"high"}' \ --host 127.0.0.1 --port 8080 That starts a chat page at http://127.0.0.1:8080 http://127.0.0.1:8080 and an API at /v1. On a Mac, Unsloth says “Metal support is enabled by default”. If you run out of memory, it says to lower the context size or pick a smaller quant. Unsloth Desktop does the same on macOS, Windows and Linux without the command line. Two settings come from the model card: temperature 1.0 and top p 0.95 for every reasoning effort. The ggml-org page adds that “Reasoning cannot be turned off”: low, medium and high are the choices, and high is the default. How to serve it on GPUs with vLLM how-to-serve-it-on-gpus-with-vllm Mistral recommends vLLM https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B for production. The model card serves it from the official Docker image across 8 GPUs: docker run --runtime nvidia --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 8000:8000 --ipc=host \ vllm/vllm-openai:mistral-large-4 \ --model mistralai/Mistral-Large-4-1T-A52B \ --tensor-parallel-size 8 \ --tool-call-parser mistral --reasoning-parser mistral \ --enable-auto-tool-choice --gpu-memory-utilization 0.85 The two parser flags are required for tool calls and to keep reasoning out of the answer. vLLM’s recipe lists validated hardware: NVIDIA GB300, 8x B300, 8x B200, two nodes of 8 H200 for FP8, and AMD Instinct MI350X. It says FP8 “needs ~128 GiB of weights per GPU on 8 GPUs”, NVFP4 about 76 GiB. The recipe validated contexts up to 256K, not the full 1M. SGLang works too, with its own image. What does the licence allow? what-does-the-licence-allow Almost everything except selling the model as a hosted service. Mistral’s post of 29 Oct 2026 https://mistral.ai/news/mistral-large-4-oss/ calls it the Mistral Ecosystem Open-Weight license, “a modified MIT license”, free to “use, modify, fine-tune, and commercialize”. The one condition: “businesses that operate a Model-as-a-Service offering must obtain a separate commercial license from Mistral”. The licence text https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/LICENSE is in the repo. Should you self host or use the API? should-you-self-host-or-use-the-api Use the API unless the data has to stay on hardware you control. Mistral’s post priced it at $0.70 per million input tokens, $0.07 cached and $2.10 output on 29 Oct 2026, and the model card says to use the Mistral API “if local serving is subpar”. The post also lists OpenRouter, Microsoft Foundry and NVIDIA hosted endpoints. Self hosting earns its cost when the model must run under your own control, or when you want to fine tune it; Unsloth, Axolotl and TRL all list guides. Can you use it with coding agents? can-you-use-it-with-coding-agents Yes, through any agent that accepts an OpenAI compatible base URL; both servers above expose one. Mistral points to its own coding agent, Mistral Vibe. Munder Difflin runs a team of agents for research, writing, ops and coding on engines such as Claude Code, Codex and Gemini CLI, takes work in from Slack and webhooks, and connects tools through MCP servers, skills and plugins. Some of its engines accept a local base URL; running Munder Difflin on open models https://munderdiffl.in/blog/run-munder-difflin-on-open-models/ has the wiring. A Cloud Office and a mobile app are coming soon; how to install and use Munder Difflin https://munderdiffl.in/blog/how-to-install-and-use-munder-difflin/ covers setup. For a lighter local setup, see how to connect Ollama to Claude Code https://munderdiffl.in/blog/how-to-connect-ollama-to-claude-code/ . FAQ How much memory do you need to run Mistral Large 4? At least about 256 GB of RAM plus VRAM, or unified memory, for the smallest file. Unsloth's guide of 29 Oct 2026 says its 1 bit UD-IQ1 S quant uses about 233 GB of disk and should fit on 256 GB, including a Mac Studio. The 4 bit UD-Q4 K XL file is 605 GB and the full FP8 checkpoint about 1.06 TB. Can Mistral Large 4 run on a Mac? Only a Mac with 256 GB or more of unified memory, using a heavily quantized GGUF. Unsloth names a Mac Studio for its 233 GB 1 bit file, and llama.cpp enables Metal by default on macOS. ggml-org lists its 381 GB Q2 K and 298 GB IQ2 XXS files as fitting in 512 GB of unified memory. What GPUs does vLLM support for Mistral Large 4? vLLM's recipe, read 29 Oct 2026, lists validated runs on NVIDIA GB300, 8x B300, 8x B200, two nodes of 8 H200 with FP8, and AMD Instinct MI350X with 8 GPUs. It says FP8 needs about 128 GiB of weights per GPU on 8 GPUs and NVFP4 about 76 GiB. Is Mistral Large 4 free for commercial use? Mostly. Mistral's post of 29 Oct 2026 says the Mistral Ecosystem Open-Weight license, a modified MIT license, lets you use, modify, fine tune and commercialize the model, with one condition: a business that runs a Model as a Service offering needs a separate commercial license from Mistral. Is it cheaper to use the Mistral API than to self host? For most people, yes. Mistral's post of 29 Oct 2026 prices the API at $0.70 per million input tokens, $0.07 cached and $2.10 output, and the model card recommends the Mistral API if local serving is subpar. Self hosting pays off when the data must stay on your own hardware.