How to Choose a Local LLM: Models, Hardware, and Quantization

This is a guest blog prepared by the amazing team at Atomic Chat. In this article you’ll learn:

• How to match a model to your hardware
• How to select the right quantization level and model file format
• Which models to choose by task
• What some of the best models are based on hardware tiers

Match the Model to Your Hardware #

Check if the model will fit comfortably in the memory available to your runtime, taking into account the KV cache, runtime allocations, and the rest of the system.

You can estimate how much memory you need for the weights using the formula below:

parameters × bits per weight ÷ 8

For example, by this calculation, a 27B model at FP16 needs roughly 54GB of VRAM or unified memory, before accounting for the additional memory required to hold context and runtime overhead. The KV cache can be estimated separately with the following formula:

2 × layers × context tokens × KV heads × head dimension × bytes per element

The 2 accounts for storing both keys and values. For a model using an FP16 or BF16 KV cache, use 2 bytes per element; an 8-bit KV cache uses roughly 1 byte per element. Multiply the result by the batch size if you are processing more than one sequence at a time.

For example, a model with 48 layers, 8 KV heads, a head dimension of 128, and a 32,768-token context requires about 6GB for an FP16 KV cache: 2 × 48 × 32,768 × 8 × 128 × 2 ≈ 6GB

Add that to the model’s loaded size, then leave further headroom for the runtime and operating system.

VRAM vs. Unified Memory

A discrete GPU has its own high-bandwidth VRAM, so models that fit entirely within it tend to deliver the best performance. NVIDIA also has the advantage of broad CUDA support in most local AI apps, such as Ollama, LM Studio, and Atomic Chat. However, if the weights don’t fit in memory entirely, they’re partially offloaded to RAM, which leads to much slower performance.

Unified-memory systems trade some of that GPU-specific performance for a larger shared memory pool, allowing you to run models that potentially have higher parameter counts without off part of the weights to the CPU.

For example, Apple’s Mac Studio can be configured with substantially more memory than a typical discrete GPU, while AMD’s Ryzen AI Max+ 395 supports up to 128GB of unified memory.

Choose Quantization #

For local inference you’ll rarely run the model uncompressed. Instead, local models are usually compressed and their weights—and sometimes the KV cache—are stored at lower precision, allowing for a lower memory footprint at the cost of some accuracy. There are multiple quantization formats, the most popular being:

Most local models today are shared in the GGUF file format, which comes at several levels of compression:

• Q8 - in practice, barely noticeable performance impact
• Q6 - almost identical performance to the base model
• Q5 - slight reduction in reasoning ability
• Q4 - a good balance of performance and reasoning
• Q3 - compression effects start to noticeably affect reasoning
• Q2 - aggressive quantization, best used for edge cases, like on-device mobile

The general consensus in the community is to start with a GGUF Q4_K_M quantization, which is supported by providers like Atomic Chat, Ollama, LM Studio, llama.cpp, and others, if it fits.

Additionally, it is usually better to choose a bigger model at a more aggressive quantization level than a model that has fewer parameters but uses a less compressed quant. Consider this example:

In this example, the 14B model is quantized down to 4-bit weights, yet it still beats the smaller 8B model in every benchmark. It also requires substantially less space for its model weights: around 10 GB instead of 16.4 GB.

Evaluate the Model’s Capabilities #

Check whether the model supports the capabilities that you’ll need for your workflow:

• Structured output — lets the model return data in a fixed format such as JSON or a JSON schema.
• Vision — lets the model accept images alongside text.
• Audio — allows the model to accept or generate speech or other audio.
• Video — allows the model to process multiple frames or video directly.
• Context length — a model supporting 128K or 256K tokens will be less forgetful and you’ll need to compress context less frequently.
• Reasoning / thinking modes — chain of thought reasoning can improve performance significantly at the cost of higher token usage and longer task completion times.

Model Shortlist by Workload #

For specific model recommendations, we’ve made the following shortlist sorted by task and by hardware. OpenAI gpt-oss-20b has relatively modest memory requirements for its capability class. It has 21B total parameters, 3.6B active parameters, a 131,072-token context window, native MXFP4 weights, and an Apache 2.0 license. It was designed for reasoning and tool use as well as ordinary chat.

Qwen3.8-27B is a larger dense model that requires more fast memory or stronger quantization. Its features 262,144-token native context, an extension path to about 1M tokens, thinking mode, vision, coding, and agentic capabilities. The model scores 90.3 on LiveCodeBench v6 and 79.5 on IFBench.

Gemma 4 comes in several sizes and modality configurations. The official family includes 31B, 26B-A4B, 12B, E4B, and E2B variants, with 128K context on smaller variants and 256K on larger ones. On the MMLU-Pro benchmark, the models scores 85.2 for Gemma 4 31B. It also features multilingual and multimodal support.

Devstral Small 2505 is a 24B dense, text-only model with 128K context, built for software-engineering agents. The model can run on an RTX 4090 or a Mac with 32GB RAM.

Qwen3.8-27B is a great general-purpose AI model with vision support, making it useful for any tasks where visual input is important, such as UI design and front-end coding.

Qwen3.6-35B-A3B is an MoE model for agentic coding. It activates only 3B of its 35B parameters per token and documents a 262K native context, with an extension method to roughly 1.01M.

gpt-oss-20b is the only open weight model from OpenAI that can run on consumer hardware. It’s a good choice if you want to run an offline model that’s tuned similarly to official ChatGPT.

Best AI Models for RAG

RAG models need to be able to retrieve the right evidence from a large corpus of data, making it crucial that the model maximizes usable context length. For RAG, consider the following options:

- Qwen3.8-27B
- Qwen3.6-35B-A3B
- gpt-oss

• Devstral
• Gemma 4

Best AI Models With Vision Support

For vision, consider models from the following families:

• Qwen3.8
• Qwen3.6
• Gemma 4

Model Shortlist by Hardware Tier #

Where to Run These Models #

Atomic Chat is a free local AI chat app for Mac, Windows, Linux, iPhone, and Android that allows you to select and run 1,000+ open models from Hugging Face — including the ones we’ve reviewed above.

Atomic Chat runs inference locally on-device, so nothing you type leaves your machine, and there are no rate limits or message caps. It features Google’s TurboQuant compression, which shrinks models down to 3 bits with zero accuracy loss and compresses the KV cache by at least 6× — allowing you to fit larger models into memory than would otherwise be possible, while also computing attention up to 8× faster than standard 32-bit models. Alongside integrated chat, Atomic Chat works with the Kilo agent out of the box.