# Stop the Download-and-Crash Cycle: Choose the Right Local AI Model with llmfit > Source: > Published: 2026-09-16 22:01:01+00:00 Local AI has a discovery problem disguised as a hardware problem. The model catalog is expanding faster than most people can evaluate it. A single family may appear in several parameter sizes, multiple quantizations, different file formats, and runtime-specific variants. The model card tells you what was released. Your computer tells you how much RAM and VRAM it has. The difficult part is connecting the two. That gap produces a familiar cycle: The cost is not only bandwidth. Every failed experiment consumes time, disk space, and attention. [llmfit](https://github.com/AlexsJones/llmfit) is an open-source terminal tool built to put a pre-flight check in front of that workflow. It detects the machine, evaluates a catalog of models, chooses a feasible quantization, estimates speed and memory fit, and ranks the results before you download anything. The project has attracted more than 34,000 GitHub stars, which is a strong signal that this is a widely shared problem. [1] The important word, however, is **estimate**. llmfit can replace blind guessing with an inspectable model of your hardware. It cannot promise that every runtime, driver, model file, workload, and context length will behave exactly as predicted. Used with that distinction in mind, it is one of the most useful planning tools in the local-LLM stack. A model can fit on paper and still be a poor choice in practice. First, the weights need somewhere to live. On a discrete GPU, the ideal path is usually to keep them in VRAM. If they spill into system RAM, the model may still run through CPU/GPU offloading, but generation can slow dramatically. On Apple Silicon, CPU and GPU share a unified memory pool, so the calculation follows a different path. Second, quantization changes the memory-quality trade-off. A heavily compressed model may fit where a higher-precision variant does not, but “fits” does not mean “best.” The strongest quantization that fits with sensible headroom is often more useful than the smallest possible file. The underlying llama.cpp ecosystem itself distinguishes many encodings — from Q2 variants through Q8 — with different storage and quality characteristics. [2] Third, context consumes memory. The model’s weights are only part of the budget. During inference, the runtime stores key and value tensors for previous tokens in a KV cache. That cache grows with sequence length, so a model that works comfortably at 4K context may become marginal at 32K. [3] Finally, usable performance depends on the execution path. A technically runnable model generating two tokens per second is a very different experience from one generating 30. So the real question is not: *Can this model start?* It is: *Which model, quantization, context length, and runtime give me the best useful experience on this hardware?* That is the question llmfit tries to answer. Run llmfit with no arguments and it opens an interactive terminal interface. At startup, it detects RAM, CPU, GPU or accelerator, VRAM, and the available backend. Its documented hardware paths include NVIDIA through nvidia-smi, AMD through rocm-smi, Intel Arc, Apple Silicon unified memory, and Ascend devices. Support varies by platform; for example, Android GPU autodetection is not currently supported. [4] It then evaluates hundreds of model entries drawn from Hugging Face metadata and embedded into the llmfit release. For each candidate, it searches a quantization hierarchy, estimates memory requirements at the relevant context, selects a run mode, and calculates four scores: [5] Dimension What it is trying to representQualityModel family, scale, quantization penalty, and task alignmentSpeedEstimated token-generation throughput on the detected backendFitHow safely the model uses the available memoryContextHow the feasible context compares with the target use case Those dimensions feed a composite score. The weights change with the requested use case: coding, reasoning, chat, multimodal, embedding, or general use. In other words, a larger general model should not automatically outrank a smaller coding specialist when you ask for a coding recommendation. Fit is also more nuanced than a green or red light. llmfit classifies run paths such as full GPU, CPU/GPU offload, CPU-only, and Mixture-of-Experts offload, then labels the result from **Perfect** through **Good**, **Marginal**, and **Too Tight**. The terminology communicates an operational truth: starting a model and running it comfortably are not the same achievement. Autoregressive token generation often spends much of its time moving model weights through memory. llmfit uses this property to estimate throughput with a memory-bandwidth model. Its documented baseline is: ``` tokens per second ≈ (memory bandwidth / model size) × efficiency factor ``` The calculation is adjusted for quantization and execution modes such as pure GPU, CPU offload, MoE offload, and tensor parallelism. For recognized GPUs, llmfit uses a hardware bandwidth table; for unknown devices, it falls back to backend-level constants. [5] This produces a better starting point than parameter count alone, but it is still a model. Runtime implementation, kernel efficiency, prompt processing, thermal behavior, background memory use, driver versions, and architecture-specific details can move the real result. The project has responded to that limitation in a good way: it made the estimates inspectable and added measurement. llmfit info "" shows the assumptions behind a recommendation and provides commands to verify it. llmfit bench measures tokens per second and time to first token against a running Ollama, vLLM, MLX, or llama.cpp server. A local measurement takes priority over the formula, and users can optionally contribute benchmark results through a GitHub pull request. Merged measurements are embedded in later releases, creating an auditable community data set rather than opaque telemetry. [6] That creates a sensible progression: ``` estimate → download → run → benchmark → calibrate ``` llmfit is most valuable at the beginning of that chain. The benchmark closes the loop. Mixture-of-Experts models are where simple calculators often become misleading. An MoE model may contain many experts while routing each token through only a subset. Active parameter count therefore matters for compute and potential throughput. It does **not** mean the inactive weights disappear. llmfit models an expert-offload path in which active experts occupy VRAM while inactive experts remain in system RAM. This can make a large MoE model feasible on a machine that cannot hold every expert in VRAM, but the machine still needs enough combined memory, and the slower memory path can affect performance. [5] This nuance matters because earlier llmfit versions did overstate fit for some MoE models by leaning too heavily on active parameters. Users reported cases where recommendations could not actually load, and the project corrected related paths over several releases. Version 1.1.11, released on August 25, 2026, changed fallback planning to use active MoE parameters; the current run-path logic separately checks whether inactive expert weights fit in system RAM. The same release also corrected hybrid-attention KV-cache calculations. [7] The lesson is not that estimation tools are useless. It is that their assumptions must be visible, their defects must be correctable, and their outputs should not be treated as proof. For an MoE recommendation, inspect both VRAM and system-RAM requirements. Then benchmark the exact model file, quantization, runtime, and context you intend to use. The official project currently offers several installation paths. [1] On macOS or Linux with Homebrew: ``` brew install llmfit ``` On Windows with Scoop: ``` scoop install llmfit ``` With Python tooling: ``` uv tool install -U llmfit ``` Or use the project’s installation script on macOS or Linux: ``` curl -fsSL https://llmfit.axjns.dev/install.sh | sh ``` As always, inspect remote installation scripts before piping them into a shell if your environment requires that level of control. Then begin with three commands: ``` llmfit systemllmfit fit --perfect -n 5llmfit recommend --json --use-case coding --limit 5 ``` The first verifies what the tool detected. This step is easy to skip and important not to skip. If GPU memory or unified memory is wrong, every downstream recommendation inherits the error. The second produces a small list of models with the strongest fit. The third provides machine-readable recommendations for a coding workload. If autodetection is unreliable — inside a VM, for example — or if you are evaluating hardware you do not own yet, override the inputs: ``` llmfit --memory=24G --ram=64G --cpu-cores=16 fit ``` In the TUI, the same idea appears as hardware simulation. You can change RAM, VRAM, and CPU cores and immediately recalculate the table. The normal workflow asks, “What runs on my machine?” The plan command reverses it: ``` llmfit plan "Qwen/Qwen3-4B-MLX-4bit" \ --context 8192 \ --target-tps 25 ``` The report estimates minimum and recommended resources, feasible GPU and offload paths, and the upgrade delta between the current machine and the target. JSON output makes the result usable in procurement notebooks, deployment checks, or internal tooling. [8] This is arguably llmfit’s most interesting professional use. Hardware planning normally begins with a vague request — “We need to run a good coding model locally” — and quickly turns into a spreadsheet of model sizes, GPU prices, context assumptions, and hoped-for throughput. plan gives that conversation a reproducible starting point. It is not a purchase guarantee. Before spending serious money, validate the plan with benchmarks from the same accelerator class and runtime. But it is far better than choosing a GPU from model parameter count alone. llmfit is useful interactively, but it is designed to be automated. llmfit recommend --json can act as a pre-deployment gate. A script can reject a model assignment when the node reports insufficient memory, select only models rated Good or better, or constrain results to the runtime already installed on the fleet. For a node-level service, llmfit serve exposes the same analysis through a REST API. The documented endpoints return detected hardware, filtered model lists, and the best runnable models for a node. The project recommends conservative scheduler defaults such as excluding Too Tight results and requiring at least a Good fit. It also provides an MCP server mode for agent integrations. [8] The architecture is deliberately local: each node reports its own hardware and fit analysis; an external scheduler or controller makes the placement decision. That does not make llmfit a full orchestrator. It does not replace Kubernetes scheduling, runtime health checks, load testing, or service-level objectives. It supplies one useful signal: **given the hardware and the model assumptions, is this placement plausible?** I would trust llmfit to narrow a catalog, expose obvious memory mismatches, compare quantization options, and turn hardware planning into a repeatable process. I would still verify: Also keep llmfit current. Its model catalog is embedded at build time, so catalog and estimation improvements arrive when you upgrade the tool. The changelog shows an active project with frequent fixes to hardware detection, provider mapping, MoE estimates, and KV-cache calculations. [7] The old workflow starts with a model name and hopes the hardware cooperates. A more disciplined workflow starts with constraints: This does not turn local inference into a deterministic science. Too many layers still matter: model metadata, quantization, runtime, drivers, memory pressure, prompt shape, and the workload itself. What it does is move the uncertainty to the right place. Instead of spending hours discovering that a model cannot load, you spend seconds generating a defensible shortlist. Instead of treating estimated tokens per second as truth, you use it as a hypothesis and replace it with a measurement. Instead of buying hardware from intuition, you state the model, context, and throughput target explicitly. llmfit’s real value is not that it always knows the answer. It is that it helps you ask the complete question before the download begins. *Research note: This article was fact-checked on August 27, 2026, against llmfit documentation and changelog through version 1.1.11. The project and its model catalog evolve quickly; commands and estimates may change in later releases.* [Stop the Download-and-Crash Cycle: Choose the Right Local AI Model with llmfit](https://pub.towardsai.net/stop-the-download-and-crash-cycle-choose-the-right-local-ai-model-with-llmfit-2a769392c013) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.