llmfit does the local-LLM math you've been faking

Llmfit, a Rust CLI and TUI created by Alex Jones, the developer behind k8sgpt, right-sizes local LLMs to a user's hardware by detecting RAM, CPU, and GPU and scoring hundreds of models on quality, speed, fit, and context. The tool, which has collected over 30,000 GitHub stars, walks quantization levels from Q8_0 to Q2_K, models multi-GPU setups and MoE expert offloading, and integrates with Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio. Its `recommend --json` command enables machine-readable output for provisioning scripts, and the `plan` command inverts the question to specify hardware requirements for a given model and context length.

AI https://sourcefeed.dev/c/ai Article llmfit does the local-LLM math you've been faking The k8sgpt creator's Rust CLI right-sizes models to your GPU — keeping its catalog honest is the hard part. Mariana Souza https://sourcefeed.dev/u/mariana souza Every local-LLM forum has the same recurring thread: "I have a 4070 Ti and 32GB of RAM — what can I run?" The answers are folklore. Somebody quotes a rule of thumb about gigabytes per billion parameters, somebody else corrects them for quantization, a third person points out that nobody accounted for the KV cache at 32k context, and the original poster downloads a 40GB GGUF that dies at load time anyway. llmfit https://github.com/AlexsJones/llmfit exists to end that thread. It's a Rust CLI and TUI from Alex Jones — the developer behind k8sgpt https://k8sgpt.ai/ , the CNCF Kubernetes diagnostics tool — that detects your RAM, CPU, and GPU, then scores hundreds of models on quality, speed, fit, and context to tell you what will actually run, and roughly how fast. It has collected over 30,000 GitHub stars in short order, which is a remarkable number for a tool whose core job is arithmetic. That's not a dig. The arithmetic is genuinely fiddly, everybody has been doing it by hand, and llmfit does it better than the folklore does. But the star count says more about the size of the pain than the depth of the moat — and the moat question is where this gets interesting. The arithmetic you've been faking What makes model-fitting hard isn't any single number, it's the interactions. Quantization changes memory footprint and quality and throughput. Context length inflates the KV cache, which competes with weights for the same VRAM. Mixture-of-experts models like DeepSeek's break the naive parameter-count math entirely, because you can offload inactive experts. And the same model behaves differently on CUDA, Metal, and ROCm. llmfit's design takes each of these seriously. Instead of assuming a fixed quantization, it walks the ladder from Q8 0 down to Q2 K and picks the highest-quality quant that fits your memory. It models multi-GPU setups and MoE expert offloading. Hardware detection goes through vendor paths — nvidia-smi , rocm-smi , system profiler on Macs — rather than guessing from OS totals. Speed estimates come from a memory-bandwidth model, which is the right first-order physics for token generation, and the info command shows the assumptions behind each estimate so you can check its work. It also plugs into the runtimes people actually use — Ollama https://ollama.com/ , llama.cpp https://github.com/ggml-org/llama.cpp , MLX, Docker Model Runner, and LM Studio https://lmstudio.ai/ — so a recommendation is one step from a running model, not a shopping list. Where it earns a place in your workflow The interactive TUI is the demo, but the useful surface is the plain CLI. A sensible first session looks like: brew install llmfit scoop on Windows; script, Docker, or cargo elsewhere llmfit doctor verify it sees your GPU correctly llmfit fit ranked table of what runs on this box llmfit recommend --json top picks, machine-readable That last command is the one the "this should be a website" crowd on Hacker News missed. A web form can tell a human what to download once. recommend --json can be piped into provisioning scripts, dev-container setup, or an agent that bootstraps its own local model — pick the best model this machine supports, pull it via Ollama, go. As more tooling assumes a local model is available, "query the hardware, choose the model" becomes a build step, and build steps want a binary, not a browser tab. The sleeper feature is plan , which inverts the question: given a model and a context length, what hardware do I need? If you're speccing a Mac Studio or arguing with finance about a GPU order, llmfit plan "Qwen/Qwen3-4B-MLX-4bit" --context 8192 is a better basis for the conversation than a Reddit thread. The database is the product, and databases rot Now the skepticism, because the 301-point Hacker News thread supplied plenty. Users with an M4 MacBook Pro and 128GB of RAM reported recommendations topped by dated models. Another said llmfit claimed their machine couldn't run a model it was running at that moment. Neither report means the math is wrong — they mean the catalog was, and that's the structural risk. llmfit ships an embedded model database great for offline speed, zero startup latency , but the local-model meta shifts monthly, and a newer 8B routinely beats last year's 30B. A fit calculator with a stale catalog gives you confidently precise answers to the wrong question. The field is also crowded and getting commoditized. VRAM-calculator websites are a genre unto themselves, a near-clone called whichllm hit the Hacker News front page within months, and LM Studio already flags model compatibility inline for its users. The fit formula is not defensible; anyone can reimplement it. What is potentially defensible is measurement. llmfit's bench command records real tokens-per-second on your hardware, and the project solicits those numbers back to ground its estimates in community data from identical configurations. That's the k8sgpt playbook — turn a diagnostic tool into a feedback loop — and it's the difference between a calculator and something like a Geekbench for local inference. Estimates decay; a corpus of measured results on real hardware compounds. Verdict Install it. It's a two-minute setup that replaces an error-prone ritual, and the JSON output slots into automation in a way none of the web calculators can. But treat its output as a strong prior, not a verdict: sanity-check the model list against what's actually current, and use info to see the assumptions before you trust a speed number. Whether llmfit is still the answer in a year depends entirely on the parts that aren't arithmetic — how fast the catalog tracks new releases, and whether the community benchmark loop reaches critical mass. The math was never the hard part. Keeping the answers true is. Sources & further reading - AlexsJones/llmfit https://github.com/AlexsJones/llmfit — github.com - llmfit - Right-sizes LLM models to your system's RAM, CPU, and GPU https://www.llmfit.org/ — llmfit.org - Right-sizes LLM models to your system's RAM, CPU, and GPU https://news.ycombinator.com/item?id=47211830 — news.ycombinator.com - Stop guessing. Start running. llmfit picks the right LLM for your hardware. https://www.ajeetraina.com/stop-guessing-start-running-llmfit-picks-the-right-llm-for-your-hardware/ — ajeetraina.com Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.