AIArticle
The k8sgpt creator's Rust CLI right-sizes models to your GPU — keeping its catalog honest is the hard part.
Every local-LLM forum has the same recurring thread: "I have a 4070 Ti and 32GB of RAM — what can I run?" The answers are folklore. Somebody quotes a rule of thumb about gigabytes per billion parameters, somebody else corrects them for quantization, a third person points out that nobody accounted for the KV cache at 32k context, and the original poster downloads a 40GB GGUF that dies at load time anyway.
llmfit exists to end that thread. It's a Rust CLI and TUI from Alex Jones — the developer behind k8sgpt, the CNCF Kubernetes diagnostics tool — that detects your RAM, CPU, and GPU, then scores hundreds of models on quality, speed, fit, and context to tell you what will actually run, and roughly how fast. It has collected over 30,000 GitHub stars in short order, which is a remarkable number for a tool whose core job is arithmetic.
That's not a dig. The arithmetic is genuinely fiddly, everybody has been doing it by hand, and llmfit does it better than the folklore does. But the star count says more about the size of the pain than the depth of the moat — and the moat question is where this gets interesting.
The arithmetic you've been faking #
What makes model-fitting hard isn't any single number, it's the interactions. Quantization changes memory footprint and quality and throughput. Context length inflates the KV cache, which competes with weights for the same VRAM. Mixture-of-experts models like DeepSeek's break the naive parameter-count math entirely, because you can offload inactive experts. And the same model behaves differently on CUDA, Metal, and ROCm.
llmfit's design takes each of these seriously. Instead of assuming a fixed quantization, it walks the ladder from Q8_0 down to Q2_K and picks the highest-quality quant that fits your memory. It models multi-GPU setups and MoE expert off. Hardware detection goes through vendor paths — nvidia-smi
, rocm-smi
, system_profiler
on Macs — rather than guessing from OS totals. Speed estimates come from a memory-bandwidth model, which is the right first-order physics for token generation, and the info
command shows the assumptions behind each estimate so you can check its work.
It also plugs into the runtimes people actually use — Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio — so a recommendation is one step from a running model, not a shopping list.
Where it earns a place in your workflow #
The interactive TUI is the demo, but the useful surface is the plain CLI. A sensible first session looks like:
brew install llmfit # scoop on Windows; script, Docker, or cargo elsewhere
llmfit doctor # verify it sees your GPU correctly
llmfit fit # ranked table of what runs on this box
llmfit recommend --json # top picks, machine-readable
That last command is the one the "this should be a website" crowd on Hacker News missed. A web form can tell a human what to download once. recommend --json
can be piped into provisioning scripts, dev-container setup, or an agent that bootstraps its own local model — pick the best model this machine supports, pull it via Ollama, go. As more tooling assumes a local model is available, "query the hardware, choose the model" becomes a build step, and build steps want a binary, not a browser tab.
The sleeper feature is plan
, which inverts the question: given a model and a context length, what hardware do I need? If you're speccing a Mac Studio or arguing with finance about a GPU order, llmfit plan "Qwen/Qwen3-4B-MLX-4bit" --context 8192
is a better basis for the conversation than a Reddit thread.
The database is the product, and databases rot #
Now the skepticism, because the 301-point Hacker News thread supplied plenty. Users with an M4 MacBook Pro and 128GB of RAM reported recommendations topped by dated models. Another said llmfit claimed their machine couldn't run a model it was running at that moment. Neither report means the math is wrong — they mean the catalog was, and that's the structural risk. llmfit ships an embedded model database (great for offline speed, zero startup latency), but the local-model meta shifts monthly, and a newer 8B routinely beats last year's 30B. A fit calculator with a stale catalog gives you confidently precise answers to the wrong question.
The field is also crowded and getting commoditized. VRAM-calculator websites are a genre unto themselves, a near-clone called whichllm hit the Hacker News front page within months, and LM Studio already flags model compatibility inline for its users. The fit formula is not defensible; anyone can reimplement it.
What is potentially defensible is measurement. llmfit's bench
command records real tokens-per-second on your hardware, and the project solicits those numbers back to ground its estimates in community data from identical configurations. That's the k8sgpt playbook — turn a diagnostic tool into a feedback loop — and it's the difference between a calculator and something like a Geekbench for local inference. Estimates decay; a corpus of measured results on real hardware compounds.
Verdict #
Install it. It's a two-minute setup that replaces an error-prone ritual, and the JSON output slots into automation in a way none of the web calculators can. But treat its output as a strong prior, not a verdict: sanity-check the model list against what's actually current, and use info
to see the assumptions before you trust a speed number.
Whether llmfit is still the answer in a year depends entirely on the parts that aren't arithmetic — how fast the catalog tracks new releases, and whether the community benchmark loop reaches critical mass. The math was never the hard part. Keeping the answers true is.
Sources & further reading #
AlexsJones/llmfit— github.com - llmfit - Right-sizes LLM models to your system's RAM, CPU, and GPU— llmfit.org - Right-sizes LLM models to your system's RAM, CPU, and GPU— news.ycombinator.com - Stop guessing. Start running. llmfit picks the right LLM for your hardware.— ajeetraina.com
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.