{"slug": "llmfit-does-the-local-llm-math-you-ve-been-faking", "title": "llmfit does the local-LLM math you've been faking", "summary": "Llmfit, a Rust CLI and TUI created by Alex Jones, the developer behind k8sgpt, right-sizes local LLMs to a user's hardware by detecting RAM, CPU, and GPU and scoring hundreds of models on quality, speed, fit, and context. The tool, which has collected over 30,000 GitHub stars, walks quantization levels from Q8_0 to Q2_K, models multi-GPU setups and MoE expert offloading, and integrates with Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio. Its `recommend --json` command enables machine-readable output for provisioning scripts, and the `plan` command inverts the question to specify hardware requirements for a given model and context length.", "body_md": "[AI](https://sourcefeed.dev/c/ai)Article\n\n# llmfit does the local-LLM math you've been faking\n\nThe k8sgpt creator's Rust CLI right-sizes models to your GPU — keeping its catalog honest is the hard part.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\nEvery local-LLM forum has the same recurring thread: \"I have a 4070 Ti and 32GB of RAM — what can I run?\" The answers are folklore. Somebody quotes a rule of thumb about gigabytes per billion parameters, somebody else corrects them for quantization, a third person points out that nobody accounted for the KV cache at 32k context, and the original poster downloads a 40GB GGUF that dies at load time anyway.\n\n[llmfit](https://github.com/AlexsJones/llmfit) exists to end that thread. It's a Rust CLI and TUI from Alex Jones — the developer behind [k8sgpt](https://k8sgpt.ai/), the CNCF Kubernetes diagnostics tool — that detects your RAM, CPU, and GPU, then scores hundreds of models on quality, speed, fit, and context to tell you what will actually run, and roughly how fast. It has collected over 30,000 GitHub stars in short order, which is a remarkable number for a tool whose core job is arithmetic.\n\nThat's not a dig. The arithmetic is genuinely fiddly, everybody has been doing it by hand, and llmfit does it better than the folklore does. But the star count says more about the size of the pain than the depth of the moat — and the moat question is where this gets interesting.\n\n## The arithmetic you've been faking\n\nWhat makes model-fitting hard isn't any single number, it's the interactions. Quantization changes memory footprint *and* quality *and* throughput. Context length inflates the KV cache, which competes with weights for the same VRAM. Mixture-of-experts models like DeepSeek's break the naive parameter-count math entirely, because you can offload inactive experts. And the same model behaves differently on CUDA, Metal, and ROCm.\n\nllmfit's design takes each of these seriously. Instead of assuming a fixed quantization, it walks the ladder from Q8_0 down to Q2_K and picks the highest-quality quant that fits your memory. It models multi-GPU setups and MoE expert offloading. Hardware detection goes through vendor paths — `nvidia-smi`\n\n, `rocm-smi`\n\n, `system_profiler`\n\non Macs — rather than guessing from OS totals. Speed estimates come from a memory-bandwidth model, which is the right first-order physics for token generation, and the `info`\n\ncommand shows the assumptions behind each estimate so you can check its work.\n\nIt also plugs into the runtimes people actually use — [Ollama](https://ollama.com/), [llama.cpp](https://github.com/ggml-org/llama.cpp), MLX, Docker Model Runner, and [LM Studio](https://lmstudio.ai/) — so a recommendation is one step from a running model, not a shopping list.\n\n## Where it earns a place in your workflow\n\nThe interactive TUI is the demo, but the useful surface is the plain CLI. A sensible first session looks like:\n\n```\nbrew install llmfit        # scoop on Windows; script, Docker, or cargo elsewhere\nllmfit doctor              # verify it sees your GPU correctly\nllmfit fit                 # ranked table of what runs on this box\nllmfit recommend --json    # top picks, machine-readable\n```\n\nThat last command is the one the \"this should be a website\" crowd on Hacker News missed. A web form can tell a human what to download once. `recommend --json`\n\ncan be piped into provisioning scripts, dev-container setup, or an agent that bootstraps its own local model — pick the best model this machine supports, pull it via Ollama, go. As more tooling assumes a local model is available, \"query the hardware, choose the model\" becomes a build step, and build steps want a binary, not a browser tab.\n\nThe sleeper feature is `plan`\n\n, which inverts the question: given a model and a context length, what hardware do I need? If you're speccing a Mac Studio or arguing with finance about a GPU order, `llmfit plan \"Qwen/Qwen3-4B-MLX-4bit\" --context 8192`\n\nis a better basis for the conversation than a Reddit thread.\n\n## The database is the product, and databases rot\n\nNow the skepticism, because the 301-point Hacker News thread supplied plenty. Users with an M4 MacBook Pro and 128GB of RAM reported recommendations topped by dated models. Another said llmfit claimed their machine couldn't run a model it was running at that moment. Neither report means the math is wrong — they mean the *catalog* was, and that's the structural risk. llmfit ships an embedded model database (great for offline speed, zero startup latency), but the local-model meta shifts monthly, and a newer 8B routinely beats last year's 30B. A fit calculator with a stale catalog gives you confidently precise answers to the wrong question.\n\nThe field is also crowded and getting commoditized. VRAM-calculator websites are a genre unto themselves, a near-clone called whichllm hit the Hacker News front page within months, and LM Studio already flags model compatibility inline for its users. The fit *formula* is not defensible; anyone can reimplement it.\n\nWhat is potentially defensible is measurement. llmfit's `bench`\n\ncommand records real tokens-per-second on your hardware, and the project solicits those numbers back to ground its estimates in community data from identical configurations. That's the k8sgpt playbook — turn a diagnostic tool into a feedback loop — and it's the difference between a calculator and something like a Geekbench for local inference. Estimates decay; a corpus of measured results on real hardware compounds.\n\n## Verdict\n\nInstall it. It's a two-minute setup that replaces an error-prone ritual, and the JSON output slots into automation in a way none of the web calculators can. But treat its output as a strong prior, not a verdict: sanity-check the model list against what's actually current, and use `info`\n\nto see the assumptions before you trust a speed number.\n\nWhether llmfit is still the answer in a year depends entirely on the parts that aren't arithmetic — how fast the catalog tracks new releases, and whether the community benchmark loop reaches critical mass. The math was never the hard part. Keeping the answers true is.\n\n## Sources & further reading\n\n-\n[AlexsJones/llmfit](https://github.com/AlexsJones/llmfit)— github.com -\n[llmfit - Right-sizes LLM models to your system's RAM, CPU, and GPU](https://www.llmfit.org/)— llmfit.org -\n[Right-sizes LLM models to your system's RAM, CPU, and GPU](https://news.ycombinator.com/item?id=47211830)— news.ycombinator.com -\n[Stop guessing. Start running. llmfit picks the right LLM for your hardware.](https://www.ajeetraina.com/stop-guessing-start-running-llmfit-picks-the-right-llm-for-your-hardware/)— ajeetraina.com\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/llmfit-does-the-local-llm-math-you-ve-been-faking", "canonical_source": "https://sourcefeed.dev/a/llmfit-does-the-local-llm-math-youve-been-faking", "published_at": "2026-08-17 19:08:37+00:00", "updated_at": "2026-08-17 19:11:14.186520+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["llmfit", "Alex Jones", "k8sgpt", "Ollama", "llama.cpp", "MLX", "Docker Model Runner", "LM Studio"], "alternates": {"html": "https://wpnews.pro/news/llmfit-does-the-local-llm-math-you-ve-been-faking", "markdown": "https://wpnews.pro/news/llmfit-does-the-local-llm-math-you-ve-been-faking.md", "text": "https://wpnews.pro/news/llmfit-does-the-local-llm-math-you-ve-been-faking.txt", "jsonld": "https://wpnews.pro/news/llmfit-does-the-local-llm-math-you-ve-been-faking.jsonld"}}