{"slug": "llama-modes-load-one-gguf-once-then-chat-boolean-choice-and-scale", "title": "Llama-modes: load one GGUF once, then Chat, Boolean, Choice and Scale", "summary": "A developer released llama-modes, a fork of llama.cpp that loads one GGUF model once and reuses the same in-memory weights for Chat, Boolean, Choice and Scale inference modes without a second classifier, model swap or fine-tune. The project scores candidates via next-token logits for single-token choices and teacher-forced sequence log-likelihood (log P(candidate | prompt)) for multi-token choices, and its Scale mode returns a full discrete distribution over a supplied 0–10 scale with mode, median, quantiles and, for interval scales, expected value and weighted spread. The repository ships a Windows CUDA release, a React demo, Python/PowerShell/curl examples, a cookbook, API documentation and a reproducible Direct-vs-Chat benchmark harness, tested so far with GPT-OSS 20B MXFP4 and Qwen3.8 Ridge on Windows with NVIDIA CUDA; the author notes returned weights are not calibrated confidence and that the first request after loading is slower due to warm-up.", "body_md": "I wanted to explore a simple question:\n\n*can one ordinary local LLM, loaded once, be used as more than a text generator without introducing a second classifier or swapping models?*\n\nThat experiment became llama-modes, a fork of llama.cpp.\n\nThe core idea is that you load one GGUF model once, keep the same weights in memory, and use that same loaded model for several different inference modes.\n\nNormal llama-server chat remains available, but the same model can also perform:\n\nThere is no second classifier, no model swapping, and no special fine-tune required.\n\nCandidate scoring itself is obviously not new. People already use next-token logits or sequence likelihoods for classification, ranking, and multiple-choice evaluation.\n\nWhat I wanted to explore was turning that idea into a reusable llama.cpp runtime primitive rather than rebuilding the logic separately in every application.\n\nIn other words: same model, same weights in memory, different inference primitive.\n\nFor single-token choices, llama-modes can work directly from the model output at the prepared evaluation state.\n\nFor multi-token choices, it uses teacher-forced sequence log-likelihood:\n\nlog P(candidate | prompt) rather than pretending that an entire multi-token candidate has a single logit.\n\nThe SCALE mode is probably the easiest part to demonstrate visually.\n\nInstead of asking a model to generate something like:\n\n9/10 you can supply an ordered 0–10 scale and get back the entire discrete distribution across those supplied points.\n\nFor ordinal scales, llama-modes returns things such as mode, median and quantiles.\n\nFor interval scales, where the caller explicitly asserts that numeric distances have meaning, it can also return expected value and weighted spread.\n\nThe model is not generating those summary statistics. They are derived from the returned distribution.\n\nI originally became interested in this direction after the recent discussion around Jev and decision-first inference, but llama-modes is not a Jev reimplementation and does not claim Jev-style calibration.\n\nIt takes a different route: exposing structured scoring directly from ordinary local GGUF language models.\n\nSo far I have tested it with GPT-OSS 20B MXFP4 and Qwen3.8 Ridge on Windows with NVIDIA CUDA.\n\nThe repository now includes a Windows CUDA release, a local React demo, Python / PowerShell / curl examples, a cookbook, API documentation, and a reproducible Direct-vs-Chat benchmark harness.\n\nRepo:\n\nRelease:\n\nOne practical note if you try the demo:\n\nthe first request after loading the model can be noticeably slower because of warm-up.\n\nRun 2–3 requests before judging interactive latency. The Direct-vs-Chat screen in the React UI is meant as an interactive demonstration, not as the benchmark itself; the repository contains a separate harness for reproducible measurements.\n\nAlso, the returned candidate/scale weights should not be interpreted as calibrated confidence. They are relative to the supplied alternatives and their representations, and label/tokenization choices can matter.\n\nOne thing I’d especially like feedback on is whether this kind of direct structured inference is useful in real applications, and which model families behave well or badly with it.\n\nThe current roadmap item is shared-context multi-question evaluation: one context evaluation, then multiple Boolean / Choice / Scale questions over it.", "url": "https://wpnews.pro/news/llama-modes-load-one-gguf-once-then-chat-boolean-choice-and-scale", "canonical_source": "https://discuss.huggingface.co/t/llama-modes-load-one-gguf-once-then-chat-boolean-choice-and-scale/180762#post_1", "published_at": "2026-09-27 17:40:40+00:00", "updated_at": "2026-09-27 17:59:55.831007+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["llama-modes", "llama.cpp", "GGUF", "GPT-OSS 20B MXFP4", "Qwen3.8 Ridge", "NVIDIA CUDA", "Jev", "Windows"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llama-modes-load-one-gguf-once-then-chat-boolean-choice-and-scale", "markdown": "https://wpnews.pro/news/llama-modes-load-one-gguf-once-then-chat-boolean-choice-and-scale.md", "text": "https://wpnews.pro/news/llama-modes-load-one-gguf-once-then-chat-boolean-choice-and-scale.txt", "jsonld": "https://wpnews.pro/news/llama-modes-load-one-gguf-once-then-chat-boolean-choice-and-scale.jsonld"}}