I wanted to explore a simple question:
can one ordinary local LLM, loaded once, be used as more than a text generator without introducing a second classifier or swapping models?
That experiment became llama-modes, a fork of llama.cpp.
The core idea is that you load one GGUF model once, keep the same weights in memory, and use that same loaded model for several different inference modes.
Normal llama-server chat remains available, but the same model can also perform:
There is no second classifier, no model swapping, and no special fine-tune required.
Candidate scoring itself is obviously not new. People already use next-token logits or sequence likelihoods for classification, ranking, and multiple-choice evaluation.
What I wanted to explore was turning that idea into a reusable llama.cpp runtime primitive rather than rebuilding the logic separately in every application.
In other words: same model, same weights in memory, different inference primitive.
For single-token choices, llama-modes can work directly from the model output at the prepared evaluation state.
For multi-token choices, it uses teacher-forced sequence log-likelihood:
log P(candidate | prompt) rather than pretending that an entire multi-token candidate has a single logit.
The SCALE mode is probably the easiest part to demonstrate visually.
Instead of asking a model to generate something like:
9/10 you can supply an ordered 0–10 scale and get back the entire discrete distribution across those supplied points.
For ordinal scales, llama-modes returns things such as mode, median and quantiles.
For interval scales, where the caller explicitly asserts that numeric distances have meaning, it can also return expected value and weighted spread.
The model is not generating those summary statistics. They are derived from the returned distribution.
I originally became interested in this direction after the recent discussion around Jev and decision-first inference, but llama-modes is not a Jev reimplementation and does not claim Jev-style calibration.
It takes a different route: exposing structured scoring directly from ordinary local GGUF language models.
So far I have tested it with GPT-OSS 20B MXFP4 and Qwen3.8 Ridge on Windows with NVIDIA CUDA.
The repository now includes a Windows CUDA release, a local React demo, Python / PowerShell / curl examples, a cookbook, API documentation, and a reproducible Direct-vs-Chat benchmark harness.
Repo:
Release:
One practical note if you try the demo:
the first request after the model can be noticeably slower because of warm-up.
Run 2–3 requests before judging interactive latency. The Direct-vs-Chat screen in the React UI is meant as an interactive demonstration, not as the benchmark itself; the repository contains a separate harness for reproducible measurements.
Also, the returned candidate/scale weights should not be interpreted as calibrated confidence. They are relative to the supplied alternatives and their representations, and label/tokenization choices can matter.
One thing I’d especially like feedback on is whether this kind of direct structured inference is useful in real applications, and which model families behave well or badly with it.
The current roadmap item is shared-context multi-question evaluation: one context evaluation, then multiple Boolean / Choice / Scale questions over it.