LLM Classification Is Feature Engineering A blog post on Minimally Sufficient argues that LLM classification should be treated as feature engineering, wrapping an LLM's hard label verdict in a logistic regression of the form p(y = 1 | x) = σ(α + β · LLM(x)) rather than using the LLM's output directly as a classifier. The author contends that this approach restores calibration and threshold control, allows structured covariates and population baselines to be incorporated, and improves interpretability, since the special case of β → ∞ recovers the original LLM classifier. The post attributes the shortcomings of LLMs-as-classifiers to how they are used, not to the models themselves, which are not designed as classifiers. LLMs-as-classifiers, prompts applied to a context and returning a label, suck to work with. This is especially painful because they often perform pretty decently. But let’s consider some of the things we’d want in a classifier and see how an LLM-as-classifier stacks up: - Calibration / Threshold Control - LLM verdicts are often hard labels; you can get token log probabilities but there is no mechanism for believing these to be well-calibrated https://minimallysufficient.com/posts/well-calibrated-predictions-are-not-enough/ . You can ask the LLM for its confidence and there’s no reason to suspect that to be well-calibrated either. As a related problem it’s then rather hard to trade off precision and recall with these labels in a principled way. - Incorporating all available information - LLMs work great with unstructured data but we often have nice structured data as well. We can paste this into the prompt however the LLM doesn’t really need to use it. Even for prose parts of the prompt we don’t know if the LLM actually used it or not