LLM guesses or knows

A developer tested two large language models to determine when they are guessing versus when they actually know the answer by analyzing logprobs. The Nano model showed miscalibrated confidence, assigning 87% probability to '4' for 'what is 2+2' but also collapsing entropy when copying input, while the Mini model displayed honest uncertainty with no candidate above 31% for a story prompt. The developer concludes that Nano is not recommended for agentic autonomous decision making.

We wanted to know when LLM is guessing versus when it actually knows the answer. LLM models expose logprobs - after every word they generate, you can request the top alternative tokens and their probabilities. Low entropy means the model was certain, high means it was guessing. I tested two models using 0.0 temperature and 3 prompts: "what is 2+2", "the opposite of hot is", "once upon a time." The results: For the math prompt, Nano gave "4" at 87% and "2" at 13%. For the story prompt, it echoed the input back verbatim, so entropy collapsed because it was copying input, not generating. Mini generated the story with five competing candidates, none above 31%. That's honest uncertainty. Both LLM models are cheap and still popular, but Nano is miscalibrated and not recommended for agentic autonomous decision making.