Before You Trust AI, Understand Sampling

Large language models produce varying outputs due to sampling techniques like temperature, top-k, and top-p, which introduce controlled randomness as a feature rather than a flaw. Understanding these mechanisms is essential for trusting AI in critical applications.

Member-only story Before You Trust AI, Understand Sampling Why uncertainty is a feature of LLMs, not a bug Why do LLMs sometimes produce brilliant answers and sometimes complete nonsense? Learn how sampling, temperature, top-k, and top-p shape AI behavior and reliability. Not a Medium member? You can read this article for free on my blog here: TNKRD Tech Blog. https://blog.tnkrd.com/why-llms-are-nondeterministic/ Anyone who regularly delegates work to AI knows the uncomfortable part: you can never be sure what you are going to get. Sometimes the result is brilliant. Other times, it is merely convincing. And every so often, it is completely useless. Even when asking the same simple question twice, you may end up with two noticeably different answers. Critics see this as evidence that AI is unpredictable and therefore unsuitable for critical work. Enthusiasts argue that AI is no different from people: ask anyone to solve a problem, and you can never be sure which path they will choose. While this comparison may seem convincing at first, it doesn’t fully explain why AI can sometimes appear broken or immature. People build habits. Once we find an approach that works, we tend to reuse it. We rarely reinvent our entire method every time we repeat the same task. That’s why this kind of AI behavior seems flawed to us.