Most developers copy temperature=0.7 from a tutorial and never touch it again. That’s not engineering. That’s guessing.
Every LLM API call you make includes sampling parameters. Temperature, top-p, max tokens, penalties. These control how the model picks every single word in its response. Get them wrong and your chatbot sounds like a drunk poet. Get them right and outputs feel exactly tailored to the task.
The problem is most developers never touch these settings. They copy temperature=0.7 from whatever tutorial they followed and move on. I did this for months. Then I spent an entire weekend debugging why my extraction pipeline was returning different results for identical inputs. Turned out the fix was changing one number.
When a model generates text, it doesn’t just pick the best next word. It calculates a probability score for every token in its vocabulary. That’s over 100,000 candidates at every single step.
Temperature controls how those probabilities are distributed. Here’s the intuition:
import numpy as npdef apply_temperature(logits, temperature): scaled = logits / temperature probs = np.exp(scaled) / np.sum(np.exp(scaled)) return probslogits = np.array([5.0, 3.0, 1.0, 0.5])tokens = ["Paris", "London", "Berlin", "Tokyo"]# Temperature = 0.1 (very focused)probs_low = apply_temperature(logits, 0.1)# Paris: 99.9%, London: 0.1%, Berlin: 0%, Tokyo: 0%# Temperature = 1.0 (default)probs_default = apply_temperature(logits, 1.0)# Paris: 72%, London: 19%, Berlin: 5%, Tokyo: 3%# Temperature = 2.0 (very creative)probs_high = apply_temperature(logits, 2.0)# Paris: 45%, London: 27%, Berlin: 15%, Tokyo: 13%
At low temperature, the model almost always picks the highest-probability token. At high temperature, lower-probability tokens get a real chance. That’s why temperature=0 gives you deterministic, predictable output and temperature=1.5 gives you creative chaos.
Top-p (nucleus sampling) works differently. Instead of scaling probabilities, it dynamically limits which tokens are even considered.
With top_p=0.9, the model sorts all tokens by probability, then only samples from the smallest set of tokens whose probabilities add up to 90%. Everything else gets eliminated.
Here’s why that’s powerful. When the model is confident (one token has 85% probability), top-p=0.9 restricts sampling to just one or two tokens. When the model is uncertain (dozens of tokens each at 2%), top-p=0.9 includes all of them. It adapts to the model’s own confidence automatically. Temperature can’t do that.
def apply_top_p(probs, tokens, p=0.9): sorted_indices = np.argsort(probs)[::-1] cumulative = 0 selected = [] for idx in sorted_indices: cumulative += probs[idx] selected.append((tokens[idx], probs[idx])) if cumulative >= p: break return selected# Model is confident: "The capital of France is ___"# top_p=0.9 selects: [("Paris", 0.95)]# Only one option survives. Deterministic.# Model is uncertain: "My favorite color is ___"# top_p=0.9 selects: [("blue", 0.25), ("red", 0.22), # ("green", 0.20), ("purple", 0.15)]# Multiple options survive. Creative.
The critical rule from OpenAI’s own documentation: adjust either temperature or top-p, but not both at the same time. Changing both makes it nearly impossible to isolate which parameter is causing a specific behavior.
After building a dozen LLM applications, here are the exact settings I use for each task type:
Classification and extraction: temperature=0, top_p=1.0. You want the single most confident answer every time. Any randomness introduces inconsistency.
Customer-facing chatbot: temperature=0.7, top_p=0.9. Enough variation to feel natural. Constrained enough to stay on topic.
Creative writing and brainstorming: temperature=1.0, top_p=0.95. Let the model explore. Surprising outputs are the point.
Code generation: temperature=0.2, top_p=0.95. You want correct code, not creative code. Low temperature keeps syntax tight.
Four months ago I built an invoice data extractor. Feed it a PDF, get back structured JSON with vendor name, amount, date, and line items. It worked beautifully in testing. Then in production, the same invoice would sometimes return “Total Amount: $1,234.56” and other times “Total Amount: $1234.56”. Same input. Different formatting. My validation layer rejected half the responses because it expected a consistent format. I spent all of Saturday checking my prompt, my parsing logic, my JSON schema instructions. Sunday morning I finally looked at my API call parameters. Temperature was set to the default 1.0. I changed it to 0. The formatting became perfectly consistent. Every single time. One number. Two days wasted.
In 2026, the practical advice has converged. For commercial APIs like OpenAI and Anthropic, use temperature plus top-p. For open-source deployments with frameworks like vLLM or llama.cpp, the community has moved toward temperature plus min-p (a newer sampler that scales with the model’s own certainty). Either way, you need exactly two knobs. Not five. Not zero.
Tomorrow in Day 5, we’re covering system prompts. The invisible architecture that defines how your entire LLM application behaves. Most developers treat them as an afterthought. That’s a mistake with expensive consequences.
What temperature setting do you default to, and have you ever actually tested whether it’s the right one? Be honest in the comments.
🔥 This is Day 4 of my “21 Days of Building with LLMs” series.
If you’re learning along with me, follow so you don’t miss the next one.
👉 [Follow me here] for daily posts on AI, Python, Data Engineering, and SQL.
Missed a day? Start from Day 1 → [link]
— Eswar
21 Days of LLMs, Day 4: Temperature, Top-P, and the Parameters You’re Ignoring was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.