← All posts Deterministic AI explained for founders: why LLMs give different answers even at temperature 0 with a fixed seed, how rules, classifiers and language models differ, and the layers we use to make AI products testable and repeatable.
In September 2025, Thinking Machines sent one prompt - “Tell me about Richard Feynman” - to an open model 1,000 times at temperature 0, the setting that is supposed to make every answer identical. They got 80 different completions. All of them matched for 102 tokens. Then 992 said Feynman was born in “Queens, New York” and 8 said “New York City”.
Deterministic AI gives the same output every time it gets the same input. Rules, decision tables and trained classifiers with fixed weights work that way. So is AI deterministic? Some of it. Are LLMs deterministic? In practice, no - not even with temperature 0 and a fixed seed.
That stops being trivia the day your agent makes a decision a customer can see. I have been shipping production ML since 2017 and LLM products since 2023, and most founders I talk to find this out the same way: the demo worked, and then the same input produced a different answer in front of a client. This guide covers deterministic vs probabilistic systems, why language models drift, and the layers we use to build products that decide the same way twice.
What is deterministic AI? #
A system is deterministic when its output is fully decided by its input. Run it twice on the same data and you get the same answer, byte for byte. Nothing in the process is sampled.
Most of the AI that quietly runs businesses works this way:
- a rules engine that rejects an invoice above a limit;
- a fraud score from a gradient-boosted model with frozen weights;
- a ranking function that sorts candidates by a formula;
- a decision table the compliance team signed off.
Retrain the classifier and its behavior changes, but between releases it is a pure function: same features in, same score out. You can unit-test it, diff it between versions and explain to an auditor why it said no.
Generative models work differently. An LLM does not return an answer. It returns a probability for every possible next token and then picks one. Whether the pick is random depends on settings, and as you will see below, even the “not random” setting leaks. That is the whole deterministic vs non deterministic AI split: one computes a value, the other draws from a distribution.
The question “deterministic AI vs generative AI - which one do I need?” comes up on almost every first call. You rarely have to choose. Every serious product I have worked on uses both: the generative part reads messy input and writes text, the deterministic part makes the decisions that move money or data.
Deterministic vs probabilistic AI: what actually differs #
The useful comparison is not philosophical. It is operational: how each kind of system fails, how you test it, and what it costs you when something goes wrong.
| Deterministic | Probabilistic | |
|---|---|---|
| Output | One value per input | A sample from a distribution |
| How it fails | The same wrong answer every time | Different answers to the same input, mostly on borderline cases |
| Testing | Unit tests with exact expected values | Evaluation sets, agreement rates, thresholds |
| Audit | Replay the input, get the same decision | Replay gives a new decision unless you stored the original |
| Cost per decision | Fractions of a cent | A model call per decision, growing with prompt size |
| Good at | Limits, math, permissions, anything with one right answer | Reading unstructured input, judgment, language |
Think of it as a spectrum, not a switch. Rules sit at one end. A free-form chat completion sits at the other. In between are classic ML models, decision models that return typed answers with probabilities, and LLMs constrained to a fixed schema. Most of the engineering in an AI product is deciding where on that line each decision belongs.
Are LLMs deterministic? Not even at temperature 0 #
Temperature controls how a model picks the next token. At 1 it samples in proportion to the probabilities. At 0 it should always take the single most likely token, which in theory makes the output fixed. In practice several things break that.
Why are LLMs non deterministic with fixed settings
Batching on the provider’s servers. Your request is not computed alone. The inference server groups it with other users’ requests, and the batch size depends on load at that moment. The Thinking Machines study showed that common GPU kernels are not batch-invariant: the same prompt computed in a small batch and in a large one can produce slightly different numbers. When two candidate tokens are nearly tied, that tiny difference flips the choice, and everything after it diverges. That is what happened at token 103 in the Feynman test. Their batch-invariant kernels made all 1,000 completions identical, at the price of slower inference: 42 to 55 seconds instead of 26 for the same workload in their benchmark.
Seeds are best effort. OpenAI’s own guide describes the seed parameter as best-effort determinism. It tells you to watch system_fingerprint, which changes when the provider updates its serving stack, and promises only that outputs with the same seed and fingerprint will “mostly” match.
The model behind the name changes. A model alias points to whatever the provider serves today. Unless you pin a dated version, a silent update becomes a behavior change in your product.
Your inputs drift. Agents call tools. Search results, retrieved documents and timestamps differ between runs, so even a perfectly repeatable model would see a different prompt.
Researchers have measured how large the effect is. A 2024 study, Non-Determinism of “Deterministic” LLM Settings, ran the same benchmark tasks repeatedly at temperature 0: accuracy moved by up to 15% between runs, and the gap between the best and the worst possible run reached 70%.
I see the same in our own work. Fixed temperature, fixed seed, and agents still give different answers. Clear-cut inputs come back stable. Borderline ones do not: when the model is genuinely unsure, with two options close to each other, it changes its mind between runs, seed or no seed. Nondeterminism concentrates exactly where the decision is hardest.
So, are LLMs deterministic? Treat the answer as no, and design for it.
Where nondeterminism breaks a product #
For a lot of LLM work, variation is harmless. Two slightly different summaries of a sales call are both fine. Nobody notices when a draft email changes a word. It stops being harmless when the output is a decision:
- money - approve or decline, the price quoted, the refund issued;
- access - who sees which record, which action an agent may take;
- compliance - a claim flagged or not, a candidate screened out or not;
- anything you will have to explain later , to a customer, an auditor or a lawyer.
Here the failure is not that the model is wrong. It is that you cannot reproduce what it did. A customer complains, you rerun the same input, and the system gives the right answer. Your regression tests pass on Monday and fail on Tuesday with no code change. A provider ships a model update and you find out from support tickets.
So the useful question is not “is AI deterministic?” in general, but whether each decision in your product behaves the same way twice. That is the real reason to care about deterministic AI. Not purity - being able to test, explain and fix your own product.
How to get deterministic AI behavior from probabilistic models #
You do not make an LLM deterministic. The deterministic vs probabilistic AI choice is made per decision, not per product: you decide which decisions the model is allowed to make, and you surround it with parts that are. These are the layers we use, from most to least repeatable.
1. Put hard rules in code
Limits, permissions, eligibility, arithmetic. If a rule can be written as an if statement, it does not belong in a prompt. A prompt that says “never refund more than $500” is a request. A check in code is a guarantee. Let the LLM propose an action and let code approve or reject it.
2. Use a calibrated classifier where you have labels
If you have a few thousand labeled examples of a decision - route this ticket, flag this claim - a small trained model is usually cheaper, faster and fully repeatable. Then calibrate it. A raw classifier score is a number between 0 and 1, but it is not a probability: 0.8 does not mean the model is right 80% of the time. Calibration passes the score through a monotonic function fitted on held-out data, usually Platt scaling or isotonic regression, so that 0.8 does mean roughly 80%. Only calibrated scores are safe to put thresholds on. Training and calibrating these models is a routine part of our custom AI development work.
3. Consider a decision model
A new class of models does only this job. Jev, released by TypeSafe AI in September 2026, takes your application state and questions you define in advance - pick one of these options, give a score, answer yes or no - and returns probabilities instead of text. TypeSafe says it is trained for calibrated probabilities, so that outcomes it rates at 20% happen about 20% of the time, and its documentation says plainly that this guarantees nothing about any single answer. It is in early access, and the numbers are the vendor’s, so run it on your own evaluation set before you trust it. The idea is sound: most decisions inside an agent are classification, and a model that cannot produce prose cannot wander off into it.
4. Constrain the LLM and pin the scale
When a decision needs reading and judgment that only an LLM can provide, narrow what it may return: a JSON schema with an enum of allowed labels, validated in code, retried or escalated when validation fails.
Then give it a scale it cannot reinterpret. One technique that works for us: describe the scale itself in the prompt. Instead of “rate the risk from 1 to 10”, define what a case at 2, at 5 and at 8 looks like and which properties put it there, so the model places new cases against fixed anchors rather than inventing its own meaning of “7”. Showing pairs, a case next to a clearly worse one, is another way to pin the scale and worth testing on your data. If you do, run each pair in both orders: models tend to favor whichever option they see first.
5. Ask more than once when the answer matters
If you need a stable probability rather than a single label, send the same question several times and take the vote or the average. Five runs that agree 5 to 0 and five runs that split 3 to 2 are different answers, and the split is information: it tells you the case is borderline. It costs five model calls, so keep it for decisions where a flip is expensive.
6. Store the decision instead of regenerating it
Log the input, model version, parameters and output of every decision. When someone asks why the system did something, you show what happened instead of asking the model again. Route low-agreement and low-confidence cases to a person.
| Layer | Repeatable? | Relative cost | Use it for |
|---|---|---|---|
| Rules in code | Fully | Lowest | Limits, permissions, math |
| Calibrated classifier | Fully, between releases | Low | High-volume decisions with labeled history |
| Decision model | Typed output with probabilities | Low per the vendor, not yet independently measured | Routing, scoring, guardrails |
| Constrained LLM | Mostly; borderline cases flip | Medium | Judgment on unstructured input |
| LLM with repeated sampling | Stable estimate, not identical runs | High, N calls per decision | Expensive borderline decisions |
| Free-form LLM | No | Medium | Drafting, summarizing, conversation |
How to test whether your agent is consistent enough #
For your product, the honest answer is a number, not a yes or no. Here is how we get it:
- Build an evaluation set from real inputs , and put borderline cases in on purpose. Clean examples hide the problem.
- Run every input 10 to 20 times with production settings. Measure the agreement rate per input, not just average accuracy.
- Read the disagreements. They cluster on borderline cases, and that cluster tells you where a rule, a calibrated threshold or a human belongs.
- Pin the model version and record the provider fingerprint where there is one. Rerun the set on every model, prompt or provider change.
- Set a bar per decision type : 100% agreement for anything that touches money or access, a lower bar for drafting.
What I tell founders #
When a founder comes to us with an agent that behaves differently in production than in the demo, I usually say three things.
Seeds and temperature 0 will not save you. They reduce variation, they do not remove it, and the variation lands on the cases that matter most.
Most “LLM decisions” are classification in disguise. Pull them out into rules, a calibrated classifier or a decision model, and keep the LLM for reading and writing.
Measure agreement, not just accuracy. An agent that is right 90% of the time but answers the same input differently on different days is harder to trust - and to sell - than a slightly less accurate one that behaves the same way every time. Sierra’s τ-bench built a metric for exactly this, pass^k: the share of tasks an agent gets right on every one of k attempts. In their retail tests GPT-4o solved fewer than half of the tasks on a single try, and fewer than 25% held up across eight tries.
FAQ #
Is AI deterministic?
Some AI is. Rules engines and trained classifiers with fixed weights return the same output for the same input. LLMs served through APIs are not deterministic in practice, even with temperature 0 and a fixed seed, because batching on the provider’s servers and infrastructure updates change the computation.
What does deterministic mean in AI?
It means the output is fully decided by the input: same input, same output, on every run, with nothing sampled. A deterministic model can still be wrong. It is wrong the same way every time, which is what makes it testable.
Is ChatGPT deterministic?
No. ChatGPT samples its answers, and even the API with temperature 0 and a seed is only best-effort deterministic. Expect the same prompt to produce different answers, most often on long outputs and borderline questions.
Deterministic or generative AI: which one does my product need?
Usually both. Use generative models to read messy input and write text. Use deterministic components - rules, calibrated classifiers, decision models - for the decisions that move money, data or access.
Does temperature 0 make an LLM deterministic?
It makes the model always pick its most likely next token, which removes intentional randomness. It does not remove variation from batching on the provider’s servers, model updates or changing tool results. In one public test, 1,000 runs at temperature 0 produced 80 different answers.
The short version #
- Deterministic AI returns the same output for the same input. Are LLMs deterministic? Not in production, even at temperature 0 with a fixed seed.
- So what is deterministic AI for a founder in practice? The part of your product you can test, replay and explain.
- The line between deterministic vs probabilistic parts runs through your product decision by decision. Keep LLMs for reading and writing. Move decisions into rules, calibrated classifiers or decision models, and measure agreement on borderline cases.
If you are building an agent whose decisions have to hold up in front of customers, that is most of what our AI agent development team does. For a first budget figure, try the AI agent cost calculator.