# Do Structured Outputs Make LLM Responses Deterministic?

> Source: <https://www.vincentschmalbach.com/structured-outputs-llm-deterministic/>
> Published: 2026-08-11 09:13:46+00:00

### What Is Batch Invariance in LLM Inference?

Batch invariance means a request produces the same inference result when the server runs it alone, alongside other requests, at a different…

No. For a large language model (LLM), structured outputs make responses more predictable in **format**, not necessarily in **content** or execution. A structured output is data generated to match a defined schema, such as a JSON object with [required fields, permitted types, and allowed enum values](https://json-schema.org/draft/2020-12/json-schema-validation). The schema gives downstream code a reliable interface, but it usually leaves many valid answers available.

Determinism means producing the same observable result every time for the same fully specified request, model, configuration, inputs, and execution environment. Structured outputs narrow the set of possible results, but they do not generally force the model, serving infrastructure, tools, or external data sources to produce one identical result.

Structured outputs address a specific integration problem: they reduce malformed JSON, missing fields, extra fields, and incorrect value types. They do not turn a language model into a deterministic function.

A useful distinction is:

Structured outputs help most directly with the first property. They may improve the second indirectly by making validation easier. They do not guarantee the third or fourth.

For example, a sentiment schema might require:

```
{
  "summary": "string",
  "sentiment": "positive | neutral | negative"
}
```

Every successful response must contain the expected fields and a permitted sentiment label. However, the model can produce different summaries, choose different labels for an ambiguous sentence, or vary the wording across calls. Both results can be valid instances of the same schema.

OpenAI reported that strict Structured Outputs achieved [100% schema-matching reliability](https://openai.com/index/introducing-structured-outputs-in-the-api) in an internal evaluation for `gpt-4o-2024-08-06`

. The evaluation supports a strong claim about schema adherence for the tested model and benchmark, but it does not show that repeated calls return identical content. OpenAI describes the underlying model behavior as inherently nondeterministic and distinguishes schema enforcement from model output selection in its [Structured Outputs explanation](https://openai.com/index/introducing-structured-outputs-in-the-api).

Most structured-output implementations use ** constrained decoding**, which blocks tokens that would make the partial response invalid under a grammar or schema.

The basic sequence is:

OpenAI describes a process that compiles schemas into context-free grammars and masks invalid next tokens. Amazon Bedrock documents a similar process: it validates supported schemas, compiles grammars, and uses them to produce schema-compliant results. See the [OpenAI constrained-decoding description](https://openai.com/index/introducing-structured-outputs-in-the-api) and [Amazon Bedrock structured-output documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html).

The constraint removes invalid paths. It does not normally select one path from all valid paths.

A free-form `summary`

string permits a very large number of outputs, while an enum with three values permits three possible values. Optional fields, arrays, object properties, and unconstrained numbers expand that set. The schema defines valid documents, but not a unique document.

The distinction can be expressed formally:

```
y \in L(\text{schema})
```

means that output `y`

belongs to the language of documents permitted by the schema.

Deterministic generation requires a stronger condition:

```
f(x) = y
```

for the same fully specified input `x`

, every time.

The first condition requires a valid output, and the second requires the same output every time. A constrained decoder can enforce validity while leaving many possible values for the output.

Two valid responses can differ in:

A strict schema therefore creates a reliable response contract without necessarily creating a deterministic decision procedure.

A schema can force one successful output when it admits exactly one complete document. For example, every field could be required and fixed with `const`

, with no free-form strings, variable numbers, optional properties, or variable-length arrays.

That is a mathematical exception, not a normal property of structured generation. The schema has removed every meaningful choice from the output language. It has not demonstrated that ordinary model generation is deterministic.

Even then, the complete API operation can fail or produce a different response state. Refusals, incomplete generation, token limits, transport failures, serialization behavior, provider changes, and external-input variation remain outside the singleton schema.

Variation can remain at several layers after schema constraints are applied.

A language model assigns probabilities to possible next tokens. Temperature changes how sharply the decoder favors high-probability tokens, while other decoding settings affect selection in different ways. Structured decoding removes illegal continuations, but several legal continuations can remain.

For instance, if a schema permits `"approve"`

and `"reject"`

, the constraint guarantees that the selected value belongs to that set. It does not guarantee which value the model selects.

A schema can also change the model’s available expression paths. Research on [Grammar-Aligned Decoding](https://arxiv.org/abs/2405.21047) argues that ordinary grammar constraints can preserve formal validity while distorting the model’s original probability distribution. A separate study, [The Hidden Cost of Structure](https://aclanthology.org/2025.ranlp-1.124), found that constrained decoding affected task performance differently across models and tasks. Structural validity and task quality therefore require separate measurements.

Temperature zero, often called greedy decoding, reduces intentional sampling variation by favoring the highest-probability next token. It does not control every source of variation in a hosted inference service.

Results can change because of:

OpenAI’s historical guidance for its seed feature recommends keeping the seed, request parameters, and `system_fingerprint`

constant for mostly consistent outputs. It also states that divergence remains possible and that the fingerprint can change when the provider changes model weights, infrastructure, or related configuration. The [reproducibility guidance](https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter) is therefore a best-effort control, not a universal determinism guarantee.

A peer-reviewed study of ChatGPT code generation found residual variation at temperature zero. The study did not test current strict structured-output APIs, so its results should not be treated as a direct benchmark of JSON Schema determinism. They do show why temperature zero alone is not sufficient evidence of exact repeatability. See [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://discovery.ucl.ac.uk/id/eprint/10198256/1/3697010.pdf).

A model call cannot be reproduced exactly when its effective inputs change.

Common examples include:

A structured schema does not freeze any of these inputs. If an agent retrieves a changing customer record and returns a valid object, the object can remain schema-compliant while containing different values on the next run.

For reproducibility tests, record and replay tool calls, retrieved context, database results, timestamps, and other external inputs. Reproducing the model response is different from reproducing the entire workflow, including tool calls and downstream side effects.

A schema validates representation and permitted value forms. It does not normally establish whether those values are true.

Consider:

```
{
  "customer_id": "C-1042",
  "invoice_total_usd": 999999,
  "is_overdue": false
}
```

A schema can require a string for `customer_id`

, a number for `invoice_total_usd`

, and a Boolean for `is_overdue`

. It cannot establish that the invoice total is correct or that the account is current. Those checks require authoritative records and application logic.

Google’s [structured-output documentation](https://ai.google.dev/gemini-api/docs/generate-content/structured-output) explicitly warns that syntactically correct JSON does not guarantee semantically correct values. [OpenAI also notes](https://openai.com/index/introducing-structured-outputs-in-the-api) that a response can match its schema while containing mistakes within field values.

Applications should therefore validate at least three layers:

A field named `confidence`

, `source`

, or `evidence`

does not prove the associated claim. The application must verify those fields against trusted data when the decision matters.

“Strict structured output” is not a universal standard with identical behavior across providers. Providers generally implement documented subsets of JSON Schema or related tool-input rules.

For example:

The [JSON Schema validation specification](https://json-schema.org/draft/2020-12/json-schema-validation) specifies validation rules for JSON instances. It does not require an LLM provider to support every keyword during generation.

Guarantees also depend on the response state. A provider’s successful schema-conforming completion is different from a promise that every request returns a usable object. Refusals, incomplete responses, token limits, content filtering, invalid requests, serialization problems, and transport errors still require handling.

Document structured-output guarantees conditionally:

For supported schemas, valid requests, and successful completion states, the provider enforces the documented structural constraints.

That wording avoids extending a format guarantee into a claim about semantic correctness or universal availability.

Pass structured output from probabilistic model behavior into deterministic application logic through a typed boundary:

```
LLM → schema-constrained object → parser → business validation
    → authorization and idempotency checks → side effect
```

The model proposes an extraction, classification, or action. Ordinary code should decide whether that proposal is valid, authorized, and safe to execute.

For example, a cancellation workflow should verify that:

A valid tool argument does not establish any of those facts.

Protect high-impact actions such as refunds, deployments, account changes, or external messages with idempotency keys, transactional checks, authorization gates, replay logs, and human review where appropriate. These controls protect the workflow even when the model returns a structurally valid but incorrect decision.

Do not infer determinism from one successful response. Run repeated calls against representative prompts and controlled fixtures.

Record:

Track separate metrics for:

Canonical JSON comparison ignores irrelevant differences such as whitespace and, depending on the canonicalization method, object-key ordering. Exact byte comparison is stricter. Semantic equivalence asks whether two different representations mean the same thing. These measures answer different engineering questions and should not be combined into one “reliability” score.

Run tests both for isolated model calls and for complete workflows that include retrieval, tools, persistence, and downstream actions. Set the acceptance criterion according to the actual requirement: parseability, stable classification, factual accuracy, or safe execution.

No. They constrain responses to a supported schema or grammar, but that schema usually permits many valid values and phrasings. Sampling, backend changes, tools, retrieval, and other inputs can still produce different results.

For a supported schema in a successful completion state, strict mode generally enforces the required structure, types, and other documented constraints. It does not guarantee factual accuracy, identical values across calls, or a usable object after a refusal or incomplete response.

Not reliably. Temperature zero can reduce sampling variation, but it does not control model revisions, routing, numerical effects, tie-breaking, external inputs, or provider-side changes. Test repeatability with fixed inputs and recorded configuration instead of treating temperature zero as a guarantee.

No. A schema can require a Boolean, number, enum, or string without verifying that the value is true. Check important claims against authoritative data and apply semantic, authorization, and business-rule validation before taking action.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com