JSON mode changes what your model answers

A July 2026 study of 44 language models found that asking for JSON output collapsed answer diversity: the modal answer rose from 41% to 64% of responses, distinct answers per prompt fell from 52 to 36, and a JSON clause flipped 53% of models' stable default answers. The effect tracks tool-use training, with JSON and XML compressing diversity while YAML and CSV do not, meaning the structured output surface used by software agents is measurably narrower than the chat surface where models are evaluated.

Context Engineering MCP Model Context Protocol /blog/context-engineering-lives-in-substrates-not-harnesses/ Context engineering lives in substrates, not harnesses Key takeaway Asking a language model to answer in JSON changes which answer it gives, not just how the answer is wrapped. A July 2026 study of 44 models found structured output collapsed answer diversity: the modal answer rose from 41% to 64% of responses, distinct answers per prompt fell from 52 to 36, and a JSON clause flipped 53% of models' stable default answers. The effect tracks tool-use training: JSON and XML compress diversity, YAML and CSV do not. Since software consumes models almost entirely through structured output, the surface production agents use is measurably narrower than the chat surface where models get evaluated. Asking a language model to answer in JSON changes which answer it gives, not just how the answer is wrapped. A July 2026 study ran the same 31 single-word prompts across 44 language models, once in plain chat and once with a JSON format request appended. In chat, the most popular answer accounted for 41% of responses across the panel. Under JSON, it accounted for 64%. Distinct answers per prompt fell from 52 to 36, and mean answer surprisal dropped from 1.80 to 1.58 bits. Nothing about the questions changed. The only difference was a sentence asking for {"word": "<your answer "} . This matters for context engineering /glossary/context-engineering because structured output is not an edge case. It is the default way software consumes language models: every function call, every extraction schema, every agent pipeline that parses a response. The paper puts the deployment reality plainly: software does not consume language models in prose. It asks for JSON. And the surface it asks through turns out to be measurably narrower than the chat surface where models get evaluated, compared, and ranked. The study re-ran a 31-prompt battery called the One-Word Census across 44 models spanning OpenAI, Anthropic, Google, DeepSeek, Qwen, Meta, and IBM families, accessed through a single aggregator. Thirty prompts follow one pattern, “Name a category ”: name a color, name a tree, name a board game. The thirty-first is fully open: “Pick a word.” Every prompt requests a single word, runs with no system prompt, and samples four times per model at temperature 1.0. The baseline condition appends only “Reply with one word only.” The structured conditions append one extra clause each, asking for the same answer delivered as JSON {"word": "<your answer "} , XML <word your answer</word , YAML, CSV, or a bare bracket wrapper answer . The design detail that makes the result interesting: none of the primary conditions used constrained decoding, token masking, or a provider’s response format switch. The format request was plain text in the prompt. Whatever the models did differently under JSON, they did because of words in the context window, not because a grammar forced their token choices. The format clause acted as context, and the models’ behavior shifted the way behavior shifts when any other context changes. The collapse is format-specific, not a general property of asking for structure. JSON cut mean answer surprisal by 0.22 bits p = .0002 and XML by 0.19 bits p = .002 , while YAML and CSV produced no significant change and an arbitrary bracket wrapper moved diversity the other way, up 0.13 bits. | Format clause | Effect on answer diversity | Significance | |---|---|---| | JSON | -0.22 bits, compression toward modal answers | p = .0002 | | XML | -0.19 bits, compression toward modal answers | p = .002 | | YAML | No significant change | n.s. | | CSV | No significant change | n.s. | | Arbitrary brackets | +0.13 bits, more diverse | Reversed effect | The compression is not a subtle statistical artifact spread thinly across the panel. Six of the 44 models shifted individually and significantly toward the panel’s modal answer, and within-run resampling makes the effect concrete: when a model gave the same answer across all four chat samples for a prompt, a stable default, appending the JSON clause changed that answer 53% of the time. Half of what looked like a model’s settled preference in chat was contingent on the output format. The bracket result is the control that gives the rest of the table meaning. Brackets are structure too: a wrapper, a compliance requirement, a parseable shape. If mere formality caused convergence, brackets should compress like JSON does. Instead they loosened answers. Whatever drives the collapse is specific to certain formats, not to the presence of formatting instructions. The formats that compress are exactly the ones models are post-trained to produce as answers. JSON and XML are the delivery formats of function calling, structured-output modes, and tool-use training. YAML and CSV appear in training corpora mostly as things models read, configuration files and data tables, rather than formats they are rewarded for speaking. The paper’s interpretation is that a JSON clause does more than specify syntax. It relocates the prompt into the model’s tool-use register, where post-training has systematically rewarded convergent, schema-safe, high-probability completions. That reading fits the bracket reversal. A bracket wrapper is structural but appears nowhere in tool-use reward loops, so it carries no gravitational pull toward the tool-calling persona. It may even signal informality, which would explain the slight loosening. This is the same lesson we drew from the TOON vs JSON benchmark /blog/toon-vs-json-why-smaller-doesnt-mean-cheaper/ , extended one level deeper. That study showed a format’s cost is downstream of the training distribution: a compact format is expensive if the model is not fluent in it. This study shows a format’s content is downstream of the training distribution too. The choice of output format does not just change what you pay per token. It changes which tokens you get. Production systems consume models almost entirely through the surface this study shows is most collapsed. Models are evaluated, benchmarked, and vibe-checked in chat, but agents call them through function-calling APIs, extraction schemas, and MCP /glossary/mcp tool interfaces, where JSON is the lingua franca. If the two surfaces behave differently, then chat-based evaluation is systematically sampling a wider, more varied distribution than the one production traffic actually draws from. For tasks with a single correct answer, the gap is harmless or even helpful. Extraction, classification, routing, and lookup all want convergence: there is one right invoice number, and a model that reliably lands on it is doing its job. The collapse the paper measures lives in wide answer spaces, where many answers are valid and which one you get is the product. Those wide-answer-space tasks are more common in agent pipelines than they first appear: The consensus pull also rhymes with a problem support teams already know: AI replies that all sound the same. We covered the retrieval side of that in why AI customer support replies sound generic /blog/why-ai-customer-support-replies-sound-generic/ . This result adds an output-side mechanism: a pipeline that forces every reply through a schema is nudging the model toward its most average behavior at the same time the prompt is asking for personality. The practical fix is to separate what the model decides from how the pipeline receives it. Structure is a requirement of the software, not of the decision, and the two can be decoupled. Let the model answer unconstrained, then convert the answer to your schema with a second lightweight call or a plain parser. The two-step costs an extra round trip, but for diversity-sensitive tasks it is the honest price of sampling the distribution you think you are sampling. The bracket finding suggests minimal wrappers are safe if you need some delimiting for parseability without invoking the tool-use register. Schema-constrained output is the correct tool for extraction, routing, classification, and anything else with a verifiable answer. Convergence is the goal there, and structured context /glossary/structured-context on the input side plus structured output on the answer side is a reasonable default. The mistake is carrying that default over to generation tasks where the answer space is wide and the variance is the value. If production asks for JSON, evaluate in JSON. A model comparison, prompt tweak, or temperature setting validated in chat may not transfer to the constrained surface, because the two surfaces are measurably different distributions. This is an easy addition to the harness described in how to measure context quality for AI /blog/how-to-measure-context-quality-for-ai/ : run the same eval set through the exact output contract production uses, and treat chat-mode results as a different condition rather than a proxy. The result is narrower than the headline invites, and the paper is careful about it. The data is a single snapshot through one API aggregator, where the requested temperature of 1.0 is not uniformly honored across providers. Each format was tested with a single clause phrasing, so the study cannot fully separate the format’s register from the specific sentence used to request it. Scoring used exact-match normalization rather than embeddings or judges, and the diversity metrics are panel-relative: the bit values depend on which 44 models make up the reference pool. Most importantly for practitioners, the primary measurement is prompt-only. Native enforcement paths, response format with a JSON schema and real tool-call framing, went largely unmeasured; a supplementary schema-enforcement condition covered only the 36 models that support it, with provider-dependent implementations. Grammar-level constraint could compress answer choice more, less, or differently than a prompt clause. And because the battery is single-word prompts with wide answer spaces, the study says nothing about accuracy degradation on tasks with a right answer. The claim is not that JSON makes models worse. The claim is that JSON makes models different, and different in a consistent direction: toward consensus. Context engineering usually concentrates on what goes into the window: retrieval quality, structure, token budgets, placement. This study is a reminder that the last two lines of the prompt, the ones specifying the shape of the reply, act on the model the same way everything else in the window does. A format clause is context, and models respond to it as context, shifting half of their stable defaults on the strength of one sentence. The uncomfortable version of the conclusion: if your pipeline asks for JSON, you are not sampling the model you evaluated in chat. You are sampling a narrower one, and no amount of temperature tuning on the chat surface will tell you by how much. The only way to know what your production model does is to measure it through the contract production actually uses. Sources: Structured Output Collapses Answer Diversity Across 44 Language Models arXiv 2607.18476 https://arxiv.org/abs/2607.18476 · OpenAI: Introducing Structured Outputs in the API https://openai.com/index/introducing-structured-outputs-in-the-api/ · Anthropic: Tool use documentation https://docs.anthropic.com/en/docs/build-with-claude/tool-use Related Wire transforms your documents into structured, AI-optimized context containers. Upload files, get MCP tools instantly.