I Watched XGrammar Forbid a Token: What Grammar-Constrained Decoding Actually Does to Your Logits

XGrammar, a library used by inference engines such as vLLM and SGLang, performs grammar-constrained decoding by masking invalid tokens from the sampling distribution at each generation step, guaranteeing valid JSON output without retry loops. The library's latest version 0.2.3 was released on June 27, 2026, and it is integrated as a structured-output backend in multiple inference engines.

Member-only story I Watched XGrammar Forbid a Token: What Grammar-Constrained Decoding Actually Does to Your Logits At the very first step of generating a JSON object, I asked XGrammar which of my 15 vocabulary tokens the model was allowed to produce. The answer was one: { . The other fourteen were masked out before the model ever got to vote on them. That single number is the whole argument for constrained decoding. If your agent parses model output as JSON, you have almost certainly written a retry loop: call the model, try to json.loads the result, and on failure re-prompt with "please return valid JSON." Constrained decoding replaces that loop with a guarantee. The model cannot emit a token that would break the grammar, because at each step the invalid tokens are removed from the sampling distribution. There is no malformed output to retry. XGrammar https://github.com/mlc-ai/xgrammar is the library most inference engines reach for to do this. Version 0.2.3 landed on PyPI on June 27, 2026 https://pypi.org/project/xgrammar/ , the latest in a fast run of releases that started with the 0.2.0 line on May 1. It ships as a structured-output backend inside vLLM, SGLang, TensorRT-LLM, and MLC-LLM. Most developers use it through those engines and never see the mask. This article pulls the mask out into the open, on a CPU, with no model weights and no API key, so you can watch it work token by token.