Member-only story
At the very first step of generating a JSON object, I asked XGrammar which of my 15 vocabulary tokens the model was allowed to produce. The answer was one: {
. The other fourteen were masked out before the model ever got to vote on them.
That single number is the whole argument for constrained decoding. If your agent parses model output as JSON, you have almost certainly written a retry loop: call the model, try to json.loads
the result, and on failure re-prompt with "please return valid JSON." Constrained decoding replaces that loop with a guarantee. The model cannot emit a token that would break the grammar, because at each step the invalid tokens are removed from the sampling distribution. There is no malformed output to retry.
XGrammar is the library most inference engines reach for to do this. Version 0.2.3 landed on PyPI on June 27, 2026, the latest in a fast run of releases that started with the 0.2.0 line on May 1. It ships as a structured-output backend inside vLLM, SGLang, TensorRT-LLM, and MLC-LLM. Most developers use it through those engines and never see the mask. This article pulls the mask out into the open, on a CPU, with no model weights and no API key, so you can watch it work token by token.