An Introductory Guide to Practical Constraint Decoding

With this introductory guide to practical constraint decoding, you'll no longer need to beg your model to "output valid JSON without including any markdown."

# Introduction #

Practical constraint decoding, also known as structured generation or guided decoding, encompasses the engineering strategies to force a large language model (LLM) to generate text outputs that strictly abide by a specified data schema, grammar, or regular expression (regex) at the token selection stage.

With the introductory guide to practical constraint decoding in this article, you'll no longer need to beg your model to "output valid JSON without including any markdown", just to cite an example. Constraint decoding makes it mathematically impossible for the LLM to deliver anything outside the defined constraints.

# How Does Practical Constraint Decoding Work? #

While the typical LLM generation process works as an "act of faith" in which you pass a prompt to the model and it might output exactly what you are looking for (or might not), practical constraint decoding takes a subtly distinct approach. It deems the prompt and the text generation as a unique, interleaved program. This makes it possible to lock down certain characters that are key to maintaining a certain required syntax, allowing the model to "fill in the blanks" in between.

Shall we go into a bit more detail? When an LLM outputs the next token of its response, it initially produces a vector of raw scores, or logits — one for every possible token in the vocabulary at hand. This typically entails thousands of possible options to choose from.

But when using practical constraint decoding, something happens earlier, before the inference process starts: a finite state machine is built, whereby a target constraint is compiled — for instance, through a Pydantic model in Python. At a given inference step, the finite state machine evaluates the current state and provides a list of allowed next tokens. This "white list" is used as a mask on the LLM's raw logits vector, such that for every token outside that list, its logit is set to negative infinity, i.e. -inf

in Python.

After the masking process, the model keeps running its softmax normalization and sampling process as usual (based on parameters like temperature, top-p, or top-k) on the "surviving tokens" to eventually select the most probable one and generate it.

It may sound like applying this process to an entire vocabulary spanning many thousands of words might significantly slow down model inference. The good news: it doesn't. Modern Python libraries take advantage of the LLM's vocabulary being static and pre-compile it before the user starts typing their prompt. The state machine won't need to look up the entire vocabulary, and latency overhead is brought down drastically.

So, what is the current gold standard for implementing practical constraint decoding? Arguably, the ** outlines** library has earned this distinction. It allows us to define and pass Pydantic models, JSON schemas, or regex directly to a wrapped version of a pre-trained model, thus constraining its freedom in generating outputs.

# Example #

Let's walk through an example. First, install outlines:

pip install outlines[transformers]

Now onto the code:

from pydantic import BaseModel
import outlines
from transformers import AutoTokenizer, AutoModelForCausalLM

class UserProfile(BaseModel):
    name: str
    age: int
    is_active: bool

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

llm = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = outlines.from_transformers(llm, tokenizer)
result = model("Extract the user: John is a 34 year old pilot.", UserProfile)

print(result)

Output:

{"name": "John", "age": 34, "is_active": true}

This example showed how to use the outlines library to wrap a pre-trained model along with its tokenizer, and constrain it to output JSON objects defined by the custom class we defined — named UserProfile

and inheriting Pydantic's BaseModel

.

# Wrapping Up #

Practical constraint decoding has a series of trade-offs. Let's look at some of them.

Strengths:

• If used correctly, it provides a 100% guarantee for correct syntax, removing the need for parsing blocks in your code.
• It helps drastically save tokens in your prompts, no longer requiring token-consuming few-shot examples to show the model what a correct JSON object should look like, for instance.
• It contributes to the democratization of small models, turning a "tiny" 1B-parameter model that would otherwise jeopardize JSON generation use cases into an infallible data constructor.

Limitations:

• If the LLM needed to say it cannot answer something, but the schema forces it to output an integer, for instance, it will do so, making it no longer honest in edge cases.
• On the first run of a Pydantic schema against an LLM, there may be a freeze of a few seconds to build the finite state machine, making the first run considerably slower — although subsequent ones will be smoother.

This article introduces practical constraint decoding, unveiling why it is necessary in certain LLM-driven situations, how it works, and what the most widely adopted solution in the current landscape is: the outlines library. An example of its use is likewise provided.

is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.

Iván Palomares Carrascosa