Structured Language Model Generation with Outlines

Outlines, an open-source library, introduces deterministic certainty into LLM output generation for reliable structured outputs like JSON and classification. It masks illegal tokens during inference to enforce output constraints, demonstrated through Python examples for sentiment analysis and JSON generation.

Structured Language Model Generation with Outlines Outlines is an open-source library that introduces deterministic certainty into LLMs' output generation process for better, more reliable generation of structured outputs. Introduction Usually, when asking an LLM — abbreviation for " Large Language Model " — for a neat, structured output like JSON objects, for instance, a mix of careful prompt crafting with a "pinch" of luck is required. Otherwise, it might be tricky to get the model to obtain the perfectly structured output you are expecting. Or so it was, until a novel open-source library http://A new, open-source library that introduces deterministic certainty into LLMs' output generation process for better, more reliable generation of structured outputs. came onto the scene: outlines . This library is designed to prevent typical issues experienced by LLMs in these specific output-oriented use cases, such as hallucinations. More precisely, it introduces a degree of deterministic certainty into the output generation process. Let's uncover what outlines allows us to do through this illustrative article, in which we will show some practical examples in Python Use Case 1: Multiple-Choice Classification for Sentiment Analysis Before fully diving into the first use case, you might be wondering. How does outlines work and how does it guarantee correctness in structured model outputs? At the inference level, it masks out "syntactically illegal" tokens during generation instead of attempting to fix poor text once generated. This makes it virtually impossible to break the rules underlying the specific output format sought. Let's see a first example in which we are building an analysis pipeline for customer support tickets, and we want exactly one option from a limited, approved list of possible options. This is pretty much like a classification problem, and generate.choice is the function that helps us mimic it, by forcing the model at hand to choose one of the predefined literals or classes. But first, let's install it alongside the transformers to load pre-trained LLMs: pip install outlines transformers The following code uses outlines.from transformers to load a pre-trained model aided by Hugging Face's auto classes for a model and its associated tokenizer. But the icing on the cake is: they are both wrapped in an outlines object that will later help tell the model what exactly to obtain. At the inference stage, we pass not only the user prompt asking to classify a review, but also a Literal object that contains the output constraints the model should limit to: python import outlines from transformers import AutoTokenizer, AutoModelForCausalLM from typing import Literal 1. Loading the backend using standard Transformer-based models model name = "microsoft/Phi-3-mini-4k-instruct" We use outlines to load the model with its from transformers function model = outlines.from transformers AutoModelForCausalLM.from pretrained model name , AutoTokenizer.from pretrained model name 2. Calling the model directly, passing our approved strings as type constraints sentiment = model "Classify the sentiment of this customer review: 'I've been waiting two weeks for my delivery and it's still missing.'", Literal "Positive", "Negative", "Neutral" print sentiment Output: Negative A word of warning here: although the literal we defined is part of Python's built-in typing module, rather than outlines , our out-of-the-box library still assumes model control here: both the model and tokenizer are wrapped into an object that enforces standard Python types, building a finite state machine under the hood that limits the output to the options provided only. Use Case 2: JSON Object Generation This example first defines a Pydantic object that defines the desired structure for a JSON object describing a fictional character with a name, description, and age. It then uses our previously wrapped outlines model, passing the character object to ensure the output generated strictly follows this structure for the JSON object requested: python from pydantic import BaseModel 1. Define a Pydantic model for the desired JSON structure class Character BaseModel : name: str description: str age: int 2. Using the outlines-wrapped model to generate a JSON output conforming to the Pydantic model json output = model "Generate a JSON object describing a fictional character named 'Anya'.", Character, max new tokens=200 print json output Output: { "name": "Anya", "description": "Anya is a young, adventurous woman with a passion for exploring new places and meeting new people. She has long, curly hair and bright green eyes that sparkle with curiosity. Anya is always eager to learn and loves to share her knowledge with others. She is kind-hearted and always willing to lend a helping hand to those in need. Anya's favorite hobbies include hiking, reading, and playing the guitar. She is a free spirit who values freedom and independence above all else." ,"age": 25 } Use Case 3: Pure JSON Generation for REST APIs This third example, also JSON-related, is similar to the previous one but in a slightly different context. Imagine you are building an API backed requiring a well-defined JSON payload for updating a database. Asking a standard LLM to get this output will more often than not yield to annoying, trailing characters like commas that are likely to crash a JSON parser. With outlines, we define our JSON payload schema once again with a Pydantic-based custom class object. python from pydantic import BaseModel from typing import Literal import json class ServerHealth BaseModel : service name: str uptime seconds: int status: Literal "OK", "DEGRADED", "DOWN" 1. Outlines should produce a raw string guaranteed to be valid JSON raw json string = model "Report the current status of the main Auth database.", ServerHealth, max new tokens=50 print type raw json string This will just print: 2. Pretty-printing parsed json = json.loads raw json string print json.dumps parsed json, indent=2 Output: { "service name": "auth db status", "uptime seconds": 1623456789, "status": "OK" } Closing Remarks Since LLMs are trained to be chat-lovers capable of breaking syntax or hallucinating to "sound like humans" in their conversations with us, getting them to produce reliable, structured outputs like clean JSON objects can feel like a bit of a pain. Outlines a new, open-source library that introduces deterministic certainty into LLMs' output generation process for better, more reliable generation of structured outputs. This article showed three simple yet useful use cases for beginners with this interesting tool. is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world. Iván Palomares Carrascosa https://www.linkedin.com/in/ivanpc/