{"slug": "using-llama-cpp-python-grammars-to-generate-json", "title": "Using Llama-cpp-Python grammars to generate JSON", "summary": "Llama.cpp added grammar-constrained output generation on August 17, letting developers restrict a large language model's next-token selection so responses exactly match a specified grammar, and the llama-cpp-python library exposes the feature through the LlamaGrammar class. Simon Willison demonstrated the technique by loading the json_arr grammar and the 8-bit quantized Llama-2-13B-GGUF model (a 13.8GB file from TheBloke/Llama-2-13B-GGUF), prompting \"JSON list of name strings of attractions in SF:\" with grammar=grammar and max_tokens=-1, which returned a valid JSON array of San Francisco attractions including SFMOMA at 151 Museum Way. Willison said the most exciting possibility is building a version of OpenAI Functions on top of models like Llama 2 that can run on your own device.", "body_md": "[llama.cpp](https://github.com/ggerganov/llama.cpp) recently added the ability to control the output of any model using a grammar.\n\nThis is an incredibly powerful technique for working with a Large Language Model. Effectively it lets you insert custom code into the model's output generation process, ensuring that the overall output exactly matches the grammar that you specify.\n\nThis works by directly modifying the next-token selection logic, restricting the model to only being able to pick from the tokens that fulfill the rules of the grammar at any given point.\n\nThe most exciting possibility for this in my opinion is building a version of OpenAI Functions on top of models like Llama 2 that can run on your own device.\n\nI hadn't quite figured out how to use these yet, until Ian Maurer [tipped me in the right direction](https://twitter.com/imaurer/status/1699467351937224828).\n\nHere's how to get started with them using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) Python library.\n\nFirst, install it - and make sure you have a recent version, grammars only landed on August 17th (though there have been a ton of releases since then, it's a very fast moving project).\n\n```\npip install -U llama-cpp-python\n```\n\nYou need a grammar. There's a set of examples in the [llama.cpp/grammars](https://github.com/ggerganov/llama.cpp/tree/master/grammars) folder.\n\nMy favourite so far is the `json_arr` one, which guarantees that the response will be a valid JSON array:\n\nI'll fetch that straight into Python using `httpx`.\n\nYou also need a model. I'm using Llama 13B as GGUF from [TheBloke/Llama-2-13B-GGUF](https://huggingface.co/TheBloke/Llama-2-13B-GGUF) - I downloaded the 8bit quantized model, a 13.8GB file from here:\n\n[https://huggingface.co/TheBloke/Llama-2-13B-GGUF/resolve/main/llama-2-13b.Q8_0.gguf](https://huggingface.co/TheBloke/Llama-2-13B-GGUF/resolve/main/llama-2-13b.Q8_0.gguf)\n\nHere's my Python code that exercises it.\n\nFirst, import the modules and load the grammar:\n\n``` python\nfrom llama_cpp.llama import Llama, LlamaGrammar\nimport httpx\ngrammar_text = httpx.get(\"https://raw.githubusercontent.com/ggerganov/llama.cpp/master/grammars/json_arr.gbnf\").text\ngrammar = LlamaGrammar.from_string(grammar_text)\n```\n\nNow load the model:\n\n```\nllm = Llama(\"llama-2-13b.Q8_0.gguf\")\n```\n\nThis spews out a ton of debug output, some of which looks like this:\n\n```\nllama_model_loader: loaded meta data with 19 key-value pairs and 363 tensors from /Users/simon/Downloads/llama-2-13b.Q8_0.gguf (version GGUF V2 (latest))\nllama_model_loader: - tensor    0:                token_embd.weight q8_0     [  5120, 32000,     1,     1 ]\n...\nllama_new_context_with_model: kv self size  =  400.00 MB\nllama_new_context_with_model: compute buffer total size =   75.47 MB\nAVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |\n```\n\nNow we can prompt it, feeding in the grammar object as `grammer=` and setting `max_tokens=-1` for unlimited output tokens:\n\n```\nresponse = llm(\n    \"JSON list of name strings of attractions in SF:\",\n    grammar=grammar, max_tokens=-1\n)\n```\n\nThe result is a JSON string in `response['choices'][0]['text']`, so we can pretty-print it like this:\n\n``` python\nimport json\nprint(json.dumps(json.loads(response['choices'][0]['text']), indent=4))\n```\n\nWhich gave me:\n\n```\n[\n    {\n        \"address\": {\n            \"country\": \"US\",\n            \"locality\": \"San Francisco\",\n            \"postal_code\": 94103,\n            \"region\": \"CA\",\n            \"route\": \"Museum Way\",\n            \"street_number\": 151\n        },\n        \"geocode\": {\n            \"latitude\": 37.782569,\n            \"longitude\": -122.406605\n        },\n        \"name\": \"SFMOMA\",\n        \"phone\": \"(415) 357-4000\",\n        \"website\": \"http://www.sfmoma.org/\"\n    },\n    {\n        \"address\": {\n            \"country\": \"US\",\n            \"locality\": \"San Francisco\",\n            \"postal_code\": 94129,\n            \"region\": \"CA\",\n            \"route\": \"The Presidio\",\n            \"street_number\": 104\n        },\n        \"geocode\": {\n            \"latitude\": 37.806566,\n            \"longitude\": -122.440633\n        },\n        \"name\": \"Walt Disney Family Museum\",\n        \"phone\": \"(415) 345-6800\",\n        \"website\": \"http://www.waltdisney.org/museum\"\n    }\n]\n```\n\nAs promised, the grammar ensured I got back a valid JSON array, with no filler text (\"Here's the data you asked for as JSON:\") which Llama 2 is very prone to including.\n\nThe model invented the shape of the JSON data. The next challenge will be to figure out how to build grammars that specify the actual JSON shape that I want.\n\nI [got GPT-4 to prototype that for me a bit](https://chat.openai.com/share/bf84aed9-d2a3-4175-ac6e-d2f0873092d7), but it needs a lot more work before it's usable.\n\n[Grammar Builder](https://grammar.intrinsiclabs.ai/) by Intrinsic Labs is an interesting tool here - it can generate GBNF grammars from TypeScript declarations, and is accompanied by [an open source library](https://github.com/IntrinsicLabsAI/gbnfgen) that does the same trick. More about that in [this discussion thread](https://github.com/ggerganov/llama.cpp/discussions/2494).\n\nEvan Jones [pointed out](https://twitter.com/evanqjones/status/1701938802611151300) the `llama.cpp` script [examples/json-schema-to-grammar.py](https://github.com/ggerganov/llama.cpp/blob/master/examples/json-schema-to-grammar.py), which looks like exactly what I want. It only supports a subset of JSON schema right now but it can turn that into a working GBNF grammar.\n\nHere's one thing to watch out for: the grammar trick doesn't 100% guarantee that you will get back valid JSON, because there's always a chance that the model will run out of tokens before it's managed to produce a completed JSON array.\n\nI don't see any way to resolve this, unfortunately: the grammar is considered entirely separately from the part of the model that might decide that it needs to wrap things up because it's running out of tokens.\n\nThere are tricks for dealing with incomplete JSON though, especially if it's producing an array of objects where you could discard the incomplete object at the end.\n\nNext time I dig into this I plan to experiment with [using ijson](https://til.simonwillison.net/json/ijson-stream), a streaming JSON parser, to try and account for this.\n\nCreated 2023-09-12T20:36:33-07:00, updated 2023-09-13T09:06:53-07:00 · [History](https://github.com/simonw/til/commits/main/llms/llama-cpp-python-grammars.md) · [Edit](https://github.com/simonw/til/blob/main/llms/llama-cpp-python-grammars.md)", "url": "https://wpnews.pro/news/using-llama-cpp-python-grammars-to-generate-json", "canonical_source": "https://til.simonwillison.net/llms/llama-cpp-python-grammars", "published_at": "2026-09-23 08:32:21+00:00", "updated_at": "2026-09-23 08:59:31.367480+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "structured-data", "generative-ai"], "entities": ["llama.cpp", "llama-cpp-python", "Llama 2", "Llama-2-13B-GGUF", "TheBloke", "Simon Willison", "Ian Maurer", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/using-llama-cpp-python-grammars-to-generate-json", "markdown": "https://wpnews.pro/news/using-llama-cpp-python-grammars-to-generate-json.md", "text": "https://wpnews.pro/news/using-llama-cpp-python-grammars-to-generate-json.txt", "jsonld": "https://wpnews.pro/news/using-llama-cpp-python-grammars-to-generate-json.jsonld"}}