Parallel Constrained Decoding: Faster Structured JSON on Apple Silicon

An MLX engine for Apple Silicon evaluates JSON schema fields in parallel, cutting structured extraction latency 5.6x to 7x with guaranteed valid syntax.

What is parallel constrained decoding? #

Parallel constrained decoding is an inference technique that generates structured JSON output by evaluating every schema field at once, instead of producing tokens one at a time. Built on Apple’s MLX framework, it runs on Apple Silicon Macs and pairs with a quantized Qwen2.5 1.5B Instruct model. In benchmarks on an M4 Max, it cuts latency by 5.6x to 7.0x compared to standard autoregressive JSON generation, while guaranteeing 100% schema-valid output and producing calibrated confidence scores for every field.

TL;DR #

• Field-parallel evaluation replaces token-by-token generation for structured extraction tasks, so a 28-field schema takes one forward pass instead of hundreds.
• Benchmarked on an M4 Max withmlx-community/Qwen2.5-1.5B-Instruct-4bit , the approach hit5.6x to 7.0x latency reductions across four test scenarios.
• Output is assembled programmatically from verified candidate values rather than parsed from free text, which is why it reports100% guaranteed syntax validity .
• The engine restricts each field to its valid candidate token set (a sub-vocabulary), masking everything else before computing softmax probabilities, so confidence scores are exact rather than approximated.
• A single enum field can support up to 255 choices , tested with a high-cardinality tariff classification scenario that still ran in about 89 milliseconds.
• The project ships with a command-line benchmark runner , a Python SDK, and a web visualizer that shows the parallel and autoregressive paths side by side, including hallucinated or omitted keys in the baseline.
• It currently targets one specific model but is built on mlx-lm , so in principle any MLX-compatible decoder model could be swapped in.

Other agents start typing. Remy starts asking. #

Scoping, trade-offs, edge cases — the real work. Before a line of code.

How does parallel constrained decoding work? #

Standard structured generation, whether it’s JSON mode or grammar-constrained sampling, still decodes one token at a time. Each token requires its own forward pass through the model, and total latency scales linearly with output length. A 28-field extraction schema might need 300+ sequential passes, each with its own memory bandwidth round trip. That’s also where most JSON generation failures come from: missing keys, malformed brackets, or hallucinated fields that don’t match the schema.

The parallel approach exploits a specific property of structured extraction: most fields aren’t open-ended text, they’re picks from a bounded set (an enum with a fixed list of choices, or a boolean). That constraint means you don’t need to generate token-by-token when you can instead score which candidate is correct.

The pipeline works like this:

1. Single broadcast prefill. The context document and the schema descriptions are prefilled once into an MLX KV-cache.
2. KV-cache broadcasting. That same cache state gets broadcast across every field in the schema simultaneously, instead of being recomputed per field.
3. Sub-vocabulary logit slicing. For each field, the engine only evaluates the token IDs that correspond to valid schema choices. Everything else in the vocabulary gets masked out.
4. Calibrated softmax. Probabilities are computed as an exact normalized softmax over just the candidate slice, not the full vocabulary, so the resulting confidence numbers are honest reflections of the model’s relative preference among valid options.
5. Token tree disambiguation. When multiple candidate choices share the same starting tokens (multi-token prefixes), the engine runs a short continuation step using sliced cache states, without reallocating memory.
6. Programmatic assembly. The final JSON is built directly from the verified field values, not generated as raw text and then parsed. That’s the mechanism behind the 100% syntax validity claim: there’s no JSON parser step that can fail.

What do the benchmarks actually show? #

The published numbers come from four preset scenarios run on an M4 Max, comparing the parallel engine against an autoregressive baseline using the same quantized Qwen2.5 1.5B model:

• Fintech fraud routing (4 fields): autoregressive baseline took 420 ms across 148 sequential passes; the parallel engine finished in 75 ms with a single pass. That’s a 5.6x speedup.
• Code security audit (4 fields): 380 ms baseline versus 68 ms parallel, also 5.6x.
• High-cardinality tariff classification (1 field, 255 possible choices): 500 ms baseline versus 89 ms parallel, still 5.6x, which matters because it shows the technique holds up even when a single field has a huge candidate set.
• Enterprise support triage (28 fields): 1,900 ms baseline versus 270 ms parallel, a 7.0x speedup, the largest gain in the set, consistent with the idea that parallel evaluation scales better as field count grows.

Across all four, schema match was reported as 100% for both methods in these tests, but the parallel method’s validity is guaranteed by construction rather than measured empirically. The autoregressive baseline ran at roughly 116 to 131 tokens per second, in line with what’s typical for a 1.5B parameter model at 4-bit quantization on Apple Silicon.

One coffee. One working app. #

You bring the idea. Remy manages the project.

Why does field count change the speedup? #

The autoregressive baseline’s cost is directly proportional to the number of output tokens, which roughly tracks field count and choice complexity. Go from 4 fields to 28 fields and you’re looking at 148 passes versus 312 passes, a linear increase. The parallel method’s cost is dominated by the prefill step (encoding the context once) plus a constant number of slicing operations per field, which is far cheaper than a full forward pass. That’s why the speedup climbs from 5.6x at 4 fields to 7.0x at 28 fields: the parallel method’s overhead grows slower than the baseline’s does. In principle, that gap should widen further on even larger schemas, though the published benchmarks only go up to 28 fields.

Is parallel constrained decoding worth using? #

It’s a good fit for a narrow but common category of workload: structured classification and routing tasks where every field maps to a fixed set of choices (enums, booleans, or category labels). Fraud triage, support ticket routing, security audit flags, tariff and taxonomy classification, and similar categorical extraction jobs are exactly the use case it’s built for.

It’s a poor fit for open-ended generation. If a field needs free-text output (a summary, a customer-facing message, a paragraph of reasoning), there’s no fixed candidate set to slice logits against, so the technique doesn’t apply. It also currently targets one specific model configuration (Qwen2.5-1.5B-Instruct at 4-bit), and while the underlying mlx-lm framework supports swapping in other decoder models, published benchmarks only cover that one setup.

The hardware requirement is also a real constraint: this is MLX-based, so it runs on Apple Silicon Macs (M1 through M4 series) with macOS 14 or later. There’s no evidence in the source material of a CUDA or server-GPU equivalent, so teams running inference on non-Apple hardware won’t be able to reproduce these numbers directly.

For teams already doing local, on-device structured extraction on Mac hardware, principally for privacy-sensitive workloads or cost-conscious local inference, the combination of lower latency, guaranteed valid JSON, and per-field confidence scores is a meaningful upgrade over prompting a model for JSON and hoping it parses.

Frequently Asked Questions #

What hardware does parallel constrained decoding require?

It requires an Apple Silicon Mac (M1, M2, M3, or M4 series) running macOS 14.0 or later, plus Python 3.10+. The benchmarks were run specifically on an M4 Max.

Which model was used in the benchmarks?

mlx-community/Qwen2.5-1.5B-Instruct-4bit, a 4-bit quantized 1.5 billion parameter instruction-tuned model. The engine’s model ID is configurable, and any decoder model supported by the mlx-lm library can theoretically be loaded in its place.

How does it guarantee 100% valid JSON?

The output JSON is assembled programmatically from verified field values rather than generated as raw text and parsed afterward. Since each field’s value is chosen from a masked, pre-defined set of valid tokens, there’s no way to produce a malformed key, a broken bracket, or a hallucinated field.

How many choices can a single field have?

Remy doesn't write the code. It manages the agents who do. #

Remy runs the project. The specialists do the work. You work with the PM, not the implementers.

Up to 255 choices per enum field, based on the published schema design. A benchmark scenario using a 255-choice tariff classification field still completed in about 89 milliseconds on an M4 Max.

Does this work for free-text generation, not just classification?

No. The technique depends on fields having bounded candidate sets, like enums or booleans. Open-ended text fields don’t have a fixed set of valid tokens to slice logits against, so this method doesn’t apply to summarization, freeform reasoning, or similar generative tasks.