Parallel Constrained Decoding

A developer released Qwen-2.5-1B-RLCD on Hugging Face, an open-source model using parallel constrained decoding that delivers 5x faster on-device inference for type-safe JSON workloads, demonstrated on an M4 MacBook. The release claims the approach requires no new training and can batch-generate every JSON key simultaneously, with the creator stating they spent 2 years in stealth building the RLCD training method.

They were building in stealth for 2 years, I was building in stealth for 2 hours… Happy to open source Qwen-2.5-1B-RLCD, 5x faster on-device inference for JSON workloads that need to be type-safe. ⚡️Demo below on a M4 MacBook⚡️ every LLM has the ability to efficiently batch inference every key of a JSON at the same time and generate probabilities from a set of possible categories. No new training required, but it’s easy to optimize if you need On hugging face now 00:00 After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models RLCD , and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x 00:00