Stanford NCM: The New Way of Generating Text 31x Faster

Stanford's Next-Concept Modeling (NCM) generates 1,024 tokens in 32 passes instead of 1,024 on OpenWebText, the basis for the paper's "31x faster" claim, by writing coarse codes for the whole answer first and refining level by level. NCM pairs a compressor that rebuilds original text exactly 98.8% to 100% of the time with a writer that emits all codes of one level at once, so passes equal levels rather than tokens. Stanford's released code and checkpoints, run on an NVIDIA RTX 4060 Laptop GPU, also needed about 4x fewer training tokens (127B versus 500B+) to reach its results.

Every LLM you use today writes the same way: one token, then the next, then the next. A 1,000-token answer means about 1,000 steps in a line, and step 501 cannot start until step 500 is done . Your GPU can do a lot of work in parallel, but text generation keeps it waiting in a queue. That’s a big reason long answers feel slow. Stanford’s new Next-Concept Modeling NCM tries a different route. It sketches the whole answer roughly first, then adds detail level by level, like a blurry WhatsApp photo that gets sharper as it loads. On OpenWebText it writes 1,024 tokens in 32 passes instead of 1,024 . The paper headline is “31x faster”. That number is real, but most people read it wrong, and I’ll come to that. The first half of this post explains the idea in plain words. The second half is what I actually ran on an RTX 4060 laptop GPU with Stanford’s released checkpoints: a BBC article summarized level by level, ten more articles checked against BBC’s own summaries, two inputs that broke it, a Sudoku score, and a diffusion model for comparison. I also made a video on this, you can watch it right here: Autoregressive LLMs ChatGPT, Gemini, Llama predict one token at a time. Each new token depends on everything before it, so the steps can’t run side by side. More output means more steps, always. Diffusion language models came in to remove that queue. Think of a fill-in-the-blanks exam paper. The model starts with the whole answer as blanks, and in every round it fills many blanks at once and fixes the ones it got wrong earlier. No waiting in line, so it can be fast: Google DeepMind’s experimental Gemini Diffusion https://deepmind.google/models/gemini-diffusion/ samples at about 1,479 tokens per second. The cost is that diffusion needs many rounds, and every round looks at the whole answer again . Cut the rounds to save time and quality drops. I show that with a real diffusion model later in this post. NCM is a third way. NCM is really two models working together . 1. The compressor. It takes a piece of text and squeezes it into a list of numbers called codes . A code is just a position in a list, like ordering by item number in a restaurant with 4,096 dishes on the menu. If token IDs feel familiar, codes are the same kind of thing: not human-readable on their own. The key part is that the codes come in levels . The first level has only a few codes and carries the rough gist. Every level after that has more codes and stores only what the earlier levels missed . By the last level there’s enough detail to get the exact words back. This compressor is very good at its job: it rebuilds the original text exactly 98.8 to 100% of the time, depending on the task. That’s what the “concept” in the name means. It isn’t a human idea or a sentence. It’s the information sitting in the top, coarse levels. The name sounds a little grander than the thing, and that’s fine. 2. The writer. This is the model that generates. Like an LLM that only outputs token IDs, it only outputs codes. It writes all codes of level 1 in one go , then all codes of level 2, and so on. Inside a level nothing waits in line. The only waiting is between levels, because each level corrects the one above it. After the last level, the compressor’s decoder turns the codes into words. That’s the first moment any text exists. So the number of passes equals the number of levels, not the number of tokens. That’s the whole source of the speed. Where it does look promising: it needed about 4x fewer training tokens on OpenWebText 127B vs 500B+ to get its results, and it does very well on puzzles where every part of the answer depends on every other part. More on that in Step 5. Everything below runs on Stanford’s released code https://github.com/HazyResearch/ncm and checkpoints, on an NVIDIA RTX 4060 Laptop GPU. The decoding settings are the same ones the repo’s official eval uses. Before NCM, I ran MDLM 170M parameters, trained on OpenWebText , one of the diffusion models the paper compares against. Same prompt and seed, a 1,024-token canvas, once with 48 rounds and once with only 8: for steps in 48, 8 : print f"{steps} rounds, {res 'secs' } s: {text}\n" 48 rounds, 5.38 s: Scientists have found that Paris is receiving support from around the world in this regard. The numbers see many lines in the fight against pollution. To see these improvements other industrial solutions will be limited8 rounds, 1.38 s: Scientists have found that the grasp is of, is not innate. Humans function like adults, except of the many lines, the liver sense a leak and see essential closing to get released and be gone Neither is great writing; it’s a small model. But 48 rounds gives sentences, 8 rounds gives word salad . That’s the trade-off NCM is trying to escape: fewer rounds without falling apart. The XSum checkpoint is trained on BBC news: give it an article, it writes a one-line summary. Loading it means loading both halves: tk = VQVAETokenizer.from config cfg.tokenizer the compressor text <- codes net = NextConceptTransformer vqvae local=tk.vqvae, config=cfg.model .cuda the writerlevels = list tk.vqvae.vq.levels print "codes per level:", levels print "passes:", len levels , "| summary length in codes:", levels -1 codes per level: 4, 6, 10, 14, 20, 26, 32, 40, 48 passes: 9 | summary length in codes: 48writer: 88M params Nine levels, so nine passes for the whole summary . Level 1 has 4 codes, the last level has 48. Normally you only see the final line. Here I asked the decoder to turn every level into words. The input is a BBC article about a fasting-style diet that helped the pancreas heal itself in mice : codes = net.generate None, x, temperature=1.0, top k=1, top p=0.0, cfg scale=0.0, cfg null eos=EOS , z levels = tk.vqvae.vq.idx to zhats codes, return zlevels=True texts = to text pick tokens tk.vqvae.decoder z 0 .tolist for z in zs level 1 4 codes decades - eurozonelevel 2 6 codes habits - cancerlevel 3 10 codes discovered containing - cause.level 4 14 codes have that iodinere - type.level 5 20 codes have diet iodine emits theas - cause type.level 6 26 codes Scientists discovered diet that causes pancreas possibly root causes 2.level 7 32 codes Scientists discovered vegan diet reduces causes the pancre - possibly root of type diabetes.level 8 40 codes Scientists discovered a vegan diet reduces causes in the pancre - possibly cause root causes type 2 diabetes.level 9 48 codes Scientists have discovered a vegan diet that causes in the pancreas - possibly cause the causes of type 2 diabetes.passes: 9 time: 0.30 s article tokens: 479 Three things stand out. Level 1 has nothing to do with the article. “Decades, eurozone” is what four coarse codes look like when you force them into words. That’s exactly why streaming doesn’t work: the early levels are not a short, rough version of the answer you could show a user. By level 6 the topic is there : scientists, diet, pancreas, type 2. The words are still broken, but the model clearly knows what the article is about. The final line has the gist, and the details are off. The article describes a fasting-mimicking diet that resembles a vegan diet and helped regenerate pancreas cells. NCM says “a vegan diet that causes… type 2 diabetes”. Right neighbourhood, wrong claim. The whole thing took 9 passes and well under half a second : 0.30 s in this notebook run, and between 0.18 and 0.39 s across my other runs on the same laptop. I didn’t run an LLM on the same article, so I’m not putting a speed ratio next to that number. One article is an anecdote. So I took the first ten articles of the XSum test set and printed NCM’s line next to the summary BBC actually wrote. A few of them: 1 NCM: A man has been arrested in connection with a number of firearms ammunition and ammunition were seized in Edinburgh. BBC: A man has appeared in court after firearms, ammunition and cash were seized by police in Edinburgh. 6 NCM: British declining Houvenaghel has been sacked by the British Cycling for its "systematic failings" in the sport. BBC: A "medal at any cost" approach created a "culture of fear" at British Cycling, says former rider Wendy Houvenaghel. 7 NCM: A coffee coffee coach at a restaurant in the US to sell a coffee shop he owns to sell or buy off a sum. BBC: Have you heard the one about the computer programmer who bought a failing comedy club in Texas and turned it into a million dollar a year business? 9 NCM: Celtic manager Brendan Rodgers says he is excited to fans to his fans as he prepare to face Queens Park Rangers on Sunday. BBC: Manager Brendan Rodgers is sure Celtic can exploit the wide open spaces of Hampden when they meet Rangers in Sunday's League Cup semi-final.time per summary: 0.32-0.36 s, 9 passes each Counting all ten by hand: 8 of 10 got the topic right the right people, place and event . 2 of 10 were garbled article 5, about the Essilor and Luxottica merger, came out as invented brand names; article 7 became a coffee shop . Even the good ones bend facts: Houvenaghel wasn’t sacked, and Celtic were playing Rangers, not Queens Park Rangers. For an 88M research model that’s decent. It’s nowhere near what an LLM gives you , and the paper doesn’t claim it is. Two inputs from outside its training world: a short article about Voyager 1 and the NCM paper itself. for name in "voyager-1-one-light-day.pdf", "ncm-paper-2610.14162.pdf" : voyager-1-one-light-day.pdf 574 tokens, reads 512 NCM: The latest launch of space Nasa agency - Nasa's astronaut spacecraft mission - has launched its launch launch from the Nasa space astronauts astronaut - and an unmanned space mission.ncm-paper-2610.14162.pdf 30810 tokens, reads 512 NCM: acy-acy--Quible and analyticalocrographed, test, unique scenery, digital technology and tablets. Two lessons here. It’s a specialist : a space article becomes “launch launch… astronaut astronaut”, and a research paper becomes nonsense. And it only reads the first 512 tokens . The paper is 30,810 tokens long, so the model saw the title page and part of the abstract. Easy to miss if you only look at the output. In Sudoku you can’t fill row 1 and move on. Every number depends on numbers elsewhere on the board, which is bad news for a model that writes left to right. NCM looks at the whole board at every level. I ran the released 25M Sudoku checkpoint on all 2,000 hard test boards: php merged 1 shards - 2000 sudoku-hard outputs solved % solved cell % valid % 86.15 1723/2000 98.35 86.15 1,723 of 2,000 hard boards solved, 86.15% , which matches the paper’s 86.1%. A same-size autoregressive model solves about 1% in the paper’s table, and the strongest non-autoregressive baseline in the table 81.3%. 1. Keep building on LLMs. NCM is a research model, not a product. The checkpoints are small, task-specific and clearly behind same-size LLMs on open text and translation. 2. Read speed claims with “faster than what?” The 31x is against diffusion models, on one task, on one H100. Speed headlines in this space almost always need that question. 3. Watch it for “whole answer at once” problems. Puzzles, constraint-style outputs and tasks where every part depends on every other part are where coarse-to-fine already wins. 4. Watch the training-cost angle. Getting results with about 4x fewer training tokens matters as good data gets more expensive. Diffusion language models showed that writing doesn’t have to happen in a queue. NCM borrows its level-by-level idea from VAR, which did coarse-to-fine for images, and brings it to text. It isn’t the model you ship next sprint, but it’s a clear sign that not every piece of writing has to happen one token at a time . Credit to the Hazy Research and Scaling Intelligence teams at Stanford, with Together AI, for releasing the code and checkpoints so anyone can check these numbers on their own GPU. Video version: watch the NCM video on YouTube https://youtu.be/-nDR kE6Obg Everything above comes from one notebook. Here is all of it, so you can run it on your own GPU. It needs a CUDA GPU; the XSum summarizer fits easily in 8 GB. Setup, once. Clone the repo, install it, and download the released checkpoints and test data: git clone https://github.com/HazyResearch/ncm.git repopip install -r repo/requirements.txt pypdfcd repohf download hazyresearch/ncm-checkpoints --local-dir artifactshf download hazyresearch/ncm-datasets --repo-type dataset --local-dir datapython evaluate.py --config config/eval/sudoku-hard.yaml --checkpoint-dir artifacts/sudoku/hard --data-dir data/sudoku/hard --out eval/sudoku-hard-full The last command runs the Sudoku checkpoint on all 2,000 hard boards and saves the generations; the notebook only re-scores them. Then point NCM HOME at the folder that holds repo/, and put any PDFs you want summarized in demo/docs/ I saved a BBC article as bbc-article.pdf . Settings and paths python import os, sys, time, json, pickle, re, contextlib, io, warnings, subprocesswarnings.filterwarnings "ignore" PROJ = os.environ "NCM HOME" folder with repo/ git clone of HazyResearch/ncm , hf-cache/ and demo/docs/os.environ "HF HOME" = os.path.join PROJ, "hf-cache" os.environ "TRANSFORMERS VERBOSITY" = "error"REPO = os.path.join PROJ, "repo" sys.path.insert 0, REPO ; os.chdir REPO Load the XSum summarizer python import torchprint torch.cuda.get device name 0 from omegaconf import OmegaConffrom transformers import AutoTokenizerfrom models.next concept import NextConceptTransformerfrom tokenization.vqvae tokenizer import VQVAETokenizer cfg = OmegaConf.load "config/eval/xsum.yaml" cfg.tokenizer.vqvae.checkpoint path = "artifacts/xsum/tokenizer.pt"with contextlib.redirect stdout io.StringIO : tk = VQVAETokenizer.from config cfg.tokenizer the compressor text <- codes tk.vqvae = tk.vqvae.cuda .eval net = NextConceptTransformer vqvae local=tk.vqvae, config=cfg.model .cuda the writersd = torch.load "artifacts/xsum/generator.pt", map location="cpu", weights only=False net.load state dict sd "model" , strict=True ; net.eval levels = list tk.vqvae.vq.levels print "codes per level:", levels print "passes:", len levels , "| summary length in codes:", levels -1 print f"writer: {sum p.numel for p in net.parameters /1e6:.0f}M params" Decoding helpers same settings as the repo’s official eval helpers: the same decoding settings as the repo's official eval scripts/eval shard.py G = cfg.generationP = int cfg.model.prefix len EOS, PAD = tk.pre tokenizer.eos token id, tk.pre tokenizer.pad token idsrc tok = AutoTokenizer.from pretrained "gpt2" only to tokenize the articleALLOW = sd "inference" "token unigram" .double .cuda freq, mu = float G.frequency penalty , float G.frequency deficit ADJ = float v for v in str G.adjacency penalty .split "," def pick tokens logits : B, T, V = logits.shape lg = logits.double ; allow = ALLOW :V T None, : al, aw, ad = ADJ 0 , int ADJ 1 , ADJ 2 cnt = torch.zeros B, V, dtype=torch.float64, device=lg.device out = torch.empty B, T, dtype=torch.long, device=lg.device ; hist = for t in range T : step = lg :, t, : .clone step -= freq cnt - allow .clamp min 0.0 step += mu T - t / T allow - cnt .clamp min 0.0 for j, h in enumerate reversed hist -aw: : step.scatter add 1, h :, None , torch.full B, 1 , -al ad j, dtype=torch.float64, device=lg.device p = step.argmax -1 ; out :, t = p; hist.append p cnt.scatter add 1, p :, None , torch.ones B, 1, dtype=torch.float64, device=lg.device return out def to text ids : ids = list ids if EOS in ids: ids = ids :ids.index EOS return tk.pre tokenizer.decode t for t in ids if t not in EOS, PAD , skip special tokens=True .strip Summarize a PDF and decode every level python @torch.no grad def summarize article, every level=False : ids = src tok article "input ids" x = torch.tensor ids :P + PAD P :P , device="cuda" the model reads the first 512 tokens torch.cuda.synchronize ; t0 = time.perf counter codes = net.generate None, x, temperature=1.0, top k=1, top p=0.0, cfg scale=0.0, cfg null eos=EOS torch.cuda.synchronize ; secs = time.perf counter - t0 , z levels = tk.vqvae.vq.idx to zhats codes, return zlevels=True zs = z levels if every level else z levels -1: texts = to text pick tokens tk.vqvae.decoder z 0 .tolist for z in zs return texts, secs, len ids from pypdf import PdfReaderdef read pdf path : return re.sub r"\s+", " ", " ".join p.extract text or "" for p in PdfReader path .pages .strip article = read pdf os.path.join PROJ, "demo", "docs", "bbc-article.pdf" summarize article warm-uptexts, secs, n tok = summarize article, every level=True for k, n, t in enumerate zip levels, texts , 1 : print f"level {k} {n:2d} codes {t}" print f"\npasses: {len levels } time: {secs:.2f} s article tokens: {n tok}" Ten XSum test articles against BBC’s own summaries passages = pickle.load open "data/xsum/passages.pickle", "rb" refs = pickle.load open "data/xsum/reference summaries.pickle", "rb" def article text item : if isinstance item, str : return item art = list item :P ; art = art :art.index PAD if PAD in art else art return src tok.decode art def ref text r : return r if isinstance r, str else to text r times = for i in range 10 : s, , secs, = summarize article text passages i ; times.append secs print f" {i} NCM: {s}\n BBC: {ref text refs i }\n" print f"time per summary: {min times :.2f}-{max times :.2f} s, {len levels } passes each" Inputs that aren’t BBC news for name in "voyager-1-one-light-day.pdf", "ncm-paper-2610.14162.pdf" : s, , secs, n tok = summarize read pdf os.path.join PROJ, "demo", "docs", name print f"{name} {n tok} tokens, reads {min n tok, P } \n NCM: {s}\n" Score the Sudoku run r = subprocess.run sys.executable, "scripts/eval merge.py", "--config", "config/eval/sudoku-hard.yaml", "--glob", "eval/sudoku-hard-full/shard .pkl" , capture output=True, text=True, encoding="utf-8", env={ os.environ, "PYTHONIOENCODING": "utf-8"} print r.stdout.strip -600: The diffusion comparison in Step 0 uses the sampler from the MDLM repo https://github.com/kuleshov-group/mdlm with the kuleshov-group/mdlm-owt https://huggingface.co/kuleshov-group/mdlm-owt weights, so it isn’t repeated here. Sources Stanford NCM: The New Way of Generating Text 31x Faster https://pub.towardsai.net/stanford-ncm-next-concept-modeling-explained-76585414562f was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.