Every LLM you use today writes the same way: one token, then the next, then the next. A 1,000-token answer means about 1,000 steps in a line, and step 501 cannot start until step 500 is done. Your GPU can do a lot of work in parallel, but text generation keeps it waiting in a queue. That’s a big reason long answers feel slow.
Stanford’s new Next-Concept Modeling (NCM) tries a different route. It sketches the whole answer roughly first, then adds detail level by level, like a blurry WhatsApp photo that gets sharper as it loads. On OpenWebText it writes 1,024 tokens in 32 passes instead of 1,024.
The paper headline is “31x faster”. That number is real, but most people read it wrong, and I’ll come to that.
The first half of this post explains the idea in plain words. The second half is what I actually ran on an RTX 4060 laptop GPU with Stanford’s released checkpoints: a BBC article summarized level by level, ten more articles checked against BBC’s own summaries, two inputs that broke it, a Sudoku score, and a diffusion model for comparison.
I also made a video on this, you can watch it right here:
Autoregressive LLMs (ChatGPT, Gemini, Llama) predict one token at a time. Each new token depends on everything before it, so the steps can’t run side by side. More output means more steps, always.
Diffusion language models came in to remove that queue. Think of a fill-in-the-blanks exam paper. The model starts with the whole answer as blanks, and in every round it fills many blanks at once and fixes the ones it got wrong earlier. No waiting in line, so it can be fast: Google DeepMind’s experimental Gemini Diffusion samples at about 1,479 tokens per second.
The cost is that diffusion needs many rounds, and every round looks at the whole answer again. Cut the rounds to save time and quality drops. I show that with a real diffusion model later in this post.
NCM is a third way.
NCM is really two models working together.
1. The compressor. It takes a piece of text and squeezes it into a list of numbers called codes. A code is just a position in a list, like ordering by item number in a restaurant with 4,096 dishes on the menu. If token IDs feel familiar, codes are the same kind of thing: not human-readable on their own.
The key part is that the codes come in levels. The first level has only a few codes and carries the rough gist. Every level after that has more codes and stores only what the earlier levels missed. By the last level there’s enough detail to get the exact words back. This compressor is very good at its job: it rebuilds the original text exactly 98.8 to 100% of the time, depending on the task.
That’s what the “concept” in the name means. It isn’t a human idea or a sentence. It’s the information sitting in the top, coarse levels. The name sounds a little grander than the thing, and that’s fine.
2. The writer. This is the model that generates. Like an LLM that only outputs token IDs, it only outputs codes. It writes all codes of level 1 in one go, then all codes of level 2, and so on. Inside a level nothing waits in line. The only waiting is between levels, because each level corrects the one above it.
After the last level, the compressor’s decoder turns the codes into words. That’s the first moment any text exists.
So the number of passes equals the number of levels, not the number of tokens. That’s the whole source of the speed.
Where it does look promising: it needed about 4x fewer training tokens on OpenWebText (127B vs 500B+) to get its results, and it does very well on puzzles where every part of the answer depends on every other part. More on that in Step 5.
Everything below runs on Stanford’s released code and checkpoints, on an NVIDIA RTX 4060 Laptop GPU. The decoding settings are the same ones the repo’s official eval uses.
Before NCM, I ran MDLM (170M parameters, trained on OpenWebText), one of the diffusion models the paper compares against. Same prompt and seed, a 1,024-token canvas, once with 48 rounds and once with only 8:
for steps in [48, 8]: print(f"{steps} rounds, {res['secs']} s: {text}\n")
48 rounds, 5.38 s: Scientists have found that Paris is receiving support from around the world in this regard. The numbers see many lines in the fight against pollution. To see these improvements other industrial solutions will be limited8 rounds, 1.38 s: Scientists have found that the grasp is of, is not innate. Humans function like adults, except of the many lines, the liver sense a leak and see essential closing to get released and be gone
Neither is great writing; it’s a small model. But 48 rounds gives sentences, 8 rounds gives word salad. That’s the trade-off NCM is trying to escape: fewer rounds without falling apart.
The XSum checkpoint is trained on BBC news: give it an article, it writes a one-line summary. it means both halves:
tk = VQVAETokenizer.from_config(cfg.tokenizer) # the compressor (text <-> codes)net = NextConceptTransformer(vqvae_local=tk.vqvae, config=cfg.model).cuda() # the writerlevels = list(tk.vqvae.vq.levels)print("codes per level:", levels)print("passes:", len(levels), "| summary length in codes:", levels[-1])
codes per level: [4, 6, 10, 14, 20, 26, 32, 40, 48]passes: 9 | summary length in codes: 48writer: 88M params
Nine levels, so nine passes for the whole summary. Level 1 has 4 codes, the last level has 48.
Normally you only see the final line. Here I asked the decoder to turn every level into words. The input is a BBC article about a fasting-style diet that helped the pancreas heal itself (in mice):
codes = net.generate(None, x, temperature=1.0, top_k=1, top_p=0.0, cfg_scale=0.0, cfg_null_eos=EOS)_, z_levels = tk.vqvae.vq.idx_to_zhats(codes, return_zlevels=True)texts = [to_text(pick_tokens(tk.vqvae.decoder(z))[0].tolist()) for z in zs]
level 1 ( 4 codes) decades - eurozonelevel 2 ( 6 codes) habits - cancerlevel 3 (10 codes) discovered containing - cause.level 4 (14 codes) have that iodinere - type.level 5 (20 codes) have diet iodine emits theas - cause type.level 6 (26 codes) Scientists discovered diet that causes pancreas possibly root causes 2.level 7 (32 codes) Scientists discovered vegan diet reduces causes the pancre - possibly root of type diabetes.level 8 (40 codes) Scientists discovered a vegan diet reduces causes in the pancre - possibly cause root causes type 2 diabetes.level 9 (48 codes) Scientists have discovered a vegan diet that causes in the pancreas - possibly cause the causes of type 2 diabetes.passes: 9 time: 0.30 s article tokens: 479
Three things stand out.
Level 1 has nothing to do with the article. “Decades, eurozone” is what four coarse codes look like when you force them into words. That’s exactly why streaming doesn’t work: the early levels are not a short, rough version of the answer you could show a user.
By level 6 the topic is there: scientists, diet, pancreas, type 2. The words are still broken, but the model clearly knows what the article is about.
The final line has the gist, and the details are off. The article describes a fasting-mimicking diet that resembles a vegan diet and helped regenerate pancreas cells. NCM says “a vegan diet that causes… type 2 diabetes”. Right neighbourhood, wrong claim.
The whole thing took 9 passes and well under half a second: 0.30 s in this notebook run, and between 0.18 and 0.39 s across my other runs on the same laptop. I didn’t run an LLM on the same article, so I’m not putting a speed ratio next to that number.
One article is an anecdote. So I took the first ten articles of the XSum test set and printed NCM’s line next to the summary BBC actually wrote. A few of them:
[1] NCM: A man has been arrested in connection with a number of firearms ammunition and ammunition were seized in Edinburgh. BBC: A man has appeared in court after firearms, ammunition and cash were seized by police in Edinburgh.[6] NCM: British declining Houvenaghel has been sacked by the British Cycling for its "systematic failings" in the sport. BBC: A "medal at any cost" approach created a "culture of fear" at British Cycling, says former rider Wendy Houvenaghel.[7] NCM: A coffee coffee coach at a restaurant in the US to sell a coffee shop he owns to sell or buy off a sum. BBC: Have you heard the one about the computer programmer who bought a failing comedy club in Texas and turned it into a million dollar a year business?[9] NCM: Celtic manager Brendan Rodgers says he is excited to fans to his fans as he prepare to face Queens Park Rangers on Sunday. BBC: Manager Brendan Rodgers is sure Celtic can exploit the wide open spaces of Hampden when they meet Rangers in Sunday's League Cup semi-final.time per summary: 0.32-0.36 s, 9 passes each
Counting all ten by hand: 8 of 10 got the topic right (the right people, place and event). 2 of 10 were garbled (article 5, about the Essilor and Luxottica merger, came out as invented brand names; article 7 became a coffee shop). Even the good ones bend facts: Houvenaghel wasn’t sacked, and Celtic were playing Rangers, not Queens Park Rangers.
For an 88M research model that’s decent. It’s nowhere near what an LLM gives you, and the paper doesn’t claim it is.
Two inputs from outside its training world: a short article about Voyager 1 and the NCM paper itself.
for name in ["voyager-1-one-light-day.pdf", "ncm-paper-2610.14162.pdf"]:
voyager-1-one-light-day.pdf (574 tokens, reads 512) NCM: The latest launch of space Nasa agency - Nasa's astronaut spacecraft mission - has launched its launch launch from the Nasa space astronauts astronaut - and an unmanned space mission.ncm-paper-2610.14162.pdf (30810 tokens, reads 512) NCM: acy-acy--Quible and analyticalocrographed, test, unique scenery, digital technology and tablets.
Two lessons here. It’s a specialist: a space article becomes “launch launch… astronaut astronaut”, and a research paper becomes nonsense. And it only reads the first 512 tokens. The paper is 30,810 tokens long, so the model saw the title page and part of the abstract. Easy to miss if you only look at the output.
In Sudoku you can’t fill row 1 and move on. Every number depends on numbers elsewhere on the board, which is bad news for a model that writes left to right. NCM looks at the whole board at every level.
I ran the released 25M Sudoku checkpoint on all 2,000 hard test boards:
merged 1 shards -> 2000 sudoku-hard outputs solved % solved cell % valid % 86.15 1723/2000 98.35 86.15
1,723 of 2,000 hard boards solved, 86.15%, which matches the paper’s 86.1%. A same-size autoregressive model solves about 1% in the paper’s table, and the strongest non-autoregressive baseline in the table 81.3%.
1. Keep building on LLMs. NCM is a research model, not a product. The checkpoints are small, task-specific and clearly behind same-size LLMs on open text and translation.
2. Read speed claims with “faster than what?” The 31x is against diffusion models, on one task, on one H100. Speed headlines in this space almost always need that question.
3. Watch it for “whole answer at once” problems. Puzzles, constraint-style outputs and tasks where every part depends on every other part are where coarse-to-fine already wins.
4. Watch the training-cost angle. Getting results with about 4x fewer training tokens matters as good data gets more expensive.
Diffusion language models showed that writing doesn’t have to happen in a queue. NCM borrows its level-by-level idea from VAR, which did coarse-to-fine for images, and brings it to text. It isn’t the model you ship next sprint, but it’s a clear sign that not every piece of writing has to happen one token at a time. Credit to the Hazy Research and Scaling Intelligence teams at Stanford, with Together AI, for releasing the code and checkpoints so anyone can check these numbers on their own GPU.
Video version: watch the NCM video on YouTube
Everything above comes from one notebook. Here is all of it, so you can run it on your own GPU. It needs a CUDA GPU; the XSum summarizer fits easily in 8 GB.
Setup, once. Clone the repo, install it, and download the released checkpoints and test data:
git clone https://github.com/HazyResearch/ncm.git repopip install -r repo/requirements.txt pypdfcd repohf download hazyresearch/ncm-checkpoints --local-dir artifactshf download hazyresearch/ncm-datasets --repo-type dataset --local-dir datapython evaluate.py --config config/eval/sudoku-hard.yaml --checkpoint-dir artifacts/sudoku/hard --data-dir data/sudoku/hard --out eval/sudoku-hard-full
The last command runs the Sudoku checkpoint on all 2,000 hard boards and saves the generations; the notebook only re-scores them. Then point NCM_HOME at the folder that holds repo/, and put any PDFs you want summarized in demo/docs/ (I saved a BBC article as bbc-article.pdf).
Settings and paths
import os, sys, time, json, pickle, re, contextlib, io, warnings, subprocesswarnings.filterwarnings("ignore")PROJ = os.environ["NCM_HOME"] # folder with repo/ (git clone of HazyResearch/ncm), hf-cache/ and demo/docs/os.environ["HF_HOME"] = os.path.join(PROJ, "hf-cache")os.environ["TRANSFORMERS_VERBOSITY"] = "error"REPO = os.path.join(PROJ, "repo")sys.path.insert(0, REPO); os.chdir(REPO)
Load the XSum summarizer
import torchprint(torch.cuda.get_device_name(0))from omegaconf import OmegaConffrom transformers import AutoTokenizerfrom models.next_concept import NextConceptTransformerfrom tokenization.vqvae_tokenizer import VQVAETokenizer cfg = OmegaConf.load("config/eval/xsum.yaml")cfg.tokenizer.vqvae.checkpoint_path = "artifacts/xsum/tokenizer.pt"with contextlib.redirect_stdout(io.StringIO()): tk = VQVAETokenizer.from_config(cfg.tokenizer) # the compressor (text <-> codes) tk.vqvae = tk.vqvae.cuda().eval() net = NextConceptTransformer(vqvae_local=tk.vqvae, config=cfg.model).cuda() # the writersd = torch.load("artifacts/xsum/generator.pt", map_location="cpu", weights_only=False)net.load_state_dict(sd["model"], strict=True); net.eval() levels = list(tk.vqvae.vq.levels)print("codes per level:", levels)print("passes:", len(levels), "| summary length in codes:", levels[-1])print(f"writer: {sum(p.numel() for p in net.parameters())/1e6:.0f}M params")
Decoding helpers (same settings as the repo’s official eval)
Summarize a PDF and decode every level
@torch.no_grad()def summarize(article, every_level=False): ids = src_tok(article)["input_ids"] x = torch.tensor([(ids[:P] + [PAD] * P)[:P]], device="cuda") # the model reads the first 512 tokens torch.cuda.synchronize(); t0 = time.perf_counter() codes = net.generate(None, x, temperature=1.0, top_k=1, top_p=0.0, cfg_scale=0.0, cfg_null_eos=EOS) torch.cuda.synchronize(); secs = time.perf_counter() - t0 _, z_levels = tk.vqvae.vq.idx_to_zhats(codes, return_zlevels=True) zs = z_levels if every_level else z_levels[-1:] texts = [to_text(pick_tokens(tk.vqvae.decoder(z))[0].tolist()) for z in zs] return texts, secs, len(ids) from pypdf import PdfReaderdef read_pdf(path): return re.sub(r"\s+", " ", " ".join(p.extract_text() or "" for p in PdfReader(path).pages)).strip() article = read_pdf(os.path.join(PROJ, "demo", "docs", "bbc-article.pdf"))summarize(article) # warm-uptexts, secs, n_tok = summarize(article, every_level=True)for k, (n, t) in enumerate(zip(levels, texts), 1): print(f"level {k} ({n:2d} codes) {t}")print(f"\npasses: {len(levels)} time: {secs:.2f} s article tokens: {n_tok}")
Ten XSum test articles against BBC’s own summaries
passages = pickle.load(open("data/xsum/passages.pickle", "rb"))refs = pickle.load(open("data/xsum/reference_summaries.pickle", "rb")) def article_text(item): if isinstance(item, str): return item art = list(item[:P]); art = art[:art.index(PAD)] if PAD in art else art return src_tok.decode(art) def ref_text(r): return r if isinstance(r, str) else to_text(r) times = []for i in range(10): (s,), secs, _ = summarize(article_text(passages[i])); times.append(secs) print(f"[{i}] NCM: {s}\n BBC: {ref_text(refs[i])}\n")print(f"time per summary: {min(times):.2f}-{max(times):.2f} s, {len(levels)} passes each")
Inputs that aren’t BBC news
for name in ["voyager-1-one-light-day.pdf", "ncm-paper-2610.14162.pdf"]: (s,), secs, n_tok = summarize(read_pdf(os.path.join(PROJ, "demo", "docs", name))) print(f"{name} ({n_tok} tokens, reads {min(n_tok, P)})\n NCM: {s}\n")
Score the Sudoku run
r = subprocess.run([sys.executable, "scripts/eval_merge.py", "--config", "config/eval/sudoku-hard.yaml", "--glob", "eval/sudoku-hard-full/shard_*.pkl"], capture_output=True, text=True, encoding="utf-8", env={**os.environ, "PYTHONIOENCODING": "utf-8"})print(r.stdout.strip()[-600:])
The diffusion comparison in Step 0 uses the sampler from the MDLM repo with the kuleshov-group/mdlm-owt weights, so it isn’t repeated here.
Sources
Stanford NCM: The New Way of Generating Text 31x Faster was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.