Lower loss, worse stories: small NCMs trained on TinyStories

A developer built ncm-mlx, an independent MLX implementation of Next-Concept Modeling (NCM), and trained small NCM models from scratch on TinyStories. The runs showed that reducing the corruption schedule improved teacher-forced clean code cross-entropy from about 2.25 to about 1.39 nats per code while worsening self-generated text, with GPT-2 large perplexity rising from 47.9 to 97.8 in seed 0 and 45.6 to 93.8 in seed 1. The best NCM run still scored roughly three times the perplexity of an autoregressive baseline built from the same transformer blocks (45.568 vs 14.311 in seed 1).

I trained small Next-Concept Modeling NCM models from scratch in MLX on TinyStories. They reconstruct held-out stories almost perfectly, but the text they write on their own is a lot worse than text from a plain autoregressive AR baseline built from the same transformer blocks. And the part I didn't expect: the training change that made the teacher-forced loss better made the generated text worse. Same result in two generator seeds. NCM is the method of Kumbong et al. https://arxiv.org/abs/2610.14162 . My project, ncm-mlx https://github.com/janmus1c/ncm-mlx , is an independent MLX implementation of it. The models here are my own small training runs, not reproductions of the authors' models. In an earlier post https://dev.to/janmusic/running-released-next-concept-modeling-checkpoints-on-apple-silicon-28mi I wrote about running their released checkpoints in MLX. Below I only use their TinyStories checkpoint as a reference point. NCM doesn't write one token at a time. A codec turns a 128-token text window into a stack of discrete codes, from one coarse code at the top down to fine detail at the bottom. A generator then makes a new stack one level at a time, predicting all codes of a level in parallel, and the codec decodes the finished stack back to text. Here the hierarchy has 12 levels. Level 0 is sampled from a stored unigram distribution, then 11 generator forward passes predict the rest. What I trained: Data is TinyStories V2 GPT-4 split at a pinned revision. Generation is unconditional, so every model writes 128-token story openings with no prompt. GPT-2 large, also pinned, scores how likely the text looks. Each model writes 1,280 outputs 256 for each of five sampling seeds, temperature 1, no top-k or top-p , and the score is the mean perplexity over the five sampling seeds. Lower is better. The codec gets 99.816% of non-padding tokens right on 4,096 held-out windows. So the weak numbers below come from the generator making up code stacks, not from the codec losing information. During generator training, part of the earlier-level codes the generator conditions on gets replaced with random codes. The targets stay clean. The idea is to prepare the generator for sampling time, where it conditions on its own earlier levels, and those can be wrong. I tried three schedules, with the corrupted fraction going down linearly from the first to the last level: Everything else stayed fixed: the frozen codec, tokenizer, architecture, optimizer and number of training windows. Each schedule was trained twice, with generator seeds 0 and 1, on the same frozen codec. Two numbers matter per model. Clean code CE is the generator's cross-entropy when every earlier level comes from the real text teacher forcing . Generative perplexity is the GPT-2 large score of text the model wrote by itself. | Model | Clean code CE nats/code , seed 0 / 1 | GPT-2 large perplexity, seed 0 / 1 | |---|---|---| | NCM, strong corruption | 2.24875 / 2.24909 | 47.897 / 45.568 | | NCM, reduced corruption | 1.66515 / 1.66106 | 54.923 / 55.114 | | NCM, no corruption | 1.38995 / 1.38638 | 97.776 / 93.833 | | AR baseline | n/a | 14.676 / 14.311 | | Released TinyStories checkpoint Kumbong et al. | different codec | 16.259 | | Held-out real stories | n/a | 8.529 | The two columns go in opposite directions. With less corruption, clean code CE improves from about 2.25 to about 1.39 nats per code. Over the same change, generated text gets worse, from 47.9 to 97.8 perplexity in seed 0 and from 45.6 to 93.8 in seed 1. Same ordering in both seeds. The best NCM run is still about three times the AR baseline's perplexity 45.568 vs 14.311 in seed 1 . To give a feel for what a perplexity near 48 means here, this is the start of the first output of the strong-corruption seed-0 model, not picked for quality: Once there was a little boy called Jack. Joe was was three years old and discover. One One day, he wanted to go and look for exploring. The story vocabulary is there, but broken grammar, repeated words and names that change mid-story are typical. Teacher-forced loss measures how well the generator predicts the next level when everything above it is correct. At sampling time nothing guarantees that. The generator conditions on levels it made itself, and a mistake in an early level gets carried into every later prediction. A model that only ever saw perfect earlier levels during training never learned what to do with its own mistakes. That train/test mismatch is usually called exposure bias, and it's the same problem scheduled sampling was proposed for in autoregressive models. The no-corruption model fits that picture: best teacher-forced loss and by far the worst text. Corruption shows the model imperfect context during training, and here more of it helped generation while hurting the loss. I think that's the most plausible reading, but these runs don't isolate it. Corruption strength, how it's spread across levels and optimization effects aren't separated, and I didn't measure per-level error with teacher-forced versus self-generated context. The full write-up is the E21 to E23 assessment https://janmus1c.github.io/ncm-mlx/research/e21-e23-assessment/ and Section 10 of working paper v0.12 https://doi.org/10.5281/zenodo.23897923 . The paper-v0.12 release https://github.com/janmus1c/ncm-mlx/releases/tag/paper-v0.12 has two archives next to the paper editions: ncm-mlx-text-training-source-d323f4f.tar.gz : the 44 training-source files exactly as run. ncm-mlx-tinystories-evidence-e21-e24.tar.gz : the audited evaluation reports, all generated samples and the per-output scorer records for all six runs. SHA256SUMS covers both. With them you can recompute every perplexity above, read every output or rescore the samples with another model. The text-training code is in the repository too. The evaluation driver that produced the reports isn't included, and I haven't verified a clean-checkout reproduction of the training runs yet. From the roadmap https://github.com/janmus1c/ncm-mlx roadmap , in this order: Nothing so far shows NCM will get there, not sure yet. What these runs did give me is a clear reason to report reconstruction, teacher-forced loss and free-running quality as separate numbers, because here they disagreed. AI assistance disclosure: AI coding assistants helped with implementation, experiment analysis and drafting. They are not authors, and this assistance is not independent verification or human review.