ByteDance Seed has published Cola2 on Hugging Face (we noticed it on November 6, 2026). It is a roughly 9B-parameter base model that does not write one token at a time. Instead, it plans text as compact continuous latents, generates them with a diffusion transformer, and decodes them into words. The model card reports an eight-task average of 60.10 after training on 346.03B source tokens. That is below Llama-3-8B (66.25), which saw more than 15T tokens. The point of Cola2 is the design, not the leaderboard.
If you build with open-weight models, this is a "watch and learn" release, not a "swap it into production" release. Below: what Cola2 does, what the numbers do and do not say, how to run it, and how it fits next to other diffusion-style language models we have covered.
TL;DR: the questions readers ask first #
| Question | Short answer |
|---|---|
| What is it? | A ~9B continuous-latent base model from ByteDance Seed (model card, GitHub README). |
| Is it a chat model? | No. The card says it is a base model, not an instruction-tuned assistant. |
| How was it trained? | 346.03B source tokens; the latent diffusion transformer is described as still unconverged. |
| How good is it? | 60.10 average on eight tasks, versus 66.25 for Llama-3-8B and 75.87 for Qwen3-8B-Base. |
| Where does it win? | HellaSwag (41.75 vs 39.21) and StoryCloze (80.42 vs 79.49) against Llama-3-8B. |
| License? | Code is Apache 2.0. Weights and tokenizer assets carry their own terms. |
| Can I fine-tune it? | The repo ships no training code, so not with ByteDance's own recipe. |
| Should I use it in a product? | Not yet. It is a research checkpoint. |
What is Cola2, in plain language? #
Most language models, including Llama and Qwen, are autoregressive. They predict the next token, append it, and repeat. Cola2 takes a different route. Per the model card, it has three parts:
- A dynamic-compression VAE. It groups text into semantic spans and encodes each span as one compact continuous latent vector.
- A block-causal Diffusion Transformer (DiT). About 4B parameters, trained with flow matching. It generates latent blocks, each conditioned on the blocks before it.
- A span-causal token decoder. It turns the latents back into discrete tokens, with latent feedback.
At inference time the model proposes a latent block, decodes tokens from it, and re-encodes the committed text before moving on. In other words, it plans a chunk of meaning first and spells it out second. The README describes the whole system as "a continuous-latent language model that co-adapts semantic compression and discrete text generation."
The paper behind it is titled "Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation" (arXiv 2610.31305).
Cola2 continuous latent language model overview of method and results Image: ByteDance Seed
How is Cola2 different from Cola DLM? #
Cola2 builds on an earlier ByteDance Seed project, Cola DLM (a continuous latent diffusion language model). The Cola DLM project page describes a Text VAE, a block-causal DiT that learns a prior over latents with flow matching, and a conditional decoder. Its authors report that, in strictly matched comparisons at about 2B parameters against autoregressive and LLaDA baselines across eight benchmarks, Cola DLM is competitive and still improving with compute (arXiv 2605.06548). Those are the authors' own claims.
Cola2's addition is the word "co-adaptive." The compression step (how many tokens go into one latent) and the discrete generation step are trained to adapt to each other rather than being fixed up front. The Cola2 README also places a compressed variant, Cola2-MDM, inside a controlled scaling study next to autoregressive models, LLaDA, Cola DLM and an uncompressed variant. There, the uncompressed Cola-MDM shows the strongest aggregate scaling, while the compressed Cola2-MDM converges more slowly but keeps narrowing its gap to LLaDA after roughly 150B tokens. That is an honest admission: compression costs quality at small training budgets, and the bet is that it pays off later.
What do the benchmark numbers actually say? #
The README reports Cola2's checkpoint at an eight-task average of 60.10. Each task score is the mean of three latent-noise sampled evaluations. The comparison table in the card:
| Model | 8-task average | Pretraining tokens |
|---|---|---|
| Llama-3-8B | 66.25 | 15T+ |
| LLaDA-8B-Base | 72.11 | 2.3T |
| Qwen2.5-7B | 75.08 | 18T |
| Qwen3-8B-Base | 75.87 | about 36T |
| Cola2 (about 9B) | 60.10 | 346B |
Three honest readings of this table:
- Cola2 loses on the headline number. It is 6 points under Llama-3-8B and about 15 under Qwen3-8B-Base. Anyone choosing a base model for quality should pick one of the others.
- The data gap is huge. The README says Cola2 used more than 43 times fewer pretraining tokens than Llama-3-8B. That is the argument for the design: a lot less data to get within reach. It does not prove the gap closes with more data.
- The table is not a controlled test. The card itself warns that the models use different recipes, tokenizers and token-counting methods, and that latent generation is stochastic. Treat it as context. The only controlled comparison is the scaling study mentioned above.
Cola2 does beat Llama-3-8B on two tasks: HellaSwag (41.75 vs 39.21) and StoryCloze (80.42 vs 79.49). Both are commonsense and story-continuation tasks, which fits a model that plans meaning in spans, though the card does not claim that explanation.
How does it relate to other diffusion language models? #
Diffusion for text is a live research thread. Mercury 2 is the commercial example: a diffusion LLM sold on speed, at about 1,000 tokens per second. LLaDA is the open diffusion baseline Cola2 is measured against. The difference is where the diffusion happens. LLaDA-style models denoise tokens directly. Cola DLM and Cola2 denoise in a continuous latent space and decode afterwards, which the Cola DLM authors call "latent prior transport."
For builders, the practical difference is simple. Mercury 2 is a product you call by API. Cola2 is a research checkpoint you run yourself to study the idea. They answer different questions.
If you want a sense of where open-weight models stand today for real work, see our top 10 open-weight models for a laptop and the Qwen3.8 Max open-weights release. Those are the models Cola2 is trailing in the table above.
How do I run Cola2? #
The README gives this path. It needs Python 3.10 or newer, PyTorch 2.2 or newer, Transformers 4.51 up to 4.x, and safetensors 0.4 or newer. A CUDA GPU with bfloat16 is recommended.
git clone https://github.com/ByteDance-Seed/Cola2
cd Cola2
python -m pip install -e .
export COLA2_MODEL_PATH=/path/to/Cola2
bash scripts/run_chat.sh
In Python, load the model with AutoModel and AutoTokenizer and set trust_remote_code=True, then call generate(). The default preset uses an 8-step Euler DiT schedule, DiT classifier-free guidance of 10, decoder guidance of -0.9, and max_new_tokens of 32. Pass a seeded torch.Generator if you need repeatable output, because generation starts from Gaussian noise.
Two safety notes. First, trust_remote_code=True runs code from the model repository, so read it before running it, and prefer an isolated environment. Second, the README says benchmark JSONL files are hosted separately on Hugging Face, so verify the checksums it provides.
What are the limits and open issues? #
The README is candid, and these points matter more than the score:
- Base model only. There is no instruction tuning, so do not expect chat quality or tool use. Our guide toopen-weight models that startups use to cut costs shows what production-ready looks like, and Cola2 is not that.
- Unconverged latent DiT. The card says the latent diffusion transformer is still unconverged. The reported number is a snapshot, not a ceiling.
- No training code. The repository has no training code, so you cannot reproduce the run or continue it with ByteDance's recipe.
- Non-deterministic by default. Greedy token selection still varies, because the latent starts from noise. The original paper seeds are not published, so local reruns may differ slightly from reported results.
- Split licensing. Repository code is Apache 2.0. Weights, tokenizer assets and benchmark datasets stay under their own terms. Check them before shipping anything.
- Short default output.
max_new_tokensdefaults to 32. Long-form generation is not what the release demonstrates.
Why does this matter for people who build with AI? #
Three reasons, none of them about replacing your current model.
1. It tests whether autoregression is the only good design. The Cola DLM authors argue that "autoregression is one self-consistent design choice, not the only one." If latent planning scales, models could generate several tokens of meaning per step, which would change how inference cost works. That is a hypothesis, and Cola2's numbers do not settle it. But a ~9B model at 60.10 after 346B tokens shows the approach is trainable at that scale.
2. It reduces the data needed to get started. The 43x token gap, if it holds with a fair comparison, matters for labs that cannot afford 15T-token runs. ByteDance Seed has shown a similar research habit elsewhere, for example in HarnessDev, where LLMs build their own agent harness.
3. It is a clean teaching example. A VAE, a diffusion transformer and a token decoder is a clear stack to study. If you teach or learn how generative models differ, Cola2 is a good case next to the architecture notes in our Kimi K3 architecture breakdown.
What should I do this week? #
- Researchers: read the arXiv paper, then run the README's chat script and compare outputs across seeds. Note the variance.
- Builders: do nothing in production. Keep using an instruction-tuned open model. Revisit if ByteDance releases an instruct version.
- Teams that need numbers: if you want to claim "Cola2 vs Qwen3" in a report, run your own controlled test. The card's table does not support it.
- Everyone: check the weights' license terms on the Hugging Face page before any redistribution.
Related reading #
Details are accurate as of November 6, 2026. Scores are the authors' own, from the model card and README; no independent replication was available at publication.