# Meituan dNaViT: Open Discrete Vision Tokens for Unified Multimodal Models

> Source: <https://explainx.ai/blog/meituan-dnavit-discrete-vision-tokenizer-longcat-next-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-11 00:00:00+00:00

**Meituan LongCat** published **dNaViT** in **[arXiv:2611.12426](https://arxiv.org/abs/2611.12426?ref=explainx)** on **November 9, 2026**, with **open code** in **[github.com/meituan-longcat/LongCat-Next](https://github.com/meituan-longcat/LongCat-Next?ref=explainx)** and model assets on **[Hugging Face](https://huggingface.co/meituan-longcat/LongCat-Next?ref=explainx)**. The project asks a builder-relevant question: can **finite discrete codes** carry **both semantics and pixels**, so vision trains like **language next-token prediction**?

The answer in the paper is **yes, with caveats**: a **shared vocabulary** can serve **reconstruction, classification, multimodal understanding, generation, and editing** when training balances **semantic and pixel branches**.

## TL;DR

| Piece | Detail | 
|---|---|
| Architecture | Frozen encoder + **ResMod** +**grouped product quantization** | 
| Downsample | **f = 32** in main reporting | 
| Reconstruction | **28.12 dB PSNR** at**512×512** in paper tables | 
| Understanding | Pairs with **Qwen3-8B** in cited OCRBench-style rows | 
| Code | `LongCat-Next/dNaViT/` directory + HF weights | 
| Lineage | Extends **[LongCat-Next](https://explainx.ai/blog/meituan-longcat-2-5-launch-api-1m-context-10-million-free-tokens-anthropic-compatible-endpoint-2026)** discrete multimodal work | 

## The discrete vision bottleneck

Most multimodal stacks **bolt continuous vision encoders** onto **language cores**. Training infrastructure, sampling tricks, and **speculative decoding** patterns diverge. **Chameleon**, **Emu3**, and **LongCat-Next** instead push **VQ-style indices** that slot into **autoregressive** training.

The hard part is **information loss**: aggressive quantization hurts **understanding** benchmarks; gentle quantization hurts **generation fidelity**. dNaViT''s design targets that tradeoff explicitly.

## Architecture in plain language

**Frozen encoder** supplies strong **semantic features** without retraining the whole ViT.

**Residual Modulator (ResMod)** fuses **early and late encoder layers**, then applies a **zero-initialized scale-and-shift** residual so training starts near the pretrained representation and gradually adds **reconstruction-aware detail**.

**Grouped product quantization** expands **effective codebook capacity** without lengthening the **spatial token sequence** — important for **inference cost** when you treat images as **token streams**.

**Dual-branch losses** supervise **semantic reconstruction** and **pixel reconstruction**, trained in **three stages** (continuous init → discrete alignment → joint optimization).

Optional **Refiner** modules improve **perceptual quality** without changing underlying discrete codes, per the README in **LongCat-Next**.

Open-weight distribution graphic for Meituan dNaViT tokenizer code in LongCat-Next

## Reported results (from the paper)

Treat numbers as **author-reported** until you reproduce:

- **Understanding** : with**Qwen3-8B** , paper tables cite**885** on**OCRBench** and competitive document reasoning vs**Qwen3-VL-8B** baselines.
- **Generation** :**GenEval2 49.5** and**Qwen-Image-Bench 52.17** in listed runs.
- **Reconstruction** :**PSNR 28.12** and**SSIM 0.81** at**512×512** .

The authors stress **data efficiency**: strong scores with **smaller training budgets** than some compared systems — relevant if you **fine-tune tokenizers** on proprietary imagery.

Multimodal fusion graphic for dNaViT vision and language token streams in one model

## Training stages (why three phases matter)

The paper describes **progressive training** to stop semantic and pixel objectives from **fighting each other**:

1. **Continuous initialization** — align ResMod on frozen encoder features without quantization noise.
2. **Discrete feature alignment** — introduce**grouped product quantization** while keeping semantic anchors stable.
3. **Joint optimization** — balance**representation loss** and**pixel reconstruction loss** with gradient interference controls.

Skipping stage two is a common failure mode in VQ-VAE projects: codes collapse to a **tiny subset** of indices, wrecking **understanding** metrics. dNaViT''s claim is that **ResMod + grouped PQ** keeps **code utilization** high enough for **downstream LMs**.

## Discrete vs continuous vision in one table

| Approach | Pros | Cons | 
|---|---|---|
| Continuous CLIP/SigLIP encoder | Strong zero-shot understanding | Hard to unify with AR image generation | 
| VQ-VAE (classic) | Pure autoregressive images | Historically weak on OCR/documents | 
| **dNaViT discrete** | Shared tokenizer for **edit + generate + understand** | Training complexity, codebook tuning | 

If you build **RAG over screenshots** or **UI agents**, discrete tokens can let you **log visual tokens** beside text in one JSONL stream — useful for **[multimodal knowledge agents](https://explainx.ai/blog/multimodal-knowledge-agent-mka-benchmark-google-2026)**-style evals.

## Reported benchmark rows (selected)

Treat as **author tables pending reproduction**:

| Area | Example metric cited | 
|---|---|
| Reconstruction | **28.12 PSNR** ,**0.81 SSIM** at 512² | 
| Docs / OCR | **885 OCRBench** with Qwen3-8B pairing | 
| Generation | **GenEval2 49.5** | 
| Editing / bilingual text | **Qwen-Image-Bench 52.17** | 

The paper argues these land with **less training data** than some baselines — if true, **tokenizer quality** not **brute-force scale** drove the gain. Your finetune team should still run **held-out product screenshots** before believing OCR numbers.

## How to try dNaViT this week

1. Clone **[LongCat-Next](https://github.com/meituan-longcat/LongCat-Next?ref=explainx)** and read**`dNaViT/README`** .
2. Install **dNaViT-local dependencies** (separate from`sft/` per repo layout).
3. Run **reconstruction eval** scripts on a held-out image folder before you plug codes into a generative head.
4. Compare **token rate** and**VRAM** against your current**continuous CLIP/SigLIP** pipeline on the**same downstream task** .

If you already serve **[LongCat-2.5 APIs](https://explainx.ai/blog/meituan-longcat-2-5-launch-api-1m-context-10-million-free-tokens-anthropic-compatible-endpoint-2026)**, dNaViT is the **tokenizer research lane**, not a drop-in API swap.

## Connections across explainx.ai coverage

- **[LongCat sparse attention](https://explainx.ai/blog/longcat-2-5-1-6-trillion-agent-model-2026)** — same lab, different layer of the stack.
- **[Minimax H3 open image weights](https://explainx.ai/blog/minimax-h3-image-official-open-weights-16-megapixel-2026)** — alternative open vision generation line.
- **[Meta Muse wearable agents](https://explainx.ai/blog/meta-muse-apple-watch-wear-os-agent-2026)** — consumer multimodal, mostly closed.
- **[WorldFoundry world-model benchmark](https://explainx.ai/blog/worldfoundry-open-world-model-benchmark-infrastructure-2026)** — when your tokens must predict**physics** , not just pixels.

## Integration patterns for multimodal apps

Three patterns map cleanly to production:

**Understanding-only** — swap your vision tower input with **dNaViT codes + embedding table**, keep LM frozen initially.

**Generation** — train or adapt an **autoregressive head** predicting visual token indices, similar to **LongCat-Next** training recipes in the parent repo.

**Editing** — condition on **partial code sequences** plus text instructions; the paper lists a dedicated **editing benchmark** row.

For **agent harnesses**, export **token IDs + spatial metadata** so tools can **crop regions** without re-encoding pixels each step — important for **[Gemini widget demos](https://explainx.ai/blog/gemini-intelligence-november-2026-rambler-create-my-widget-2026)**-style interfaces you may compare against.

## Licensing and deployment notes

README states **MIT** for core **LongCat-Next** weights with **`dNaViT/` under Apache-2.0**. Legal review should catch:

- **Trademark** restrictions on Meituan branding in your UI.
- **Model card** attribution when you ship**fine-tuned tokenizers** .
- **Export control** if you deploy in regulated regions — orthogonal to code license but common in**China-origin open weights** reviews.

## FAQ for practitioners

**Do I need LongCat-Next weights to use the tokenizer?** Read **`dNaViT/README`**: reconstruction eval can run with tokenizer checkpoints before you train a full **DiNA** stack.

**Will this run on Apple Silicon?** Expect **PyTorch + CUDA** assumptions in research code; plan a **cloud GPU** finetune unless README lists **MPS** support when you clone.

**How many visual tokens per 1024² image?** Scales with **downsample factor f=32** in the paper''s main config — count tokens before you promise **real-time** mobile generation.

**Can I pair dNaViT with Claude or GPT vision APIs?** Tokenizer is **open**; API pairing requires you to **export embeddings** into your own serving graph, not a plug-in swap.

**Is this competitive with Gemini image stacks?** Benchmarks in the paper claim strong **editing and bilingual text** scores; verify on **your** locale and font coverage.

## LongCat ecosystem context

Meituan has been stacking **open multimodal** pieces across explainx.ai coverage:

- **[LongCat-2.5 API](https://explainx.ai/blog/meituan-longcat-2-5-launch-api-1m-context-10-million-free-tokens-anthropic-compatible-endpoint-2026)** for agent serving.
- **[LongCat 2.5 1.6T agent model](https://explainx.ai/blog/longcat-2-5-1-6-trillion-agent-model-2026)** for sparse attention at scale.
- **[LongCat 2.0 MoE coding agent](https://explainx.ai/blog/longcat-2-0-open-source-moe-coding-agent-2026)** for software-focused agents.

dNaViT is the **vision token layer** underneath **native multimodal** training — not a chat API by itself. Product teams should slot it into **data pipelines** first, **user-facing APIs** second.

## Hardware and throughput planning

Discrete autoregressive vision increases **sequence length** versus continuous patch embeddings. Before production:

- Estimate **tokens per megapixel** using the paper''s**f=32** downsampling story.
- Profile **embedding table memory** for codebook size after**grouped product quantization** .
- Compare **batch latency** against your current**SigLIP + cross-attn** stack on identical prompts.

If throughput fails, hybridize: **continuous encoder for retrieval**, **dNaViT codes for generation**, accepting **two forward passes** until unified checkpoints mature.

## What to watch on Hugging Face and GitHub

Watch **meituan-longcat/LongCat-Next** for:

- **Tokenizer checkpoint** revisions and**vocab size** changes that break saved indices.
- **Issue tracker** threads on**ResMod** training instability or**codebook collapse** .
- **Community forks** that pair dNaViT with**Qwen3.5** or**DeepSeek** LMs — useful canaries for**transfer** before you commit.

When benchmarks update, re-run **your** document-OCR suite first; that is where discrete vision tokenizers historically underperform continuous towers.

## Comparison to other open discrete vision lines

**Chameleon** and **Emu3** proved unified autoregressive multimodal training is possible; **dNaViT** stresses **native resolution** and **grouped product quantization** rather than a single VQ bottleneck. Against **Minimax H3** image weights, Meituan optimizes **tokenizer generality** first, **single-shot T2I scores** second — pick your integration path based on whether you need **one LM vocabulary** or **best still-image FID** today. Re-benchmark quarterly if you ship both understanding and generation in one product surface.

## Limits and honest skepticism

- **Benchmark tables** need your own**domain shift** tests (medical, industrial, low-light).
- **Discrete autoregressive vision** still pays**sampling cost** at high resolution — count**tokens per megapixel** before production.
- **License split** (**MIT** vs**Apache-2.0** in`dNaViT/` ) matters for**enterprise redistribution** — read both LICENSE files.

Pin **tokenizer commit hashes** in your internal model registry the same way you pin **LM revisions**, because discrete indices are not backward compatible when codebooks change over time.

## Related reading

*arXiv v1 and LongCat-Next README checked November 11, 2026.*