Meituan LongCat published dNaViT in arXiv:2611.12426 on November 9, 2026, with open code in github.com/meituan-longcat/LongCat-Next and model assets on Hugging Face. The project asks a builder-relevant question: can finite discrete codes carry both semantics and pixels, so vision trains like language next-token prediction?
The answer in the paper is yes, with caveats: a shared vocabulary can serve reconstruction, classification, multimodal understanding, generation, and editing when training balances semantic and pixel branches.
TL;DR #
| Piece | Detail |
|---|---|
| Architecture | Frozen encoder + ResMod +grouped product quantization |
| Downsample | f = 32 in main reporting |
| Reconstruction | 28.12 dB PSNR at512×512 in paper tables |
| Understanding | Pairs with Qwen3-8B in cited OCRBench-style rows |
| Code | LongCat-Next/dNaViT/ directory + HF weights |
| Lineage | Extends LongCat-Next discrete multimodal work |
The discrete vision bottleneck #
Most multimodal stacks bolt continuous vision encoders onto language cores. Training infrastructure, sampling tricks, and speculative decoding patterns diverge. Chameleon, Emu3, and LongCat-Next instead push VQ-style indices that slot into autoregressive training.
The hard part is information loss: aggressive quantization hurts understanding benchmarks; gentle quantization hurts generation fidelity. dNaViT''s design targets that tradeoff explicitly.
Architecture in plain language #
Frozen encoder supplies strong semantic features without retraining the whole ViT.
Residual Modulator (ResMod) fuses early and late encoder layers, then applies a zero-initialized scale-and-shift residual so training starts near the pretrained representation and gradually adds reconstruction-aware detail.
Grouped product quantization expands effective codebook capacity without lengthening the spatial token sequence — important for inference cost when you treat images as token streams.
Dual-branch losses supervise semantic reconstruction and pixel reconstruction, trained in three stages (continuous init → discrete alignment → joint optimization).
Optional Refiner modules improve perceptual quality without changing underlying discrete codes, per the README in LongCat-Next.
Open-weight distribution graphic for Meituan dNaViT tokenizer code in LongCat-Next
Reported results (from the paper) #
Treat numbers as author-reported until you reproduce:
- Understanding : withQwen3-8B , paper tables cite885 onOCRBench and competitive document reasoning vsQwen3-VL-8B baselines.
- Generation :GenEval2 49.5 andQwen-Image-Bench 52.17 in listed runs.
- Reconstruction :PSNR 28.12 andSSIM 0.81 at512×512 .
The authors stress data efficiency: strong scores with smaller training budgets than some compared systems — relevant if you fine-tune tokenizers on proprietary imagery.
Multimodal fusion graphic for dNaViT vision and language token streams in one model
Training stages (why three phases matter) #
The paper describes progressive training to stop semantic and pixel objectives from fighting each other:
- Continuous initialization — align ResMod on frozen encoder features without quantization noise.
- Discrete feature alignment — introducegrouped product quantization while keeping semantic anchors stable.
- Joint optimization — balancerepresentation loss andpixel reconstruction loss with gradient interference controls.
Skipping stage two is a common failure mode in VQ-VAE projects: codes collapse to a tiny subset of indices, wrecking understanding metrics. dNaViT''s claim is that ResMod + grouped PQ keeps code utilization high enough for downstream LMs.
Discrete vs continuous vision in one table #
| Approach | Pros | Cons |
|---|---|---|
| Continuous CLIP/SigLIP encoder | Strong zero-shot understanding | Hard to unify with AR image generation |
| VQ-VAE (classic) | Pure autoregressive images | Historically weak on OCR/documents |
| dNaViT discrete | Shared tokenizer for edit + generate + understand | Training complexity, codebook tuning |
If you build **RAG over screenshots** or **UI agents**, discrete tokens can let you **log visual tokens** beside text in one JSONL stream — useful for **[multimodal knowledge agents](https://explainx.ai/blog/multimodal-knowledge-agent-mka-benchmark-google-2026)**-style evals.
## Reported benchmark rows (selected)
Treat as author tables pending reproduction:
| Area | Example metric cited |
|---|---|
| Reconstruction | 28.12 PSNR ,0.81 SSIM at 512² |
| Docs / OCR | 885 OCRBench with Qwen3-8B pairing |
| Generation | GenEval2 49.5 |
| Editing / bilingual text | Qwen-Image-Bench 52.17 |
The paper argues these land with less training data than some baselines — if true, tokenizer quality not brute-force scale drove the gain. Your finetune team should still run held-out product screenshots before believing OCR numbers.
How to try dNaViT this week #
- Clone LongCat-Next and read**
dNaViT/README** . - Install dNaViT-local dependencies (separate from
sft/per repo layout). - Run reconstruction eval scripts on a held-out image folder before you plug codes into a generative head.
- Compare token rate andVRAM against your currentcontinuous CLIP/SigLIP pipeline on thesame downstream task .
If you already serve LongCat-2.5 APIs, dNaViT is the tokenizer research lane, not a drop-in API swap.
Connections across explainx.ai coverage #
- **[LongCat sparse attention](https://explainx.ai/blog/longcat-2-5-1-6-trillion-agent-model-2026)** — same lab, different layer of the stack.
- **[Minimax H3 open image weights](https://explainx.ai/blog/minimax-h3-image-official-open-weights-16-megapixel-2026)** — alternative open vision generation line.
- **[Meta Muse wearable agents](https://explainx.ai/blog/meta-muse-apple-watch-wear-os-agent-2026)** — consumer multimodal, mostly closed.
- **[WorldFoundry world-model benchmark](https://explainx.ai/blog/worldfoundry-open-world-model-benchmark-infrastructure-2026)** — when your tokens must predict**physics** , not just pixels.
Integration patterns for multimodal apps #
Three patterns map cleanly to production:
Understanding-only — swap your vision tower input with dNaViT codes + embedding table, keep LM frozen initially.
Generation — train or adapt an autoregressive head predicting visual token indices, similar to LongCat-Next training recipes in the parent repo.
Editing — condition on partial code sequences plus text instructions; the paper lists a dedicated editing benchmark row.
For agent harnesses, export token IDs + spatial metadata so tools can crop regions without re-encoding pixels each step — important for Gemini widget demos-style interfaces you may compare against.
Licensing and deployment notes #
README states MIT for core LongCat-Next weights with dNaViT/ under Apache-2.0. Legal review should catch:
- Trademark restrictions on Meituan branding in your UI.
- Model card attribution when you shipfine-tuned tokenizers .
- Export control if you deploy in regulated regions — orthogonal to code license but common inChina-origin open weights reviews.
FAQ for practitioners #
Do I need LongCat-Next weights to use the tokenizer? Read dNaViT/README: reconstruction eval can run with tokenizer checkpoints before you train a full DiNA stack.
Will this run on Apple Silicon? Expect PyTorch + CUDA assumptions in research code; plan a cloud GPU finetune unless README lists MPS support when you clone.
How many visual tokens per 1024² image? Scales with downsample factor f=32 in the paper''s main config — count tokens before you promise real-time mobile generation.
Can I pair dNaViT with Claude or GPT vision APIs? Tokenizer is open; API pairing requires you to export embeddings into your own serving graph, not a plug-in swap.
Is this competitive with Gemini image stacks? Benchmarks in the paper claim strong editing and bilingual text scores; verify on your locale and font coverage.
LongCat ecosystem context #
Meituan has been stacking open multimodal pieces across explainx.ai coverage:
- **[LongCat-2.5 API](https://explainx.ai/blog/meituan-longcat-2-5-launch-api-1m-context-10-million-free-tokens-anthropic-compatible-endpoint-2026)** for agent serving.
- **[LongCat 2.5 1.6T agent model](https://explainx.ai/blog/longcat-2-5-1-6-trillion-agent-model-2026)** for sparse attention at scale.
- **[LongCat 2.0 MoE coding agent](https://explainx.ai/blog/longcat-2-0-open-source-moe-coding-agent-2026)** for software-focused agents.
dNaViT is the vision token layer underneath native multimodal training — not a chat API by itself. Product teams should slot it into data pipelines first, user-facing APIs second.
Hardware and throughput planning #
Discrete autoregressive vision increases sequence length versus continuous patch embeddings. Before production:
- Estimate tokens per megapixel using the paper''sf=32 downsampling story.
- Profile embedding table memory for codebook size aftergrouped product quantization .
- Compare batch latency against your currentSigLIP + cross-attn stack on identical prompts.
If throughput fails, hybridize: continuous encoder for retrieval, dNaViT codes for generation, accepting two forward passes until unified checkpoints mature.
What to watch on Hugging Face and GitHub #
Watch meituan-longcat/LongCat-Next for:
- Tokenizer checkpoint revisions andvocab size changes that break saved indices.
- Issue tracker threads onResMod training instability orcodebook collapse .
- Community forks that pair dNaViT withQwen3.5 orDeepSeek LMs — useful canaries fortransfer before you commit.
When benchmarks update, re-run your document-OCR suite first; that is where discrete vision tokenizers historically underperform continuous towers.
Comparison to other open discrete vision lines #
Chameleon and Emu3 proved unified autoregressive multimodal training is possible; dNaViT stresses native resolution and grouped product quantization rather than a single VQ bottleneck. Against Minimax H3 image weights, Meituan optimizes tokenizer generality first, single-shot T2I scores second — pick your integration path based on whether you need one LM vocabulary or best still-image FID today. Re-benchmark quarterly if you ship both understanding and generation in one product surface.
Limits and honest skepticism #
- Benchmark tables need your owndomain shift tests (medical, industrial, low-light).
- Discrete autoregressive vision still payssampling cost at high resolution — counttokens per megapixel before production.
- License split (MIT vsApache-2.0 in
dNaViT/) matters forenterprise redistribution — read both LICENSE files.
Pin tokenizer commit hashes in your internal model registry the same way you pin LM revisions, because discrete indices are not backward compatible when codebooks change over time.
Related reading #
arXiv v1 and LongCat-Next README checked November 11, 2026.