Meituan dNaViT: Open Discrete Vision Tokens for Unified Multimodal Models

Meituan LongCat published dNaViT, a discrete vision tokenizer for unified multimodal models, in arXiv:2611.12426 on November 9, 2026, with open code in the LongCat-Next GitHub repository and weights on Hugging Face. The tokenizer pairs a frozen encoder with a Residual Modulator (ResMod) and grouped product quantization at a downsample factor of 32, and the paper reports 28.12 dB PSNR and 0.81 SSIM at 512×512 reconstruction, 885 on OCRBench when paired with Qwen3-8B, and GenEval2 49.5 and Qwen-Image-Bench 52.17 for generation. The authors report the shared vocabulary serves reconstruction, classification, multimodal understanding, generation and editing, and stress data efficiency, with results described as author-reported until reproduced.

Meituan LongCat published dNaViT in arXiv:2611.12426 https://arxiv.org/abs/2611.12426?ref=explainx on November 9, 2026 , with open code in github.com/meituan-longcat/LongCat-Next https://github.com/meituan-longcat/LongCat-Next?ref=explainx and model assets on Hugging Face https://huggingface.co/meituan-longcat/LongCat-Next?ref=explainx . The project asks a builder-relevant question: can finite discrete codes carry both semantics and pixels , so vision trains like language next-token prediction ? The answer in the paper is yes, with caveats : a shared vocabulary can serve reconstruction, classification, multimodal understanding, generation, and editing when training balances semantic and pixel branches . TL;DR | Piece | Detail | |---|---| | Architecture | Frozen encoder + ResMod + grouped product quantization | | Downsample | f = 32 in main reporting | | Reconstruction | 28.12 dB PSNR at 512×512 in paper tables | | Understanding | Pairs with Qwen3-8B in cited OCRBench-style rows | | Code | LongCat-Next/dNaViT/ directory + HF weights | | Lineage | Extends LongCat-Next https://explainx.ai/blog/meituan-longcat-2-5-launch-api-1m-context-10-million-free-tokens-anthropic-compatible-endpoint-2026 discrete multimodal work | The discrete vision bottleneck Most multimodal stacks bolt continuous vision encoders onto language cores . Training infrastructure, sampling tricks, and speculative decoding patterns diverge. Chameleon , Emu3 , and LongCat-Next instead push VQ-style indices that slot into autoregressive training. The hard part is information loss : aggressive quantization hurts understanding benchmarks; gentle quantization hurts generation fidelity . dNaViT''s design targets that tradeoff explicitly. Architecture in plain language Frozen encoder supplies strong semantic features without retraining the whole ViT. Residual Modulator ResMod fuses early and late encoder layers , then applies a zero-initialized scale-and-shift residual so training starts near the pretrained representation and gradually adds reconstruction-aware detail . Grouped product quantization expands effective codebook capacity without lengthening the spatial token sequence — important for inference cost when you treat images as token streams . Dual-branch losses supervise semantic reconstruction and pixel reconstruction , trained in three stages continuous init → discrete alignment → joint optimization . Optional Refiner modules improve perceptual quality without changing underlying discrete codes, per the README in LongCat-Next . Open-weight distribution graphic for Meituan dNaViT tokenizer code in LongCat-Next Reported results from the paper Treat numbers as author-reported until you reproduce: - Understanding : with Qwen3-8B , paper tables cite 885 on OCRBench and competitive document reasoning vs Qwen3-VL-8B baselines. - Generation : GenEval2 49.5 and Qwen-Image-Bench 52.17 in listed runs. - Reconstruction : PSNR 28.12 and SSIM 0.81 at 512×512 . The authors stress data efficiency : strong scores with smaller training budgets than some compared systems — relevant if you fine-tune tokenizers on proprietary imagery. Multimodal fusion graphic for dNaViT vision and language token streams in one model Training stages why three phases matter The paper describes progressive training to stop semantic and pixel objectives from fighting each other : 1. Continuous initialization — align ResMod on frozen encoder features without quantization noise. 2. Discrete feature alignment — introduce grouped product quantization while keeping semantic anchors stable. 3. Joint optimization — balance representation loss and pixel reconstruction loss with gradient interference controls. Skipping stage two is a common failure mode in VQ-VAE projects: codes collapse to a tiny subset of indices, wrecking understanding metrics. dNaViT''s claim is that ResMod + grouped PQ keeps code utilization high enough for downstream LMs . Discrete vs continuous vision in one table | Approach | Pros | Cons | |---|---|---| | Continuous CLIP/SigLIP encoder | Strong zero-shot understanding | Hard to unify with AR image generation | | VQ-VAE classic | Pure autoregressive images | Historically weak on OCR/documents | | dNaViT discrete | Shared tokenizer for edit + generate + understand | Training complexity, codebook tuning | If you build RAG over screenshots or UI agents , discrete tokens can let you log visual tokens beside text in one JSONL stream — useful for multimodal knowledge agents https://explainx.ai/blog/multimodal-knowledge-agent-mka-benchmark-google-2026 -style evals. Reported benchmark rows selected Treat as author tables pending reproduction : | Area | Example metric cited | |---|---| | Reconstruction | 28.12 PSNR , 0.81 SSIM at 512² | | Docs / OCR | 885 OCRBench with Qwen3-8B pairing | | Generation | GenEval2 49.5 | | Editing / bilingual text | Qwen-Image-Bench 52.17 | The paper argues these land with less training data than some baselines — if true, tokenizer quality not brute-force scale drove the gain. Your finetune team should still run held-out product screenshots before believing OCR numbers. How to try dNaViT this week 1. Clone LongCat-Next https://github.com/meituan-longcat/LongCat-Next?ref=explainx and read dNaViT/README . 2. Install dNaViT-local dependencies separate from sft/ per repo layout . 3. Run reconstruction eval scripts on a held-out image folder before you plug codes into a generative head. 4. Compare token rate and VRAM against your current continuous CLIP/SigLIP pipeline on the same downstream task . If you already serve LongCat-2.5 APIs https://explainx.ai/blog/meituan-longcat-2-5-launch-api-1m-context-10-million-free-tokens-anthropic-compatible-endpoint-2026 , dNaViT is the tokenizer research lane , not a drop-in API swap. Connections across explainx.ai coverage - LongCat sparse attention https://explainx.ai/blog/longcat-2-5-1-6-trillion-agent-model-2026 — same lab, different layer of the stack. - Minimax H3 open image weights https://explainx.ai/blog/minimax-h3-image-official-open-weights-16-megapixel-2026 — alternative open vision generation line. - Meta Muse wearable agents https://explainx.ai/blog/meta-muse-apple-watch-wear-os-agent-2026 — consumer multimodal, mostly closed. - WorldFoundry world-model benchmark https://explainx.ai/blog/worldfoundry-open-world-model-benchmark-infrastructure-2026 — when your tokens must predict physics , not just pixels. Integration patterns for multimodal apps Three patterns map cleanly to production: Understanding-only — swap your vision tower input with dNaViT codes + embedding table , keep LM frozen initially. Generation — train or adapt an autoregressive head predicting visual token indices, similar to LongCat-Next training recipes in the parent repo. Editing — condition on partial code sequences plus text instructions; the paper lists a dedicated editing benchmark row. For agent harnesses , export token IDs + spatial metadata so tools can crop regions without re-encoding pixels each step — important for Gemini widget demos https://explainx.ai/blog/gemini-intelligence-november-2026-rambler-create-my-widget-2026 -style interfaces you may compare against. Licensing and deployment notes README states MIT for core LongCat-Next weights with dNaViT/ under Apache-2.0 . Legal review should catch: - Trademark restrictions on Meituan branding in your UI. - Model card attribution when you ship fine-tuned tokenizers . - Export control if you deploy in regulated regions — orthogonal to code license but common in China-origin open weights reviews. FAQ for practitioners Do I need LongCat-Next weights to use the tokenizer? Read dNaViT/README : reconstruction eval can run with tokenizer checkpoints before you train a full DiNA stack. Will this run on Apple Silicon? Expect PyTorch + CUDA assumptions in research code; plan a cloud GPU finetune unless README lists MPS support when you clone. How many visual tokens per 1024² image? Scales with downsample factor f=32 in the paper''s main config — count tokens before you promise real-time mobile generation. Can I pair dNaViT with Claude or GPT vision APIs? Tokenizer is open ; API pairing requires you to export embeddings into your own serving graph, not a plug-in swap. Is this competitive with Gemini image stacks? Benchmarks in the paper claim strong editing and bilingual text scores; verify on your locale and font coverage. LongCat ecosystem context Meituan has been stacking open multimodal pieces across explainx.ai coverage: - LongCat-2.5 API https://explainx.ai/blog/meituan-longcat-2-5-launch-api-1m-context-10-million-free-tokens-anthropic-compatible-endpoint-2026 for agent serving. - LongCat 2.5 1.6T agent model https://explainx.ai/blog/longcat-2-5-1-6-trillion-agent-model-2026 for sparse attention at scale. - LongCat 2.0 MoE coding agent https://explainx.ai/blog/longcat-2-0-open-source-moe-coding-agent-2026 for software-focused agents. dNaViT is the vision token layer underneath native multimodal training — not a chat API by itself. Product teams should slot it into data pipelines first, user-facing APIs second. Hardware and throughput planning Discrete autoregressive vision increases sequence length versus continuous patch embeddings. Before production: - Estimate tokens per megapixel using the paper''s f=32 downsampling story. - Profile embedding table memory for codebook size after grouped product quantization . - Compare batch latency against your current SigLIP + cross-attn stack on identical prompts. If throughput fails, hybridize: continuous encoder for retrieval , dNaViT codes for generation , accepting two forward passes until unified checkpoints mature. What to watch on Hugging Face and GitHub Watch meituan-longcat/LongCat-Next for: - Tokenizer checkpoint revisions and vocab size changes that break saved indices. - Issue tracker threads on ResMod training instability or codebook collapse . - Community forks that pair dNaViT with Qwen3.5 or DeepSeek LMs — useful canaries for transfer before you commit. When benchmarks update, re-run your document-OCR suite first; that is where discrete vision tokenizers historically underperform continuous towers. Comparison to other open discrete vision lines Chameleon and Emu3 proved unified autoregressive multimodal training is possible; dNaViT stresses native resolution and grouped product quantization rather than a single VQ bottleneck. Against Minimax H3 image weights, Meituan optimizes tokenizer generality first, single-shot T2I scores second — pick your integration path based on whether you need one LM vocabulary or best still-image FID today. Re-benchmark quarterly if you ship both understanding and generation in one product surface. Limits and honest skepticism - Benchmark tables need your own domain shift tests medical, industrial, low-light . - Discrete autoregressive vision still pays sampling cost at high resolution — count tokens per megapixel before production. - License split MIT vs Apache-2.0 in dNaViT/ matters for enterprise redistribution — read both LICENSE files. Pin tokenizer commit hashes in your internal model registry the same way you pin LM revisions , because discrete indices are not backward compatible when codebooks change over time. Related reading arXiv v1 and LongCat-Next README checked November 11, 2026.