{"slug": "neomme-an-efficient-multimodal-native-and-multilingual-encoder", "title": "NeoMME*: an efficient Multimodal-native and Multilingual Encoder", "summary": "H Company released NeoMME, a family of 260M and 800M multilingual multimodal encoders that process text and images with a single bidirectional Transformer, trained from scratch with a masked discrete-diffusion objective. The models achieve state-of-the-art performance on the ViDoRe v3 Pareto frontier for nDCG@10 and model size, with the 260M model encoding about 51 pages per second on an NVIDIA L40S GPU, and reduce late-interaction index storage by 255× while retaining over 95% of baseline nDCG@10. All checkpoints are available under the Apache 2.0 license on Hugging Face Transformers.", "body_md": "Feature Extraction • 0.3B • Updated • 70\n\n# NeoMME: an efficient Multimodal-native and Multilingual Encoder\n\n[Team Article](/blog)\n\n## TL;DR\n\nWe introduce * NeoMME*, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models,\n\n*NeoMME*does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective.\n\nWe fine-tuned *NeoMME* for visual document retrieval using ColPali's page-image approach. *NeoMME*-Retriever returns dense and late-interaction embeddings in one forward pass. Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, or about twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of baseline nDCG@10.\n\n*NeoMME* is available in Hugging Face Transformers. We release all model checkpoints under the Apache 2.0 license.\n\n## Why another multimodal encoder?\n\nMany recent visual document retrievers are adapted from pretrained generative visual language models. A separately pretrained vision encoder produces visual features, which a projector maps into the language model's input space. A causal decoder then processes the combined image and text representations. Retrieval, classification, and token labeling do not generate text autoregressively, so they do not require a causal decoder or the parameter and compute overhead of this architecture.\n\n[ModernBERT](https://huggingface.co/blog/modernbert) brought efficient architecture and training improvements to bidirectional encoders. For visual document retrieval, [ModernVBERT](https://huggingface.co/blog/paultltc/modernvbert) applied a bidirectional ModernBERT-style text encoder while retaining a separate pretrained SigLIP2 vision tower. We wanted to push this even further by designing and training a multimodal encoder without having to carry over the parameter and compute overhead of a VLM.\n\n* NeoMME* (pronounced \"nee-oh-me\", IPA /ˈniː.oʊ.mi/) is a multilingual, multimodal foundation encoder that generates vector representations for input text and/or images using a single Transformer encoder. It is not based on an existing pretrained vision tower, text encoder, or text decoder.\n\nImages and text use the same computational path, so *NeoMME* can more easily support pretraining, fine-tuning, parallelization, and serving across both modalities.\n\n## *NeoMME* encoder backbone\n\n### One Transformer for images and text\n\n*NeoMME* comes in two sizes, [260M](https://huggingface.co/Hcompany/NeoMME-260M) and [800M](https://huggingface.co/Hcompany/NeoMME-800M). Both variants share the same architecture:\n\n**Native multimodal inputs:** text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected with a small MLP. Both enter the same Transformer encoder.**Dynamic image resolution:** images keep their aspect ratio and size. This allows the model to use more tokens on a high-resolution, information-dense document page than on a smaller image with less content.**Long bidirectional context:** both models have a context length of 16,384 tokens (enough for up to two standard 3840×2160 4K UHD images). Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention.**A modern encoder stack:*** NeoMME*uses recent encoder improvements such as grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs, among others.**Multilingual text:** we trained a BPE tokenizer with a 131k-token vocabulary from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.\n\n### Learning from images through masked text\n\nWe pretrain *NeoMME* from scratch as a discrete masked-diffusion text denoiser. For each text-only example, we sample a corruption rate uniformly between 0 and 1. Each eligible text token is then independently masked at that rate.\n\nMultimodal examples use corruption rates between 0.3 and 1. The image patches remain visible while *NeoMME* reconstructs masked text. With light masking, the model can often recover a missing word from the surrounding text alone. For example, \"cat\" is a plausible completion of \"The [MASK] sat on the mat,\" even without an image. But high masking forces the model to learn image-grounded descriptions with little to no signal from the non-masked input text tokens.\n\nPretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion tokens from text-only examples. This text budget is relatively small compared with ModernBERT's 2 trillion training token budget. Hence, we chose the NorMuon optimizer to improve data efficiency during training.\n\n## *NeoMME*-Retriever\n\nTo get a meaningful downstream evaluation of the backbone, we fine-tune *NeoMME* for visual document retrieval using the page-image methodology introduced by [ColPali](https://huggingface.co/blog/manu/colpali). While traditional text-based retrieval consists of retrieving text chunks, *NeoMME*-Retriever ranks document page screenshots and bypasses all the preprocessing OCR steps necessary to extract text from PDFs. Treating the pages as images preserves layout, charts, tables, font type and size, and other visual clues that cannot be captured even by a perfect OCR model.\n\n### A dual-head design for dense and late-interaction retrieval\n\n*NeoMME*-Retriever reuses the *NeoMME* backbone but adds two jointly trained heads on top of it for retrieval:\n\n- The dense head averages the backbone's hidden state vectors into a normalized vector (mean pooling). Dense embeddings are most common today: they are compact and work naturally with approximate nearest-neighbor (ANN) techniques for fast retrieval.\n- The late-interaction head projects each text token or image patch from the backbone's output hidden states to a 128-dimensional normalized vector. Compared to dense embeddings, the finer granularity preserves local matches between individual query tokens and image regions.\n\nOmar Khattab, who introduced late-interaction in ColBERT,\n\n[explains why]the term is more precise than \"multi-vector.\" It describes the granularity and learnability of the scoring function, not simply the number of stored vectors.To learn more about late-interaction, we recommend reading\n\n[this crash course]by Amélie Chatelain.\n\nOne *NeoMME*-Retriever forward pass returns both representations, which gives you flexibility no matter your use case and infrastructure. We recommend using late-interaction embeddings in general since they are more powerful and can be used easily with open-source libraries like [NextPlaid](https://github.com/lightonai/next-plaid). However, if you have a very large corpora, you can run a single forward pass with *NeoMME*-Retriever to get the dense embedding, retrieve a small number of documents through an ANN index, and then use late-interaction to rerank the retrieved candidates.\n\n### Competitive retrieval at compact model sizes\n\nWe report nDCG@10 on ViDoRe v3. *NeoMME*-Retriever-260M reaches 0.523, the highest score among evaluated models strictly below 800M parameters. It is within 0.002 nDCG@10 of ColQwen2.5 while using about 14× fewer parameters. *NeoMME*-Retriever-800M reaches 0.556, within 0.009 nDCG@10 of the similarly sized [Vultron Retriever Flash (0.8B)](https://huggingface.co/vultr/VultronRetrieverFlash-Qwen3.5-0.8B). Both *NeoMME*-Retriever models lie on the model-size Pareto frontier.\n\nViDoRe v1 and v2 use nDCG@5. On both benchmarks, *NeoMME*-Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. *NeoMME*-Retriever-800M outperforms ColPali v1.3 while using 3.6 times fewer parameters.\n\n| Model details | ViDoRe (nDCG@k) |\n|||\n|---|---|---|---|---|\n| Model | Params. | v3 (@10) | v2 (@5) | v1 (@5) |\n| <300M | ||||\n|\n\n†‡‡[ColSmol-256M](https://huggingface.co/vidore/colSmol-256M)†*NeoMME*-260M‡**0.523****0.522****0.860**[ColSmol-500M](https://huggingface.co/vidore/colSmol-500M)‡††[Vultron Flash](https://huggingface.co/vultr/VultronRetrieverFlash-Qwen3.5-0.8B)†**0.565****0.604****0.882*** NeoMME*-800M‡[ColQwen2.5-v0.2](https://huggingface.co/vidore/colqwen2.5-v0.2)†**0.524****0.601****0.895**[ColPali v1.3](https://huggingface.co/vidore/colpali-v1.3)†† Scores from MTEB. ‡ Results from our own evaluations.\n\n### Making high-resolution retrieval practical for late-interaction\n\nLate-interaction storage scales linearly with the number of vectors in the output embedding. Higher-resolution images contain more patches, so they produce larger embeddings. For example, a 2048×2048 square page produces embeddings containing 4,200 vectors with *NeoMME*-Retriever, or about 2.1 MB in float32. Across the ViDoRe v3 benchmark, the measured average is about 1.5 MB per document.\n\nTo reduce the storage footprint of the late-interaction index, we combine two complementary compression methods:\n\n[Hierarchical token pooling](https://www.answer.ai/posts/colbert-pooling.html)clusters similar document vectors in a given multi-vector embedding and replaces each cluster with its mean, hence reducing the number of vectors stored for each page.[Asymmetric quantization](https://www.mixedbread.com/blog/asymmetric-quant)quantizes document embeddings to int8 or binary. Because query embeddings are not stored and only generated on-the-fly, they can be kept at a higher precision.\n\nWe tested this setup on ViDoRe v3. With a pooling factor 10 and int8 queries and documents, storage decreased from about 1.5 MB to 39 kB per page, a 39× reduction, while keeping more than 99% of the baseline nDCG@10. A more aggressive configuration uses pooling factor 8, int8 queries, and binary documents. That version uses 6 kB per page (255× smaller) and keeps more than 95% of the original retrieval quality.\n\nUsers can pick a compression setting from that frontier based on storage budget and required retrieval quality.\n\n#### Fast inference for cheaper multimodal corpus indexing\n\nBefore you can search a corpus, a retriever model must turn your documents into embeddings, which will be stored in a vector store like Qdrant, Weaviate, or Milvus. Faster encoding makes building and adding new documents to the index faster, thus reducing the GPU uptime and compute cost required.\n\nSo we measured image encoding speeds for *NeoMME*-Retriever against other multimodal document retrievers. We used preprocessed image tensors and calibrated the batch size separately for each model and image size. At a matched 2048×2048 input size on one NVIDIA L40S, *NeoMME*-Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT's 26 pages per second. Both 260M and 800M *NeoMME*-Retriever models are also faster than the other models we compared on smaller input images.\n\n### Try *NeoMME*-Retriever yourself!\n\n*NeoMME*-Retriever ([260M](https://huggingface.co/Hcompany/NeoMME-260M-Retriever) and [800M](https://huggingface.co/Hcompany/NeoMME-800M-Retriever)) returns dense and multi-vector embeddings together. The example below scores two text queries against two document-page images with MeanMaxSim late interaction and dense cosine similarity.\n\n## Click to see the complete 🤗 `transformers`\n\nexample snippet\n\n```\n# accelerate is an optional dependency needed only when using device_map=\"auto\".\npip install -U accelerate \"transformers @ git+https://github.com/huggingface/transformers.git@main\" \"sentence-transformers>=6.0.0\"\npython\nfrom typing import Any, Literal\n\nimport requests\nimport torch\nfrom PIL import Image\nfrom sentence_transformers.util import cos_sim, mean_maxsim\n\nfrom transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor\n\ndef encode(\n    messages: list[list[dict[str, Any]]],\n    task: Literal[\"query\", \"document\"],\n) -> BatchFeature:\n    return processor.apply_chat_template(\n        messages,\n        task=task,\n        tokenize=True,\n        return_dict=True,\n        return_tensors=\"pt\",\n        processor_kwargs={\"padding\": \"longest\"},\n    )\n\nmodel_name = \"Hcompany/NeoMME-260M-Retriever\"\nprocessor = NeoMMEProcessor.from_pretrained(model_name)\nmodel = NeoMMEForRetrieval.from_pretrained(model_name, device_map=\"auto\")\n\n# Document images (our corpus)\nimage_urls = [\n    \"https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true\",\n    \"https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true\",\n]\ndocuments = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]\n\n# Queries\nqueries = [\n    \"Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?\",\n    \"Which hour of the day had the highest overall electricity generation in 2019?\",\n]\n\ndocument_messages = [\n    [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": document}]}] for document in documents\n]\nquery_messages = [[{\"role\": \"user\", \"content\": query}] for query in queries]\n\ninputs_documents = encode(document_messages, \"document\").to(model.device)\ninputs_text = encode(query_messages, \"query\").to(model.device)\n\nwith torch.inference_mode():\n    document_outputs = model(**inputs_documents)\n    query_outputs = model(**inputs_text)\n\nlate_scores = mean_maxsim(\n    query_outputs.embeddings,\n    document_outputs.embeddings,\n    a_mask=inputs_text[\"attention_mask\"],\n    b_mask=inputs_documents[\"attention_mask\"],\n)\ndense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings)\n\n# Expected: late_scores[0, 0] > late_scores[0, 1] and late_scores[1, 1] > late_scores[1, 0].\nprint(late_scores, dense_scores)\n```\n\n### Fine-tuning with Sentence Transformers\n\nWe provide separate [dense](https://huggingface.co/Hcompany/NeoMME-260M-Retriever-ST-dense) and [late-interaction](https://huggingface.co/Hcompany/NeoMME-260M-Retriever-ST-late) checkpoints for fine-tuning with [Sentence Transformers v6](https://huggingface.co/blog/multi-vector-encoder). Following the same pattern as text encoders such as ModernBERT, Sentence Transformers loads the backbone through [ NeoMMEModel](https://huggingface.co/docs/transformers/main/en/model_doc/neomme#transformers.NeoMMEModel) rather than the dual-head\n\n[class. Sentence Transformers currently supports one retrieval head per model, so each checkpoint lets you fine-tune the dense or late-interaction head independently. To train both heads together, use](https://huggingface.co/docs/transformers/main/en/model_doc/neomme#transformers.NeoMMEForRetrieval)\n\n`NeoMMEForRetrieval`\n\n`NeoMMEForRetrieval`\n\nwith a custom `Trainer`\n\n.###\n\nFrom retrieval to visual RAG\n\nVisual document retrieval can be used as the first stage of a visual retrieval-augmented generation (RAG) system. Unlike text RAG, which retrieves extracted text chunks, visual RAG retrieves the original page images and sends them to a visual language model. The model can then use tables, plots, diagrams, and page layout that text extraction may flatten or omit. Here is how visual RAG works:\n\n- Indexing: Convert each PDF page to an image, generate an embedding with a retrieval model, and store the embeddings in a vector store.\n- Retrieval: Generate an embedding for the user's query with the same model and retrieve the top-k most relevant pages.\n- Generation: Append the images after the query in the chat message (\n*e.g.*,`{query}{img_1}{img_2}...{img_k}`\n\n) and send it to a VLM to generate the answer.\n\nYou can test visual RAG directly with *NeoMME*-Retriever in our HF Space: 🤗 [tonywu71/neomme-retriever-demo](https://huggingface.co/spaces/tonywu71/neomme-retriever-demo).\n\n## Conclusion\n\n*NeoMME* replaces separate pretrained image and text encoders with one long-context bidirectional Transformer. We train it from scratch to process both multilingual text tokens and raw 32×32 image patches.\n\n*NeoMME*-Retriever is a fine-tuned version of *NeoMME* for visual document retrieval. One forward pass produces both dense and late-interaction representations. The 260M model outperforms all evaluated models strictly below 800M parameters and, at a matched 2048×2048 input size, encodes pages at about 2× ColModernVBERT's throughput. To reduce the large storage footprint of late-interaction embeddings for high-resolution documents, we experimented with hierarchical token pooling and asymmetric quantization and managed to reduce the late-interaction embeddings from roughly 1.5 MB to 6 kB per page, a 255× compression, while retaining more than 95% of the baseline nDCG@10.\n\nWe release all *NeoMME* model checkpoints and a day-zero Hugging Face Transformers implementation to allow practitioners to build efficient multimodal and multilingual representation models on top of our work.\n\n## Acknowledgements\n\n*NeoMME* began as a side quest between two good friends. We worked with limited time and compute, and we decided to share the results so the community can build on them. We thank H Company for supporting the work and providing the compute used to train *NeoMME*.\n\n## Citation\n\n```\n@misc{lac2026neommesingletowermultimodalnativemultilingual,\n      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},\n      author={Aurélien Lac and Tony Wu},\n      year={2026},\n      eprint={2609.01657},\n      archivePrefix={arXiv},\n      primaryClass={cs.IR},\n      url={https://arxiv.org/abs/2609.01657},\n}\n```\n\n", "url": "https://wpnews.pro/news/neomme-an-efficient-multimodal-native-and-multilingual-encoder", "canonical_source": "https://huggingface.co/blog/Hcompany/neomme", "published_at": "2026-09-03 13:13:48+00:00", "updated_at": "2026-09-03 13:27:00.125241+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-products"], "entities": ["H Company", "NeoMME", "Hugging Face Transformers", "ModernBERT", "ModernVBERT", "SigLIP2", "ColPali", "ViDoRe"], "alternates": {"html": "https://wpnews.pro/news/neomme-an-efficient-multimodal-native-and-multilingual-encoder", "markdown": "https://wpnews.pro/news/neomme-an-efficient-multimodal-native-and-multilingual-encoder.md", "text": "https://wpnews.pro/news/neomme-an-efficient-multimodal-native-and-multilingual-encoder.txt", "jsonld": "https://wpnews.pro/news/neomme-an-efficient-multimodal-native-and-multilingual-encoder.jsonld"}}