{"slug": "multi-vector-late-interaction-embedding-models-with-sentence-transformers", "title": "Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers", "summary": "Hugging Face's Sentence Transformers library v6.0 adds a fourth model type, MultiVectorEncoder, enabling ColBERT-style late interaction retrieval. The update allows loading PyLate, Stanford-NLP ColBERT, and colpali-engine checkpoints directly, supporting stronger retrieval and state-of-the-art visual document retrieval without OCR. Multi-vector models keep one vector per token and use MaxSim scoring, trading larger index size for improved token-level matching.", "body_md": "Sentence Similarity • 0.1B • Updated • 205k • 201\n\n# Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers\n\n[Update on GitHub](https://github.com/huggingface/blog/blob/main/multi-vector-encoder.md)\n\n[Sentence Transformers](https://sbert.net/)is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v6.0 update, it gains a fourth model type:\n\n`MultiVectorEncoder`\n\n, for ColBERT-style late interaction retrieval. Any [PyLate](https://github.com/lightonai/pylate)checkpoint and any\n\n[Stanford-NLP ColBERT](https://github.com/stanford-futuredata/ColBERT)checkpoint loads straight into it, and\n\n[colpali-engine](https://github.com/illuin-tech/colpali)models for visual document retrieval can be used too, through the same familiar API you already use for dense, sparse, and reranker models.\n\nWhere a regular embedding model compresses a whole text into one vector, a multi-vector model keeps **one vector per token** and scores query against document with the MaxSim operator. That preserves token-level matching information that a single vector has to average away, which usually means stronger retrieval at the cost of a bigger index. It's also the state of the art for visual document retrieval, where a text query is matched against page images directly, with no OCR step in between.\n\nIn this blogpost, we'll show you how to use these models: loading the various checkpoint formats, encoding and scoring, plugging them into a search stack, running them on page images, and keeping the index affordable. Everything below runs on a plain `pip install -U sentence-transformers`\n\n.\n\n## Table of Contents\n\n[What are Multi-Vector Models?](#what-are-multi-vector-models)[Installation](#installation)[Loading a Model](#loading-a-model)[Encoding Queries and Documents](#encoding-queries-and-documents)[Scoring with MaxSim](#scoring-with-maxsim)[Semantic Search](#semantic-search)[Retrieve and Rerank](#retrieve-and-rerank)[Indexing](#indexing)[Visual Document Retrieval](#visual-document-retrieval)[Audio Retrieval](#audio-retrieval)[Video Retrieval](#video-retrieval)[Interpretability](#interpretability)[Token Pooling](#token-pooling)[Speeding Up Inference](#speeding-up-inference)[Evaluating a Model](#evaluating-a-model)[Coming from PyLate or colpali-engine](#coming-from-pylate-or-colpali-engine)[Supported Models](#supported-models)[Acknowledgements](#acknowledgements)[Additional Resources](#additional-resources)\n\n## What are Multi-Vector Models?\n\nA dense embedding model reads a text and returns a single fixed-size vector. Everything the model noticed has to fit in those 384, 768, or 1024 numbers, and similarity is one dot product between two such summaries. This works remarkably well, but the compression is lossy in a specific way: a rare entity, an exact identifier, or one crucial clause in a long passage all have to compete for room in the same vector. A query with several requirements at once runs into the same wall. For \"green sofa with wooden legs and rounded cushions\", a single vector has to blend all four into one point, so a green sofa with the wrong legs ends up sitting close to the one you actually asked for.\n\nA multi-vector model (also called a late-interaction or ColBERT-style model, after the [ColBERT paper](https://arxiv.org/abs/2004.12832)) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9x128 matrix, not a 1x128 vector.\n\nThe interaction between query and document is then deferred until scoring time, which is where the name \"late interaction\" comes from. A cross-encoder interacts early: both texts go through the model together, which is accurate but leaves nothing to precompute, since every document has to be re-encoded for each new query. A bi-encoder, which is what the dense embedding model above is, barely interacts at all (one dot product between two finished summaries), and that is exactly what lets you encode a collection once and query it fast. Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token, which leaves far more room for the two to interact.\n\n### The MaxSim Operator\n\nScoring uses MaxSim: for each query token, take its highest similarity against any document token, then sum those maxima across the query.\n\nBecause the token embeddings are L2-normalized, each of those dot products is a cosine similarity in `[-1, 1]`\n\n, so the whole sum lands within `[-num_query_tokens, num_query_tokens]`\n\n.\n\nYou can read the operator as a soft alignment: every query token points at the one document token that best explains it, and the score is how well the document supports the query overall.\n\nThe alignment doesn't have to be lexical, since the token embeddings are contextualized. Encode \"Where do penguins live?\" against \"Penguins inhabit Antarctica.\" with [ lightonai/mLateOn](https://huggingface.co/lightonai/mLateOn) and the query token\n\n`live`\n\nfinds its best match on `inhabit`\n\nat 0.94, a word it shares no characters with! That is the thing lexical retrieval cannot do, BM25 and its relatives need the term itself, so synonyms and paraphrases slip past them. Dense embedding models bridge that gap as well, of course. What late interaction adds is that it does so without giving up the other direction: when an exact match is what matters (a product code, a surname, a function name), MaxSim still has that token sitting there on its own, where a single-vector model had to average it in with everything else. It isn't one-to-one either, since several query tokens routinely settle on the same document token.###\n\nWhat You Gain, and What It Costs\n\nYou gain retrieval quality, particularly on queries where one specific piece of a document is what makes it relevant, on multi-requirement queries like the sofa above where each requirement gets to find its own evidence, and on out-of-domain data where a dense model's compression was tuned for a different distribution. That compression is learned from the training queries, so the model learns to keep what they needed and drop everything else, which may include exactly what your production queries ask about. The effect grows with document length, since more text has to fit in the same fixed vector.\n\nThe cost is index size. One vector per token instead of one vector per document is a lot more vectors, only partly offset by the smaller dimension. Encoding 4,874 Natural Questions passages with [ lightonai/LateOn](https://huggingface.co/lightonai/LateOn) produced 608,414 token vectors, an average of 124.8 per passage:\n\n| Representation | Vectors | Dimensions | float32 size |\n|---|---|---|---|\n| Dense,\n`all-MiniLM-L6-v2` |\n\n`gte-modernbert-base`\n\n`LateOn`\n\nThat's about 42x the storage of the MiniLM index, or 62 KiB per passage. However, indexes are often compressed, e.g. the same 608,414 vectors take 92 MB as a [fast-plaid](#indexing) index, since PLAID stores a centroid id plus a quantized residual per vector rather than the vector itself. For scale, a 4096-dimensional dense model like [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) would need about 80 MB for these same 4,874 passages, so a compressed multi-vector index sits in the same territory as the dense indexes people already run. [Token Pooling](#token-pooling) cuts the vector count before any of that, and [Retrieve and Rerank](#retrieve-and-rerank) avoids building an index at all.\n\n[PyLate](https://github.com/lightonai/pylate) comes up throughout this post, so briefly: Sentence Transformers handled dense and sparse models but not late interaction, so [LightOn](https://huggingface.co/lightonai) built PyLate on top of it to close that gap, adding the training, inference, and retrieval pieces these models need. Much of what you'll load below was trained with it, and LightOn built an ecosystem around it too, including [fast-plaid](https://github.com/lightonai/fast-plaid), the late-interaction index that turns up in [Indexing](#indexing). With v6.0 those capabilities live in Sentence Transformers itself.\n\nWith the tradeoff in mind, let's get a model running.\n\n## Installation\n\nMulti-vector models work with a plain install:\n\n```\npip install -U sentence-transformers\n```\n\nFor ColPali-style visual document retrieval, you also need the image dependencies (see [Installation](https://sbert.net/docs/installation.html) for all extras, and [Multimodal Embedding & Reranker Models](https://huggingface.co/blog/multimodal-sentence-transformers) for multimodal support in general):\n\n```\npip install -U \"sentence-transformers[image]\"\n```\n\nSentence Transformers v6.0 requires\n\n`transformers`\n\nv5.x,`torch`\n\n2.2+, and`huggingface-hub`\n\nv1.x. If you pin any of those lower, plan the upgrade first. See the[Migration Guide]for the full list of breaking changes.\n\n## Loading a Model\n\nLoading a multi-vector model looks exactly like loading any other Sentence Transformers model:\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\n```\n\nTo find models that work, look for the [ multi-vector and sentence-transformers tags](https://huggingface.co/models?library=sentence-transformers&other=multi-vector) on the Hub. Any model with those tags loads with the line above, whether it started life as a PyLate checkpoint, a Stanford-NLP ColBERT checkpoint, or a ColPali-family model for visual document retrieval. We're working through the ecosystem to get that tag onto every model that works, so the list keeps growing.\n\nUnderneath, `MultiVectorEncoder`\n\nreads each of the formats these checkpoints have been published in over the years, so PyLate and Stanford-NLP checkpoints load directly even where the tag hasn't been added yet:\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\n\n# Native Sentence Transformers checkpoints. PyLate builds on the same schema,\n# so any PyLate checkpoint loads identically\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\nmodel = MultiVectorEncoder(\"mixedbread-ai/mxbai-edge-colbert-v0-17m\")\nmodel = MultiVectorEncoder(\"LiquidAI/LFM2.5-ColBERT-350M\", trust_remote_code=True)\n\n# Any Stanford-NLP ColBERT checkpoint, detected via the `HF_ColBERT` architecture\n# marker. The inline projection weight and the recipe come from `artifact.metadata`\nmodel = MultiVectorEncoder(\"colbert-ir/colbertv2.0\")\nmodel = MultiVectorEncoder(\"answerdotai/answerai-colbert-small-v1\")\n\n# A bare transformer: a fresh random projection is appended, so training is required\nmodel = MultiVectorEncoder(\"answerdotai/ModernBERT-base\")\n```\n\nVisual document retrieval models are the exception. ColPali-family checkpoints ship in colpali-engine's own format, which carries no information Sentence Transformers can use, so each one needs a small configuration added to its repository before it loads. Most of that work is done and waiting to be merged. See [Supported Models](#supported-models) for the current state and how to load them today.\n\n### Inspecting What a Checkpoint Configured\n\nMulti-vector models carry a handful of recipe knobs that differ per checkpoint: marker prefixes for queries and documents, length caps, whether queries are padded out with `[MASK]`\n\ntokens, and which tokens are skipped when scoring documents. All of them live in the module configs, so `print(model)`\n\nshows you exactly what you loaded. Here's the original ColBERTv2 checkpoint, which pads every query to exactly 32 tokens and truncates documents at 180:\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\"colbert-ir/colbertv2.0\")\nprint(model)\n\"\"\"\nMultiVectorEncoder(\n  (0): Transformer({..., 'document_length': 180,\n                    'query_expansion': {'strategy': 'fixed', 'attend': False, 'token': None, 'length': 32}})\n  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, ...})\n  (2): MultiVectorMask({'skiplist_words': ['!', '\"', '#', ...], 'skiplist_tasks': ['document'], ...})\n  (3): Normalize({...})\n)\n\"\"\"\nprint(model.prompts)\n# {'query': '[unused0] ', 'document': '[unused1] '}\n```\n\nThat's the classic ColBERT pipeline: a `Transformer`\n\nproducing contextualized token embeddings, a token-level `Dense`\n\nprojecting each of them to 128 dimensions, a `MultiVectorMask`\n\ndeciding which tokens count during scoring, and a token-level `Normalize`\n\n. Other checkpoints fill in different values. `lightonai/GTE-ModernColBERT-v1`\n\nuses the same four modules with `[Q] `\n\nand `[D] `\n\nprompts, no query expansion, and caps of 48 and 300.\n\nYou rarely need to touch any of this, since every released checkpoint configures its own. It matters when you build a model from a bare backbone, which is covered in [Creating Custom Models](https://sbert.net/docs/multi_vector_encoder/usage/custom_models.html).\n\nOne value is worth checking against your own data, though. `document_length`\n\ntruncates, so anything past it never reaches the index. For example, a 662-token passage through LateOn's cap of 300 comes back as 273 vectors, with the rest of the passage simply gone. Most of these checkpoints were trained on short passages, so if your chunks are longer than the cap, you can lift it for a single call with `encode_document(..., processing_kwargs={\"text\": {\"max_length\": 512}})`\n\n, keeping in mind that you would be running the model past the length it was trained on and that the index grows roughly in proportion. Multi-vector models tend to tolerate that well. On [MLDR](https://huggingface.co/datasets/Shitao/MLDR), a long-document retrieval benchmark, the multilingual siblings of the pair above show the gap clearly: [mLateOn scores 77.92 against mDenseOn's 51.59](https://huggingface.co/blog/lightonai/mdenseon-mlateon#long-document-retrieval-mldr).\n\n## Encoding Queries and Documents\n\nMulti-vector models are asymmetric: queries and documents go through different prefixes, different length caps, and different scoring masks. Unlike many dense models, where the two are interchangeable, [ encode_query()](https://sbert.net/docs/package_reference/multi_vector_encoder/model.html#sentence_transformers.multi_vector_encoder.model.MultiVectorEncoder.encode_query) and\n\n[are required to get correct embeddings:](https://sbert.net/docs/package_reference/multi_vector_encoder/model.html#sentence_transformers.multi_vector_encoder.model.MultiVectorEncoder.encode_document)\n\n`encode_document()`\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\"lightonai/mLateOn\")\n\nqueries = [\"What is the capital of France?\"]\ndocuments = [\n    \"Paris is the capital of France.\",\n    \"Berlin is the capital and largest city of Germany, by both area and population.\",\n]\n\nquery_embeddings = model.encode_query(queries)\ndocument_embeddings = model.encode_document(documents)\n\nprint(query_embeddings[0].shape)\n# (10, 128)\nprint(document_embeddings[0].shape, document_embeddings[1].shape)\n# (10, 128) (19, 128)\n```\n\nNote what you get back: a *list* of 2D tensors, one per input, each of shape `(num_tokens, embedding_dim)`\n\n. Unlike dense embeddings, you can't stack these into one rectangular tensor, because every input has its own token count. The second document is longer than the first, so it comes back as a taller matrix.\n\nEach call applies the model's own recipe for you. `encode_query`\n\nprepends the query marker, expands the query to a fixed length if the checkpoint asks for it, and caps it at the query length. `encode_document`\n\nprepends the document marker, caps at the document length, and drops any skiplisted tokens (punctuation, for most checkpoints) from the scoring mask.\n\nThe usual `encode()`\n\narguments all still apply, so `batch_size`\n\n, `show_progress_bar`\n\n, `convert_to_tensor`\n\n, `device`\n\n, and multi-process pools work the way you'd expect:\n\n```\ndocument_embeddings = model.encode_document(\n    documents,\n    batch_size=64,\n    convert_to_tensor=True,\n    show_progress_bar=True,\n)\n```\n\n## Scoring with MaxSim\n\n[ model.similarity()](https://sbert.net/docs/package_reference/multi_vector_encoder/model.html#sentence_transformers.multi_vector_encoder.model.MultiVectorEncoder.similarity) computes the full all-pairs MaxSim matrix:\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\n\nquery_embeddings = model.encode_query([\"Which planet is known as the Red Planet?\"])\ndocument_embeddings = model.encode_document([\n    \"Venus is often called Earth's twin because of its similar size and proximity.\",\n    \"Mars, known for its reddish appearance, is often referred to as the Red Planet.\",\n    \"Jupiter, the largest planet in our solar system, has a prominent red spot.\",\n    \"Saturn, famous for its rings, is sometimes mistaken for the Red Planet.\",\n])\n\nscores = model.similarity(query_embeddings, document_embeddings)\nprint(scores)\n# tensor([[10.7942, 11.1104, 10.9743, 11.0811]])\n```\n\nMars wins, as it should. Note how close the runners-up are: Saturn also contains the literal phrase \"the Red Planet\", and Jupiter is a planet with a red spot, so a token-level operator has plenty to latch onto in all three. The ordering is what matters.\n\nScores often sit this close together, as [GLInt](https://huggingface.co/blog/chungimungi/glint#1-mining-in-maxsim-space) shows by measuring the spread across a full candidate pool. MaxSim takes a *maximum* per query token, so a document will usually give every query token some decent best match, and scores start from a floor. Contextualized token embeddings are also anisotropic, clustering in a narrow cone rather than spreading out, so even arbitrary token pairs tend to score high.\n\nThere is also [ model.similarity_pairwise()](https://sbert.net/docs/package_reference/multi_vector_encoder/model.html#sentence_transformers.multi_vector_encoder.model.MultiVectorEncoder.similarity_pairwise), for when you already have matched pairs and just want the pair scores instead of the full similarity matrix:\n\n```\nscores = model.similarity_pairwise(query_embeddings, document_embeddings[:1])\nprint(scores)\n# tensor([10.7942])\n```\n\n### Score Magnitude and MeanMaxSim\n\nMaxSim sums over query tokens, so its magnitude scales with how many query tokens there are, which means you can't compare scores across models with different query recipes. LateOn encodes the Red Planet query above as 12 tokens. Run that same query and those same documents through ColBERTv2, which pads and truncates every query to exactly 32 tokens, and the scores land in a completely different range:\n\n```\nmodel = MultiVectorEncoder(\"colbert-ir/colbertv2.0\")\n# ... same encode_query / encode_document / similarity calls ...\nprint(scores)\n# tensor([[12.7970, 27.1945, 23.8495, 24.5656]])\n```\n\nWithin one model the ordering is all you need, but if you want scores on a bounded scale, switch the model's similarity function to MeanMaxSim, which divides by the query token count. Back on LateOn:\n\n```\nmodel = MultiVectorEncoder(\"lightonai/LateOn\", similarity_fn_name=\"meanmaxsim\")\n# or on an already-loaded model: model.similarity_fn_name = \"meanmaxsim\"\n\nprint(model.similarity(query_embeddings, document_embeddings))\n# tensor([[0.8995, 0.9259, 0.9145, 0.9234]])\n```\n\nNow every score is an average cosine similarity in `[-1, 1]`\n\n, although you'll only see `[0, 1]`\n\nin practice.\n\n## Semantic Search\n\nIf your corpus is small, exhaustive MaxSim over all of it is the simplest thing that works. Encode the corpus once, then score each query against everything:\n\n``` python\nimport time\n\nfrom datasets import load_dataset\n\nfrom sentence_transformers import MultiVectorEncoder\n\ndataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train[:5000]\")\n# Several questions share an answer passage, so drop repeats but keep the order\ncorpus = list(dict.fromkeys(dataset[\"answer\"]))  # 5,000 rows -> 4,874 passages\n\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\ncorpus_embeddings = model.encode_document(corpus, convert_to_tensor=True, show_progress_bar=True)\n\nquery = \"when did richmond last play in a preliminary final\"\nstart = time.perf_counter()\nquery_embeddings = model.encode_query([query], convert_to_tensor=True)\nscores = model.similarity(query_embeddings, corpus_embeddings)[0]  # 98ms\ntop_scores, top_indices = scores.topk(3)\nprint(f\"Search took {(time.perf_counter() - start) * 1000:.1f}ms\")\n\nfor score, index in zip(top_scores.tolist(), top_indices.tolist()):\n    print(f\"{score:.4f}  {corpus[index][:100]}\")\n\"\"\"\nSearch took 122.7ms\n11.9192  Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved\n11.7591  2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contest\n11.6710  Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fou\n\"\"\"\n```\n\nThose 4,874 passages encoded in 20 seconds on an RTX 3090, and each search takes about 120ms end to end, most of that the MaxSim scoring against all 608,414 token vectors. This is exact, but it scales linearly in total corpus tokens and keeps every token vector in memory, so reach for it when you have a few thousand documents rather than a few million. The runnable version of this script is [semantic_search.py](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/applications/semantic_search.py).\n\nPast that size you want a real late-interaction index, which Sentence Transformers doesn't ship. It doesn't need to: these indexes store whatever `encode_document`\n\nproduced, so you encode here and hand the token embeddings to something built for them. [Indexing](#indexing) has working snippets for four of the options, and the section directly below covers how to skip the index entirely.\n\n## Retrieve and Rerank\n\nYou can also get late-interaction quality without maintaining a late-interaction index, by using a multi-vector model as your *reranker*. A fast bi-encoder narrows a large corpus to a handful of candidates, then the multi-vector model rescores only those:\n\n``` python\nfrom datasets import load_dataset\n\nfrom sentence_transformers import MultiVectorEncoder, SentenceTransformer\nfrom sentence_transformers.util import semantic_search\n\ndataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train[:50000]\")\ncorpus = list(dict.fromkeys(dataset[\"answer\"]))\n\nretriever = SentenceTransformer(\"jinaai/jina-embeddings-v5-text-nano-retrieval\")\nreranker = MultiVectorEncoder(\"perplexity-ai/pplx-embed-v1-late-0.6b\", trust_remote_code=True)\n\n# First stage: index the corpus once with a fast bi-encoder\ncorpus_embeddings = retriever.encode_document(corpus, convert_to_tensor=True, show_progress_bar=True)\n\n# Retrieve the top 50\nquery = \"when did richmond last play in a preliminary final\"\nhits = semantic_search(retriever.encode_query([query], convert_to_tensor=True), corpus_embeddings, top_k=50)[0]\ncandidates = [corpus[hit[\"corpus_id\"]] for hit in hits]\n\n# Second stage: rescore just those candidates with MaxSim\nquery_embeddings = reranker.encode_query([query])\ndocument_embeddings = reranker.encode_document(candidates)\nscores = reranker.similarity(query_embeddings, document_embeddings)[0]\n\nfor index in scores.argsort(descending=True)[:3].tolist():\n    print(f\"{scores[index].item():.4f}  {candidates[index][:100]}\")\n```\n\nOnly the 50 candidates are ever encoded as multi-vectors, so your index stays a normal dense index and the token vectors are transient. This is the same role a cross-encoder plays in a retrieve-and-rerank stack, but a multi-vector model is considerably cheaper per candidate. You encode the documents in one batch and score them with a matrix multiplication, instead of one forward pass per query-document pair. The runnable script is [retrieve_rerank.py](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/applications/retrieve_rerank.py), which prints the timings of both stages.\n\n## Indexing\n\nSeveral vector databases index and score multi-vectors natively: [Qdrant](https://qdrant.tech/documentation/concepts/vectors/) since v1.10, [Weaviate](https://docs.weaviate.io/weaviate/tutorials/multi-vector-embeddings) since v1.29, [Vespa](https://blog.vespa.ai/announcing-long-context-colbert-in-vespa/) for years now, [LanceDB](https://docs.lancedb.com/search/multivector-search) since v0.15.0, and [VectorChord](https://docs.vectorchord.ai/vectorchord/usage/indexing-with-maxsim-operators.html), which adds a MaxSim operator to Postgres that plain pgvector doesn't have. [Milvus](https://milvus.io/docs/array-of-structs.md) joined them in v2.6.4, under array-of-structs rather than the unrelated feature it calls multi-vector search. If you would rather not run a server at all, LightOn's [fast-plaid](https://github.com/lightonai/fast-plaid) is a `pip install`\n\naway and implements PLAID directly, and [PyLate](https://github.com/lightonai/pylate) wraps it in a fuller retrieval stack.\n\nA few others get you partway. [OpenSearch](https://docs.opensearch.org/latest/search-plugins/search-relevance/rerank-by-field-late-interaction/) and [Elasticsearch](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/rank-vectors) can rescore candidates with MaxSim but not retrieve on it, and the Elasticsearch field is additionally in technical preview and Enterprise-tier. [turbopuffer](https://turbopuffer.com/docs/schema) has late-interaction indexing in private beta.\n\nThe snippets below index text, but nothing in them is text-specific. `encode_document`\n\nhands back the same list of token-vector matrices whether the document was a passage, a page image, an audio clip, or a video, so the ColPali-style models from [Visual Document Retrieval](#visual-document-retrieval) go into any of these unchanged. There are simply more vectors per document, which is what makes [Token Pooling](#token-pooling) worth reaching for sooner there.\n\nfast-plaid, Qdrant, Weaviate, and Vespa all take exactly what `encode_document`\n\nreturns, so the code is the same up to the client library. Here's a working snippet for each, run against the 4,874 passages and 608,414 token vectors from the [Semantic Search](#semantic-search) example. Each one carries the ingestion and query times it produced on one machine (RTX 3090, i7-13700K), with no tuning beyond what the code shows, to give a sense of the shape of the work. All four answer the query faster than the 98ms `model.similarity`\n\ntook in that section, and three of them do it on the CPU, since fast-plaid is the only one here using the GPU.\n\nAll four returned the same three passages in the same order as the exhaustive PyTorch MaxSim earlier in this post, and the three databases reproduce its scores to four decimals! That is because their snippets score every document, which is affordable at this size and removes approximation as a variable. fast-plaid is approximate by design, so its scores differ slightly. The notes under each one say what changes when you switch to an approximate index, which is where rankings start to drift.\n\n**fast-plaid**\n\n[fast-plaid](https://github.com/lightonai/fast-plaid) is LightOn's Rust implementation of PLAID, the index ColBERT was originally built around. There's no server to start, and it reads the tensors `encode_document`\n\nhands back without any conversion.\n\n``` python\n# pip install sentence-transformers datasets fast-plaid\nfrom datasets import load_dataset\nfrom fast_plaid import search\nfrom sentence_transformers import MultiVectorEncoder\n\ndataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train[:5000]\")\ncorpus = list(dict.fromkeys(dataset[\"answer\"]))\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\nquery = \"when did richmond last play in a preliminary final\"\n\ndocument_embeddings = model.encode_document(corpus, batch_size=32, convert_to_tensor=True)\nquery_embedding = model.encode_query(query, convert_to_tensor=True)\n\nfast_plaid = search.FastPlaid(index=\"natural-questions\", device=\"cuda\")\n\n# 4,874 documents (608,414 token vectors) indexed in 5s\nfast_plaid.create(documents_embeddings=document_embeddings)\n\nresults = fast_plaid.search(queries_embeddings=query_embedding.unsqueeze(0), top_k=3)  # 11ms\n\nfor index, score in results[0]:\n    print(f\"{score:.4f}  {corpus[index][:90]}\")\n\"\"\"\n11.8828  Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve\n11.7676  2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes\n11.6758  Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo\n\"\"\"\n```\n\nThe `index`\n\nargument is a directory, not just a label, so the index is written to disk as it is built. Pointing a new `FastPlaid`\n\nat the same path reopens it for searching or for adding more documents, instead of rebuilding from the embeddings each time. On this corpus it occupies 92 MB, against 311.5 MB for the raw float32 vectors.\n\nThis is the only one of the four that is approximate, and it is the one place in this section where the scores do not match the exhaustive MaxSim. PLAID prunes with centroids and stores quantized residuals, so the three scores drift by a few hundredths in both directions against the 11.9192 / 11.7591 / 11.6710 computed earlier. The ranking is unaffected here, and that is the trade PLAID is making: it was designed for corpora far larger than this one, where scanning everything is not an option.\n\n**Qdrant**\n\n[Qdrant](https://qdrant.tech/documentation/concepts/vectors/) needs a server: `docker run -p 6333:6333 qdrant/qdrant`\n\n. The client also has a local mode (`QdrantClient(\":memory:\")`\n\n) that needs no server, but it's a pure-Python reimplementation, so use it for trying things out rather than for timing them.\n\n``` python\n# pip install sentence-transformers datasets qdrant-client\nfrom datasets import load_dataset\nfrom qdrant_client import QdrantClient, models\nfrom sentence_transformers import MultiVectorEncoder\n\ndataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train[:5000]\")\ncorpus = list(dict.fromkeys(dataset[\"answer\"]))\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\nquery = \"when did richmond last play in a preliminary final\"\n\ndocument_embeddings = model.encode_document(corpus, batch_size=32)\nquery_embedding = model.encode_query(query)\n\nclient = QdrantClient(\"http://localhost:6333\")\nclient.create_collection(\n    collection_name=\"natural-questions\",\n    vectors_config=models.VectorParams(\n        size=model.get_embedding_dimension(),\n        distance=models.Distance.COSINE,\n        multivector_config=models.MultiVectorConfig(\n            comparator=models.MultiVectorComparator.MAX_SIM\n        ),\n        # MaxSim never walks the HNSW graph, so skip building one\n        hnsw_config=models.HnswConfigDiff(m=0),\n    ),\n)\n\n# 4,874 documents (608,414 token vectors) ingested in 26.3s\nclient.upload_points(\n    collection_name=\"natural-questions\",\n    points=[\n        models.PointStruct(id=idx, vector=embedding, payload={\"text\": text})\n        for idx, (embedding, text) in enumerate(zip(document_embeddings, corpus))\n    ],\n    batch_size=64,\n)\n\nresults = client.query_points(\n    collection_name=\"natural-questions\",\n    query=query_embedding,\n    limit=3,\n    with_payload=True,\n).points  # 18ms\n\nfor result in results:\n    print(f\"{result.score:.4f}  {result.payload['text'][:90]}\")\n\"\"\"\n11.9192  Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve\n11.7591  2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes\n11.6710  Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo\n\"\"\"\n```\n\n`MAX_SIM`\n\nis the only comparator Qdrant offers, and `hnsw_config=HnswConfigDiff(m=0)`\n\nis their recommendation for late-interaction fields, since the vectors are used for rescoring rather than graph traversal. Note that Qdrant themselves suggest reserving late interaction for reranking a few hundred candidates rather than scanning a whole collection, which is the [Retrieve and Rerank](#retrieve-and-rerank) pattern. At 4,874 documents the full scan costs 18ms and is exact, but that doesn't extrapolate.\n\n**Weaviate**\n\n[Weaviate](https://docs.weaviate.io/weaviate/tutorials/multi-vector-embeddings) needs a server too: `docker run -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:1.34.0`\n\n. Multi-vector support needs 1.29 or newer, and the embedded mode isn't available on Windows.\n\n``` python\n# pip install sentence-transformers datasets weaviate-client\nimport weaviate\nfrom datasets import load_dataset\nfrom sentence_transformers import MultiVectorEncoder\nfrom weaviate.classes.config import Configure, DataType, Property\nfrom weaviate.classes.query import MetadataQuery\n\ndataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train[:5000]\")\ncorpus = list(dict.fromkeys(dataset[\"answer\"]))\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\nquery = \"when did richmond last play in a preliminary final\"\n\ndocument_embeddings = model.encode_document(corpus, batch_size=32)\nquery_embedding = model.encode_query(query)\n\nclient = weaviate.connect_to_local()\ncollection = client.collections.create(\n    \"Documents\",\n    # self_provided turns on MaxSim late interaction\n    vector_config=[Configure.MultiVectors.self_provided(name=\"colbert\")],\n    properties=[Property(name=\"text\", data_type=DataType.TEXT)],\n)\n\n# 4,874 documents (608,414 token vectors) ingested in 41s\nwith collection.batch.fixed_size(batch_size=64) as batch:\n    for text, embedding in zip(corpus, document_embeddings):\n        batch.add_object(properties={\"text\": text}, vector={\"colbert\": embedding.tolist()})\n\nresults = collection.query.near_vector(\n    near_vector=query_embedding.tolist(),\n    target_vector=\"colbert\",\n    limit=3,\n    return_metadata=MetadataQuery(distance=True),\n)  # 17ms\n\nfor result in results.objects:\n    # Weaviate reports the MaxSim score as a negated distance\n    print(f\"{-result.metadata.distance:.4f}  {result.properties['text'][:90]}\")\n\"\"\"\n11.9192  Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve\n11.7591  2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes\n11.6710  Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo\n\"\"\"\n\nclient.close()\n```\n\nDefaults are enough here: Weaviate's dynamic `ef`\n\nresolves to 100 for a top-3 query, and this ranking is already exact from about 32 upward. That margin is a property of the embeddings rather than of Weaviate, so it's worth confirming on your own model instead of assuming the defaults hold.\n\nWeaviate also supports MUVERA encoding, which made ingestion 3x faster and queries 1.8x faster in our test. It cost far more accuracy than that speed is worth at this size though: the correct third passage didn't appear even in its top 50.\n\n**Vespa**\n\n[Vespa](https://docs.vespa.ai/en/tensor-user-guide.html) also runs in a container, but `pyvespa`\n\nstarts it for you, so there's no separate `docker run`\n\n.\n\n``` python\n# pip install sentence-transformers datasets pyvespa\nfrom datasets import load_dataset\nfrom sentence_transformers import MultiVectorEncoder\nfrom vespa.deployment import VespaDocker\nfrom vespa.package import (\n    ApplicationPackage, Document, Field, FirstPhaseRanking, Function, RankProfile, Schema,\n)\n\ndataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train[:5000]\")\ncorpus = list(dict.fromkeys(dataset[\"answer\"]))\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\nquery = \"when did richmond last play in a preliminary final\"\n\ndocument_embeddings = model.encode_document(corpus, batch_size=32)\nquery_embedding = model.encode_query(query)\n\n# \"dt\" is a mapped dimension over the variable token count, \"x\" the dense 128-dim vector\npackage = ApplicationPackage(\n    name=\"colbert\",\n    schema=[\n        Schema(\n            name=\"doc\",\n            document=Document(fields=[\n                Field(name=\"text\", type=\"string\", indexing=[\"summary\"]),\n                Field(name=\"colbert\", type=\"tensor<float>(dt{}, x[128])\", indexing=[\"attribute\"]),\n            ]),\n            rank_profiles=[\n                RankProfile(\n                    name=\"colbert\",\n                    inputs=[(\"query(qt)\", \"tensor<float>(qt{}, x[128])\")],\n                    functions=[Function(\n                        name=\"max_sim\",  # per query token take the best document token, then sum\n                        expression=\"sum(reduce(sum(query(qt) * attribute(colbert), x), max, dt), qt)\",\n                    )],\n                    first_phase=FirstPhaseRanking(expression=\"max_sim\"),\n                )\n            ],\n        )\n    ],\n)\napp = VespaDocker(port=8080).deploy(application_package=package)  # ~40s to boot\n\n# Vespa reads a mixed tensor as {token index: vector}, for documents and queries alike\ndef to_tensor(embedding):\n    return {str(token): vector for token, vector in enumerate(embedding.tolist())}\n\n# 4,874 documents (608,414 token vectors) ingested in ~80s\napp.feed_iterable(\n    ({\"id\": str(idx), \"fields\": {\"text\": text, \"colbert\": to_tensor(embedding)}}\n     for idx, (text, embedding) in enumerate(zip(corpus, document_embeddings))),\n    schema=\"doc\",\n)\n\nresponse = app.query(body={\n    \"yql\": \"select text from doc where true\",\n    \"ranking.profile\": \"colbert\",\n    \"hits\": 3,\n    \"input.query(qt)\": to_tensor(query_embedding),\n})  # ~75ms warm, ~115ms on the first call\n\nfor hit in response.hits:\n    print(f\"{hit['relevance']:.4f}  {hit['fields']['text'][:90]}\")\n\"\"\"\n11.9192  Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieve\n11.7591  2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contes\n11.6710  Battle of Appomattox Court House The Battle of Appomattox Court House (Virginia, U.S.), fo\n\"\"\"\n```\n\nVespa asks for the most upfront structure of the four, because you're declaring a ranking pipeline rather than just an index. In exchange you get to write MaxSim out as a tensor expression and see exactly what it computes. This version puts MaxSim in `first-phase`\n\nover `where true`\n\n, which scores all 4,874 documents and is why the output matches exhaustive MaxSim exactly. It's deliberately not what Vespa recommends at scale: their [ColBERT sample app](https://github.com/vespa-engine/sample-apps/tree/master/colbert) stores int8-binarized vectors and moves MaxSim into `second-phase`\n\nto rerank a cheaper first stage.\n\nMoving to that phased setup needs care: `second-phase`\n\nrescores only the best 100 candidates by default, and here that window left two of the three correct passages unscored entirely. Raising `rerank-count`\n\nto cover your candidate set fixes that, though at this size the phased version still came out slower than simply scanning everything.\n\n## Visual Document Retrieval\n\nLate interaction is the state of the art for visual document retrieval: matching a text query against page *images*, with charts, tables, and layout intact, and no OCR step. This is what the [ColPali](https://arxiv.org/abs/2407.01449) family of models does, and those checkpoints load and run through the same API, with the `revision`\n\npinning the open pull request that adds this one's Sentence Transformers configuration ([Supported Models](#visual-document-retrieval-models) has the full list). Image documents are passed as URLs, local paths, or PIL images:\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\"vidore/colqwen2.5-v0.2\")\n\nqueries = [\n    \"What is the variable represented on the y-axis of the graph?\",\n    \"Total outlay is maximum in which year?\",\n]\nimages = [\n    \"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg\",\n    \"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg\",\n    \"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc3.jpg\",\n    \"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc4.jpg\",\n]\n\nquery_embeddings = model.encode_query(queries)\ndocument_embeddings = model.encode_document(images)\nprint(query_embeddings[0].shape, document_embeddings[0].shape)\n# (25, 128) (755, 128)\n\nscores = model.similarity(query_embeddings, document_embeddings)\nprint(scores)\n# tensor([[13.8672, 12.3115, 12.1670, 11.0293],\n#         [ 7.2012, 14.7207,  6.9414,  6.9746]])\n```\n\nEach query retrieves its own page (the diagonal), and the second query separates much more cleanly than the first, since only one of the four pages is about outlay over time.\n\nThe code is unchanged. Underneath, the processor handles the visual prompt and the image patches, and MaxSim scores query text tokens against document image patches. A page holds many separate regions, which is exactly what makes late interaction a natural fit here, since a single vector would have to average a chart, a table, and three paragraphs into one summary. That fidelity costs index space, though. The shapes above are 755 token vectors for one page against 25 for the query, where a Natural Questions passage from earlier averaged about 125, so [token pooling](#token-pooling) is worth reaching for earlier here than it is for text.\n\nThese are VLMs, so plan for the memory they need. [The table in Supported Models](#visual-document-retrieval-models) runs from 252M to 8.8B parameters, and the small end of it stays practical on CPU where the multi-billion ones don't.\n\nPage images are the common case, but they're not the only non-text modality. Sentence Transformers accepts text, images, audio, and video, and a checkpoint supports whichever of those its processor does, which `model.modalities`\n\nreports. A single document can combine modalities too, by passing a dict like `{\"text\": ..., \"image\": ...}`\n\nin place of a bare value. [Multimodal Embedding & Reranker Models](https://huggingface.co/blog/multimodal-sentence-transformers) covers multimodal models in Sentence Transformers more broadly, and the [Usage documentation](https://sbert.net/docs/sentence_transformer/usage/usage.html) lists exactly which input formats each modality accepts.\n\n## Audio Retrieval\n\n[vidore/colqwen-omni-v0.1](https://huggingface.co/vidore/colqwen-omni-v0.1) is built on Qwen2.5-Omni and takes all four modalities. Retrieving a recorded conversation with it is the same two calls as retrieving a page:\n\n``` python\n# pip install -U \"sentence-transformers[audio,video]\"\nimport torch\nfrom datasets import Audio, load_dataset\n\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\n    \"vidore/colqwen-omni-v0.1\",\n    model_kwargs={\"dtype\": torch.bfloat16},\n)\nprint(model.modalities)\n# ['text', 'image', 'audio', 'video', 'message']\n\n# 20 recorded conversations, averaging 28 seconds each\ndataset = load_dataset(\"eustlb/dailytalk-conversations-grouped\", split=\"train[:20]\")\ndataset = dataset.cast_column(\"audio\", Audio(sampling_rate=16_000))\naudio = [row[\"array\"] for row in dataset[\"audio\"]]  # raw mono waveforms, float32 at 16 kHz\n\nquery_embeddings = model.encode_query([\"medicine for car nausea\"])\ndocument_embeddings = model.encode_document(audio, batch_size=2)\nscores = model.similarity(query_embeddings, document_embeddings)[0]\n\ntop_scores, top_indices = scores.topk(3)\nfor score, index in zip(top_scores.tolist(), top_indices.tolist()):\n    print(f\"{score:.4f}  {' / '.join(dataset[index]['texts'][:2])}\")\n\"\"\"\n50.8902  Excuse me? Do you have anything for a carsickness? / Yes, but you look fine.\n46.1028  Excuse me, could you tell me where you have got that music book? / Certainly. Let me see. Oh, it's on that shelf.\n46.0514  Jeff, I'm going to the supermarket. Do you want to come with me? / I think the supermarket is closed now.\n\"\"\"\n```\n\nColQwen-Omni was trained purely on image-text pairs, so its audio retrieval is zero-shot: it never heard a training example, and there is no transcription step anywhere in the pipeline. The query says `nausea`\n\nwhere the recording says `carsickness`\n\n, and it still picks the pharmacy conversation out of twenty by a wide margin.\n\n## Video Retrieval\n\nVideo works the same way, but sample the frames or it will eat your VRAM. Its [release blogpost](https://huggingface.co/blog/manu/colqwen-omni-omnimodal-retrieval) is blunt about this, that video \"is very memory-intensive, so it's best suited for short clips\":\n\n``` python\nimport torch\n\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\n    \"vidore/colqwen-omni-v0.1\",\n    model_kwargs={\"dtype\": torch.bfloat16},\n)\n\n# Sparse, low-resolution frames: 0.5 fps rather than the full frame rate\nmodel[0].processing_kwargs.update(\n    {\"video\": {\"max_pixels\": 32 * 28 * 28, \"do_sample_frames\": True, \"fps\": 0.5}}\n)\n\nquery_embeddings = model.encode_query([\"How to cook Mapo Tofu?\"])\ndocument_embeddings = model.encode_document([\n    \"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/mapo_tofu.mp4\",\n    \"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/zhajiang_noodle.mp4\",\n], batch_size=1)\nprint(model.similarity(query_embeddings, document_embeddings))\n# tensor([[53.3100, 51.0561]])\n```\n\nAt 1 fps and full resolution the same pair of videos produces 8,426 and 5,137 token vectors and peaks at 20.8 GB of VRAM, against 4,240 and 2,446 vectors and 12.5 GB here, for a model that occupies 9.0 GB on its own. The ranking is identical either way. Long audio wants the same treatment, and the release blogpost recommends 30-second chunks, which come to roughly 800 tokens each.\n\n## Interpretability\n\nBecause MaxSim is a sum of per-query-token maxima, a ranking decomposes exactly: every point of a document's score belongs to one query token and one document token. That lets you answer \"why did this rank here?\" precisely, rather than by eye.\n\nFor image documents, `sentence_transformers.multi_vector_encoder.interpretability`\n\noverlays that decomposition onto the page as the standard ColPali heatmap, either aggregated over the query or one map per query token. Asking \"How much was spent on water resources and power?\" against the outlays page from above, this is where the `water`\n\ntoken went:\n\n[heatmap.py](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/interpretability/heatmap.py) is the runnable version, including the masking step that lines the document embedding up with the patch grid.\n\nText documents have no patch grid to overlay, but the same decomposition applies. [text_similarity_map.py](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/interpretability/text_similarity_map.py) ranks a corpus and then attributes the top hit's score token by token, here on the Natural Questions corpus from earlier with the 32M-parameter [mxbai-edge-colbert-v0-32m](https://huggingface.co/mixedbread-ai/mxbai-edge-colbert-v0-32m):\n\n```\nQuery: when did richmond last play in a preliminary final\nTop 3 of 4874 documents by exhaustive MaxSim (191.0ms):\n  12.3489  Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved since 19\n  12.1771  2017 AFL Grand Final The 2017 AFL Grand Final was an Australian rules football game contested betwee\n  12.0591  2018 UEFA Champions League Final The 2018 UEFA Champions League Final was the final match of the 201\n\n  query token       best document token      sim   share\n  when              since                 0.9154    7.4%\n  did               had                   0.9675    7.8%\n  rich              rich                  0.9764    7.9%\n  mond              mond                  0.9856    8.0%\n  last              to                    0.9249    7.5%\n  play              game                  0.9384    7.6%\n  in                the                   0.9732    7.9%\n  a                 a                     0.9587    7.8%\n  preliminary       preliminary           0.9394    7.6%\n  final             final                 0.9654    7.8%\n  --------------------------------------------------------\n  3 special tokens                        2.8038   22.7%\n  MaxSim score                           12.3489  100.0%\n```\n\n`rich`\n\n, `mond`\n\n, `preliminary`\n\n, and `final`\n\nmatched themselves, while `when`\n\nsettled on `since`\n\nand `play`\n\non `game`\n\n. The special tokens are worth noticing too: three of them contribute 22.7% of the score while carrying none of the query's content. Below this table the script prints the passage itself, with the winning tokens highlighted in place.\n\n## Token Pooling\n\nIf the index footprint worries you, the most effective knob is to store fewer token vectors. `HierarchicalTokenPooling`\n\nimplements the [token pooling](https://arxiv.org/abs/2409.14683v1) technique from Clavié, Chaffin, and Adams: it clusters each document's token vectors with Ward linkage on cosine distance and replaces each cluster with its mean, keeping roughly `1 / pool_factor`\n\nof the tokens. Within one document a lot of token vectors end up close to each other, so much of what you drop is redundancy rather than signal:\n\n``` python\nfrom datasets import load_dataset\n\nfrom sentence_transformers import MultiVectorEncoder\nfrom sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling\n\ndataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train[:5000]\")\ndocuments = list(dict.fromkeys(dataset[\"answer\"]))\n\nmodel = MultiVectorEncoder(\"lightonai/LateOn\")\n\npooling = HierarchicalTokenPooling(pool_factor=2)\ndocument_embeddings = model.encode_document(documents, token_pooling=pooling)\n```\n\nThere are three places to apply it, depending on when you want to pay for it:\n\n```\n# 1. Per encode call, as above\ndocument_embeddings = model.encode_document(documents, token_pooling=pooling)\n\n# 2. Standalone, on embeddings you already have saved (e.g. list of [num_tokens, num_dims] tensors)\npooled = pooling.pool(document_embeddings)\n\n# 3. Baked into the model, so every consumer of the checkpoint gets pooled documents\nmodel.append(HierarchicalTokenPooling(pool_factor=2))\nmodel.save_pretrained(\"my-pooled-colbert\")\n```\n\nBy default, pooling applies to documents only, since queries are short and are the side you can't afford to distort. On the Natural Questions corpus from earlier, the reduction tracks `pool_factor`\n\nclosely, and pooling all 608k token vectors took about 6 seconds:\n\n`pool_factor` |\nToken vectors | Reduction | float32 index |\n|---|---|---|---|\n| 1 (off) | 608,414 | 1.00x | 311.5 MB |\n| 2 | 305,438 | 1.99x | 156.4 MB |\n| 3 | 204,407 | 2.98x | 104.7 MB |\n| 4 | 153,936 | 3.95x | 78.8 MB |\n\nA cluster mean is a worse match for a query token than the best of its members was, and the coarser the clusters, the more that shows. The [original experiments](https://arxiv.org/abs/2409.14683v1) measured that cost on BEIR and found very little of it: 100.6% of the unpooled retrieval performance on average at `pool_factor=2`\n\n, and 99.0% at `pool_factor=3`\n\n. Halving your index for free is a good deal, so 2 is a reasonable place to start. How much it costs on your data is corpus-specific though, so measure it with an [evaluator](#evaluating-a-model) before you settle on a factor. The runnable comparison is [token_pooling.py](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/compression/token_pooling.py).\n\nHow far you can push `pool_factor`\n\nis also partly a property of the model. LightOn's [hierarchical pooling regularization](https://huggingface.co/blog/lightonai/lateon-hpool-regularization) trains for exactly that, shaping the embedding space so pooling costs less and reporting 99.4% retention at 5x compression. Training with that regularizer isn't in Sentence Transformers yet, but the resulting checkpoints are ordinary PyLate models, so [ lightonai/LateOn-hpool-regularized](https://huggingface.co/lightonai/LateOn-hpool-regularized) loads and pools like any other.\n\n## Speeding Up Inference\n\nMulti-vector models run through the same backend machinery as the rest of Sentence Transformers, so you get `torch`\n\n(default), `onnx`\n\n, and `openvino`\n\n, alongside half precision, Flash Attention, and `torch.compile`\n\n.\n\nOn GPU, fp16 with Flash Attention is the best configuration we measured, at 2.44x the throughput of fp32 with no measurable retrieval quality loss. Flash Attention helps multi-vector models more than most, because documents are only truncated and never padded to a shared length, so your batches have widely varying sequence lengths that unpadding can exploit:\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\n    \"lightonai/GTE-ModernColBERT-v1\",\n    model_kwargs={\"attn_implementation\": \"flash_attention_2\", \"dtype\": \"float16\"},\n)\n```\n\nModels with non-attend query expansion (\n\n`attend=False`\n\n, which covers the Stanford-NLP checkpoints like`colbert-ir/colbertv2.0`\n\nand`answerdotai/answerai-colbert-small-v1`\n\n) reject Flash Attention at load time. Flash Attention strips`attention_mask=0`\n\npositions, so the`[MASK]`\n\nexpansion tokens that MaxSim scores would never receive an attention update. Use`\"sdpa\"`\n\nfor those models.\n\nOn CPU, OpenVINO is your better bet where the architecture is supported, and int8 quantization buys a further speedup at a cost of about 0.4% accuracy. See [Speeding up Inference](https://sbert.net/docs/multi_vector_encoder/usage/efficiency.html) for the full benchmark details, the export and quantization helpers, and a flowchart for picking a backend.\n\n## Evaluating a Model\n\n`MultiVectorNanoBEIREvaluator`\n\nruns the [NanoBEIR](https://huggingface.co/collections/zeta-alpha-ai/nanobeir-66e1a0af21dfd93e620cd9f6) suite of 13 small BEIR subsets with MaxSim scoring, and needs no data preparation on your side:\n\n``` python\nfrom sentence_transformers import MultiVectorEncoder\nfrom sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator\n\nmodel = MultiVectorEncoder(\"lightonai/GTE-ModernColBERT-v1\")\nevaluator = MultiVectorNanoBEIREvaluator(batch_size=16)\nresults = evaluator(model)\nprint(f\"{evaluator.primary_metric}: {results[evaluator.primary_metric]:.4f}\")\n```\n\nThis also makes it easy to check the claim from the top of this post. [ lightonai/LateOn](https://huggingface.co/lightonai/LateOn) and\n\n[were trained by LightOn on the same data with the same ModernBERT backbone and the same 149M parameters, differing only in whether they keep one vector per token or pool down to one per document. Running both over all 13 NanoBEIR datasets isolates what that choice buys:](https://huggingface.co/lightonai/DenseOn)\n\n`lightonai/DenseOn`\n\n| NanoBEIR dataset | LateOn (multi-vector, 128d) | DenseOn (dense, 768d) |\n|---|---|---|\n| MSMARCO | 0.7194 |\n0.6517 |\n| NQ | 0.7810 |\n0.7511 |\n| HotpotQA | 0.9295 |\n0.8802 |\n| FEVER | 0.9702 |\n0.9612 |\n| ClimateFEVER | 0.4887 |\n0.4846 |\n| DBPedia | 0.6836 |\n0.6748 |\n| QuoraRetrieval | 0.9795 |\n0.9687 |\n| Touche2020 | 0.5938 |\n0.5673 |\n| ArguAna | 0.5562 | 0.5660 |\n| NFCorpus | 0.3949 |\n0.3851 |\n| SciFact | 0.7978 | 0.8057 |\n| SCIDOCS | 0.4469 | 0.4484 |\n| FiQA2018 | 0.5871 | 0.6491 |\nMean |\n0.6868 |\n0.6764 |\n\nLate interaction wins on 9 of the 13 datasets and on the mean, by roughly one NDCG point. The four it loses (ArguAna, FiQA2018, SCIDOCS, and SciFact) are the shape of the tradeoff you should expect: a real gain in retrieval quality at the same model size, paid for in index footprint, rather than a universal win on every dataset. The same pair scores 57.22 against 56.20 on the full 15-dataset BEIR, a comparable gap, so the margin is not an artifact of the small benchmark.\n\nAlongside NanoBEIR, `MultiVectorInformationRetrievalEvaluator`\n\n, `MultiVectorRerankingEvaluator`\n\n, `MultiVectorTripletEvaluator`\n\n, and `MultiVectorDistillationEvaluator`\n\ncover the usual evaluation setups on your own data. They're documented in the [Evaluation API Reference](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html).\n\n## Coming from PyLate or colpali-engine\n\n`MultiVectorEncoder`\n\nabsorbs the modeling, inference, training, and evaluation of both libraries. Every PyLate checkpoint loads directly, and [Supported Models](#supported-models) lists the colpali-engine checkpoints along with the `revision`\n\nto pass where one is still needed. If you're migrating, these are the calls that change:\n\n| PyLate | Sentence Transformers |\n|---|---|\n`pylate.models.ColBERT(model_name_or_path=...)` |\n`MultiVectorEncoder(...)` |\n`model.encode(..., is_query=True)` |\n`model.encode_query(...)` |\n`model.encode(..., is_query=False)` |\n`model.encode_document(...)` |\n`pylate.scores.colbert_scores` |\n`model.similarity` |\n`pylate.indexes.PLAID` / `pylate.retrieve.ColBERT` |\nno equivalent, keep PyLate's PLAID or see\n|\n\n| colpali-engine | Sentence Transformers |\n|---|---|\n`ColQwen2.from_pretrained(...)` + `ColQwen2Processor` |\n`MultiVectorEncoder(...)` |\n`processor.process_queries(...)` + `model(**batch)` |\n`model.encode_query(queries)` |\n`processor.process_images(...)` + `model(**batch)` |\n`model.encode_document(images)` |\n`processor.score_multi_vector(qs, ds)` |\n`model.similarity(query_embeddings, document_embeddings)` |\n`mask_non_image_embeddings=True` |\n`MultiVectorMask(keep_only_token_ids=[...])` |\n`HierarchicalTokenPooler` |\n`HierarchicalTokenPooling` |\n`colpali_engine.interpretability` |\n`sentence_transformers.multi_vector_encoder.interpretability` |\n\nOne difference worth calling out: on a **bare** (non-ColBERT) checkpoint, PyLate's `ColBERT(\"bert-base-uncased\")`\n\napplies the classic recipe by default, while `MultiVectorEncoder(\"bert-base-uncased\")`\n\nbuilds a plain stack and leaves the prefixes, query expansion, and skiplist as explicit choices. The training loss and evaluator equivalents, and the data-handling differences, are in the [Migration Guide](https://sbert.net/docs/migration_guide.html#migrating-from-pylate).\n\nNote that save compatibility is one-way in every case: PyLate, Stanford-NLP ColBERT, and colpali-engine checkpoints all load into `MultiVectorEncoder`\n\n, but `MultiVectorEncoder.save_pretrained`\n\noutput isn't loadable by any of them.\n\n## Supported Models\n\nModels carrying the [ multi-vector and sentence-transformers tags](https://huggingface.co/models?library=sentence-transformers&other=multi-vector) on the Hub are the list that stays current, and we're working to get those tags onto every model that works. The tables below are what we test against directly, so treat them as a starting point rather than the full set. For text retrieval in particular, any PyLate or Stanford-NLP ColBERT checkpoint loads whether or not it carries the tag yet.\n\nSome entries need a small Sentence Transformers configuration added to their repository first, and several of those are still open pull requests at the time of writing. Where a `revision`\n\nis listed below, pass it until that pull request is merged, after which the plain model name is enough:\n\n```\nmodel = MultiVectorEncoder(\"vidore/colqwen-omni-v0.1\", revision=\"refs/pr/N\")\n```\n\n### Text Retrieval Models\n\nThese load with their trained prefix tokens, query expansion, and punctuation skiplist recovered from the saved configuration.\n\nThe NanoBEIR column reports the mean NDCG@10 (higher is better) across the 13 [NanoBEIR datasets](https://huggingface.co/datasets/sentence-transformers/NanoBEIR-en), each a 50-query subsample of a BEIR dataset, as a fast proxy for English text retrieval quality. We used the `MultiVectorNanoBEIREvaluator`\n\nto compute the scores for the primarily-English models. A `-`\n\nmeans the model was not evaluated on it. Note that NanoBEIR is a small benchmark, and its scores aren't a substitute for evaluating on your own data, which is always the right way to pick a model.\n\n### Visual Document Retrieval Models\n\nColPali-style models embed page images as documents and text as queries.\n\nThe NanoViDoRe column reports the mean NDCG@10 (higher is better) across [NanoViDoRe v3](https://huggingface.co/datasets/lightonai/NanoViDoRe_v3), a compact visual document retrieval benchmark spanning 8 subsets (computer science, energy, finance in English and French, HR, industrial, pharmaceuticals, and physics). Like with NanoBEIR, NanoViDoRe is a small benchmark which shouldn't replace evaluation on your own data.\n\n| Model | Parameters | Dimensionality | NanoViDoRe | Notes |\n|---|---|---|---|---|\n|\n\n`trust_remote_code=True`\n\n[webAI-Official/webAI-ColVec1.1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b)`trust_remote_code=True`\n\n[tencent/EVIE-Preview-4.5B](https://huggingface.co/tencent/EVIE-Preview-4.5B)[TomoroAI/tomoro-colqwen3-embed-8b](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b)`trust_remote_code=True`\n\n[TomoroAI/tomoro-colqwen3-embed-4b](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-4b)`trust_remote_code=True`\n\n[vidore/colqwen2.5-v0.2](https://huggingface.co/vidore/colqwen2.5-v0.2)[vidore/colqwen2.5-v0.1](https://huggingface.co/vidore/colqwen2.5-v0.1)[vidore/colqwen-omni-v0.1](https://huggingface.co/vidore/colqwen-omni-v0.1)[vidore/colpali-v1.3](https://huggingface.co/vidore/colpali-v1.3)[vidore/colpali-v1.3-hf](https://huggingface.co/vidore/colpali-v1.3-hf)[vidore/colpali-v1.2](https://huggingface.co/vidore/colpali-v1.2)[vidore/colqwen2-v1.0](https://huggingface.co/vidore/colqwen2-v1.0)[vidore/colqwen2-v0.1](https://huggingface.co/vidore/colqwen2-v0.1)[vidore/colpali](https://huggingface.co/vidore/colpali)[vidore/colpali-v1.1](https://huggingface.co/vidore/colpali-v1.1)[vidore/colsmolvlm-v0.1](https://huggingface.co/vidore/colsmolvlm-v0.1)[vidore/colpali-hard-v1.1](https://huggingface.co/vidore/colpali-hard-v1.1)[vidore/colSmol-500M](https://huggingface.co/vidore/colSmol-500M)[vidore/colSmol-256M](https://huggingface.co/vidore/colSmol-256M)[ModernVBERT/colmodernvbert](https://huggingface.co/ModernVBERT/colmodernvbert)[vidore/colpali-v1.2-hf](https://huggingface.co/vidore/colpali-v1.2-hf)[vidore/colqwen2-v1.0-hf](https://huggingface.co/vidore/colqwen2-v1.0-hf)Most of these are LoRA adapter repositories, with the adapter applied directly onto its base at load time. Some also have a `-merged`\n\nsibling on the Hub (e.g. [vidore/colpali-v1.3-merged](https://huggingface.co/vidore/colpali-v1.3-merged)) with the adapter already folded into the weights.\n\nThe three `-hf`\n\nentries are the transformers-native `*ForRetrieval`\n\nports. They load without any configuration, but use more modeling from `transformers`\n\nand less from `sentence_transformers`\n\n. Generally, it's preferable to use the original models instead, as the ports score approximately the same.\n\n## Acknowledgements\n\nLate interaction in Sentence Transformers rests on a lot of earlier work. Thanks to Omar Khattab and Matei Zaharia for [ColBERT](https://arxiv.org/abs/2004.12832), which everything here descends from, and to the LightOn team (Antoine Chaffin, Raphael Sourty, Paulo Moura, and Amélie Chatelain) for [PyLate](https://github.com/lightonai/pylate) and [fast-plaid](https://github.com/lightonai/fast-plaid), which carried late interaction for years and shaped a good deal of the API described above.\n\nThanks to the ColPali team (Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo) for [ColPali](https://arxiv.org/abs/2407.01449) and colpali-engine, which brought late interaction to page images, and to Benjamin Clavié, Antoine Chaffin, and Griffin Adams for [token pooling](https://arxiv.org/abs/2409.14683v1).\n\nThanks as well to the core MTEB team, Kenneth Enevoldsen and Roman Solomatin among many others, for [MTEB](https://github.com/embeddings-benchmark/mteb) and for the kind of hidden work that keeps information retrieval research running.\n\nAnd thanks to everyone who trained and released the checkpoints in [Supported Models](#supported-models). Without them this post would have had nothing to measure.\n\n## Additional Resources\n\n### Documentation\n\n[Multi-Vector Encoder > Usage](https://sbert.net/docs/multi_vector_encoder/usage/usage.html)[Multi-Vector Encoder > Pretrained Models](https://sbert.net/docs/multi_vector_encoder/pretrained_models.html)[Multi-Vector Encoder > Creating Custom Models](https://sbert.net/docs/multi_vector_encoder/usage/custom_models.html)[Multi-Vector Encoder > Speeding up Inference](https://sbert.net/docs/multi_vector_encoder/usage/efficiency.html)[Multi-Vector Encoder > API Reference](https://sbert.net/docs/package_reference/multi_vector_encoder/index.html)[Installation](https://sbert.net/docs/installation.html)[Migration Guide](https://sbert.net/docs/migration_guide.html)\n\n### Example Scripts\n\n[Semantic Search](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/applications/semantic_search.py)[Retrieve and Rerank](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/applications/retrieve_rerank.py)[Token Pooling](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/compression/token_pooling.py)[ColPali Heatmaps](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/interpretability/heatmap.py)[Text Similarity Maps](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/interpretability/text_similarity_map.py)[NanoBEIR Evaluation](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/evaluation/nano_beir.py)\n\n### Training\n\nTo learn how to train or finetune these models on your own data:\n\n[Multi-Vector Encoder > Training Overview](https://sbert.net/docs/multi_vector_encoder/training_overview.html)[Multi-Vector Encoder > Loss Overview](https://sbert.net/docs/multi_vector_encoder/loss_overview.html)[Multi-Vector Encoder > Training Examples](https://sbert.net/docs/multi_vector_encoder/training/examples.html)[LateOn and mLateOn training scripts](https://github.com/lightonai/mdenseon-mlateon): LightOn's PyLate recipes for LateOn, mLateOn, DenseOn, and mDenseOn, where the finetuning scripts show practical details like splitting a 16,384-example batch into mini-batches of 16.\n\n### Hugging Face Hub\n\n### Companion Blogposts\n\n[Training and Finetuning Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sentence-transformers): the general training guide for text-only dense embedding models.[Training and Finetuning Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-reranker): Cross Encoder training, the other way to add a precise second stage.[Training and Finetuning Sparse Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sparse-encoder): SPLADE and other sparse encoders, which combine well with late interaction in hybrid search.[Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/multimodal-sentence-transformers): single-vector multimodal models, the dense counterpart to ColPali-style retrieval.[Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): includes a Visual Document Retrieval walkthrough with single-vector models.[🪆 Introduction to Matryoshka Embedding Models](https://huggingface.co/blog/matryoshka): shrink dense embeddings by dimension, the way token pooling shrinks multi-vector ones by count.", "url": "https://wpnews.pro/news/multi-vector-late-interaction-embedding-models-with-sentence-transformers", "canonical_source": "https://huggingface.co/blog/multi-vector-encoder", "published_at": "2026-08-18 00:00:00+00:00", "updated_at": "2026-08-18 14:13:46.182881+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "ai-tools", "ai-research"], "entities": ["Hugging Face", "Sentence Transformers", "MultiVectorEncoder", "PyLate", "Stanford-NLP ColBERT", "colpali-engine"], "alternates": {"html": "https://wpnews.pro/news/multi-vector-late-interaction-embedding-models-with-sentence-transformers", "markdown": "https://wpnews.pro/news/multi-vector-late-interaction-embedding-models-with-sentence-transformers.md", "text": "https://wpnews.pro/news/multi-vector-late-interaction-embedding-models-with-sentence-transformers.txt", "jsonld": "https://wpnews.pro/news/multi-vector-late-interaction-embedding-models-with-sentence-transformers.jsonld"}}