Mean Pooling vs CLS: Watch Transformers.js Turn a Sentence Into an Embedding in Node

A developer demonstrated in Node how Transformers.js's feature-extraction pipeline returns per-token tensors by default, with pooling set to "none" and normalize to false, using an ONNX export of all-MiniLM-L6-v2 via ONNX Runtime. The walkthrough shows the tokenizer's WordPiece splits, special tokens, and attention masks, and warns that mismatched pooling between indexing and query paths silently degrades retrieval quality.

Somewhere in your RAG code there is a line like const vec = await embed text . A list of 384 floats comes back, you store it, and you never look inside again. That function hides a choice. In Transformers.js https://huggingface.co/docs/transformers.js/index , the feature-extraction pipeline has two options that decide what those floats mean, and both default to "do nothing": pooling defaults to "none" and normalize defaults to false , per the pipeline API docs https://huggingface.co/docs/transformers.js/api/pipelines . Call it with no options on "This is a simple test." and the docs show you get a tensor of shape 1, 8, 384 . You get eight vectors back, one per token. Every embedding library makes this choice for you somewhere. When the indexing job and the query path make it differently, nothing throws. Retrieval just gets quietly worse. Open the box in Node one step at a time, then break it on purpose. Install the library version 4.3.1 is the current release on npm https://registry.npmjs.org/@huggingface/transformers/latest , and in Node it runs ONNX models through ONNX Runtime https://onnxruntime.ai/ . Run the script with tsx https://github.com/privatenumber/tsx . The model downloads from the Hugging Face Hub https://huggingface.co/onnx-community/all-MiniLM-L6-v2-ONNX on first run. No API key. npm i @huggingface/transformers npx tsx embed.mts The model is an ONNX export of all-MiniLM-L6-v2 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 , the same one the Transformers.js docs use. Its model card says it maps text to a 384-dimensional space, uses mean pooling, and truncates input longer than 256 word pieces. js // embed.mts import { AutoModel, AutoTokenizer, type Tensor, } from "@huggingface/transformers"; const ID = "onnx-community/all-MiniLM-L6-v2-ONNX"; const tokenizer = await AutoTokenizer.from pretrained ID ; const model = await AutoModel.from pretrained ID, { dtype: "fp32", } ; const query = "I can't log in. How do I change my password?"; const docs = "Reset your password from the account settings page.", "Our office is closed on public holidays.", "Two-factor codes expire after 30 seconds.", "Passwords must be at least 12 characters long.", "Contact billing to update your credit card.", ; const all = query, ...docs ; console.log tokenizer.tokenize query ; const enc = await tokenizer all, { padding: true, truncation: true, } ; console.log enc.input ids.tolist 0 ; console.log enc.attention mask.tolist 1 ; const out = await model enc as Record<string, Tensor ; const hidden = out.last hidden state ?? out.token embeddings; console.log hidden.dims ; // batch, tokens, 384 Three things to read in that output. tokenize shows the WordPiece split. Words outside the vocabulary break into sub-word pieces, so the token count is not the word count. input ids has more entries than tokenize printed. The tokenizer adds special tokens at both ends by default add special tokens defaults to true https://huggingface.co/docs/transformers.js/api/tokenizers . The one at position 0 is CLS . Remember that position. attention mask is a row of 1s followed by 0s for any sentence shorter than the longest one in the batch. With padding: true , every row is stretched to the same length, and the mask marks which positions are real text. The tokenizer docs https://huggingface.co/docs/transformers.js/api/tokenizers describe it as "attended to 1 versus padded 0 ". The model output is 6, N, 384 : six texts, N positions each, 384 numbers per position. The fallback to token embeddings mirrors what the pipeline itself does in its source https://github.com/huggingface/transformers.js/blob/main/packages/transformers/src/pipelines/feature-extraction.js : it reads last hidden state , then logits , then token embeddings , depending on how the ONNX file was exported. Pooling collapses N token vectors into one. The two common choices are mean pooling average the real tokens and CLS pooling take position 0 and ignore the rest . Both poolings and L2 normalization fit in a few lines of plain TypeScript. Append this block, and every block after it, to embed.mts and rerun the script. js const H = hidden.tolist as number ; const M = enc.attention mask.tolist as bigint .map row = row.map Number ; function meanPool tokens: number , mask: number { const sum = new Array tokens 0 .length .fill 0 ; let count = 0; tokens.forEach vec, t = { if mask t === 0 return; // skip padding count++; vec.forEach x, d = sum d += x ; } ; return sum.map x = x / count ; } const clsPool = tokens: number = tokens 0 ; function l2 v: number { const norm = Math.hypot ...v ; return v.map x = x / norm ; } const dot = a: number , b: number = a.reduce s, x, i = s + x b i , 0 ; const cosine = a: number , b: number = dot a, b / Math.hypot ...a Math.hypot ...b ; The if mask t === 0 return line is the whole reason the mask exists. Padding positions still get hidden states out of the model. Average them in, and a short sentence's vector depends on how long its neighbours in the batch were. The sentence-transformers model card https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 carries a comment on its own mean pooling helper for this reason: "Take attention mask into account for correct averaging." Check your version against the library's. The pipeline does the same three steps internally: it tokenizes with padding: true, truncation: true , passes the attention mask into mean pooling for "mean" , slices position 0 for "cls" , and calls normalize 2, -1 when you ask for normalization source https://github.com/huggingface/transformers.js/blob/main/packages/transformers/src/pipelines/feature-extraction.js . js import { pipeline } from "@huggingface/transformers"; const extractor = await pipeline "feature-extraction", ID, { dtype: "fp32", } ; const ref = await extractor all, { pooling: "mean", normalize: true } .tolist as number ; const mine = all.map , i = l2 meanPool H i , M i ; const diffs = mine.flatMap v, i = v.map x, d = Math.abs x - ref i d , ; console.log "max abs difference:", Math.max ...diffs ; You are looking for a number at float-rounding scale. If it is large, your pooling does not match the library's, and that is worth finding out before you index anything. Try deleting the mask check and run it again: the drift on the shorter sentences is the padding leaking in. Cosine similarity divides by both vector lengths, so normalizing first does not change a cosine score. Print cosine a, b next to cosine l2 a , l2 b for any two pooled vectors and they agree up to rounding. What normalization changes is the dot product. On unit-length vectors, dot product and cosine are the same number. On raw vectors, dot product also rewards length, so a document whose pooled vector happens to be longer can climb the ranking without being more relevant. That matters because vector stores often ask you to pick the operator. The pgvector README https://github.com/pgvector/pgvector lists < for negative inner product and <= for cosine distance, and it recommends: "If vectors are normalized to length 1 like OpenAI embeddings , use inner product for best performance." Follow that advice with normalize: false vectors and you have changed your ranking function without changing a line of retrieval code. The experiment reuses one forward pass over the same six strings and turns the hidden states into a ranking four different ways. function rank name: string, q: number , d: number { console.log \n${name} ; d.map v, i = { i, s: dot q, v } .sort a, b = b.s - a.s .forEach { i, s } = console.log ${s.toFixed 4 } ${docs i } ; } const mean = all.map , i = meanPool H i , M i ; const cls = all.map , i = clsPool H i ; const meanL2 = mean.map l2 ; const clsL2 = cls.map l2 ; rank "A. mean + L2 model card ", meanL2 0 , meanL2.slice 1 ; rank "B. cls + L2", clsL2 0 , clsL2.slice 1 ; rank "C. mean, raw dot", mean 0 , mean.slice 1 ; rank "D. cls query vs mean index", clsL2 0 , meanL2.slice 1 ; Run A is the configuration the model card describes, so treat it as the baseline. Compare the others to it. Run B uses a different pooling on both sides. The scores move, because the sentence vectors are different vectors. Whether the top result moves depends on your sentences. The model card https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 pools with mean, so position 0 was not the pooling target during fine-tuning. Expect it to be the weaker signal. Run C keeps mean pooling but skips normalization and ranks by raw dot product. Compare the order to A. Any reshuffle you see comes from vector length alone. Run D is the one that hurts in production: the index was built with mean pooling, the query path uses CLS. Both vectors are 384 floats with unit length. Your database accepts them, the query returns five rows, and the scores look like scores. Nothing in the pipeline tells you the two sides were produced by different functions. Swap in your own query and a few dozen of your own documents, and watch each step change the output. BGE https://huggingface.co/BAAI/bge-small-en-v1.5 models are trained the other way. The sentence-transformers pooling config shipped in BAAI/bge-small-en-v1.5 sets "pooling mode cls token": true and "pooling mode mean tokens": false https://huggingface.co/BAAI/bge-small-en-v1.5/blob/main/1 Pooling/config.json , and the model card's Transformers example pools with model output 0 :, 0 . The BGE tutorial https://bge-model.com/tutorial/1 Embedding/1.2.3.html says BGE "is trained to use the last hidden state of CLS as the sentence embedding" and warns that "If you use mean pooling, there will be a significant decrease in performance." The Transformers.js-ready port of the same model, Xenova/bge-small-en-v1.5 https://huggingface.co/Xenova/bge-small-en-v1.5 , shows { pooling: 'mean', normalize: true } in its usage examples. The example is consistent with itself, because indexing and query both use mean. But copy it into a Node indexer, query the same collection from a Python service that loads BAAI/bge-small-en-v1.5 through sentence-transformers https://sbert.net/docs/package reference/sentence transformer/models.html whose modules.json https://huggingface.co/BAAI/bge-small-en-v1.5/blob/main/modules.json wires in the CLS pooling config above , and you have Run D in production. Cloudflare Workers AI https://developers.cloudflare.com/workers-ai/models/bge-base-en-v1.5/ documents the same split for its hosted bge-base-en-v1.5 . Its docs say the pooling parameter defaults to mean "in order for this to not be a breaking change," recommend cls for accuracy, and state plainly that "embeddings created with cls pooling are not compatible with embeddings generated with mean pooling." That last sentence is the rule. Pooling is part of the model, as far as your index is concerned. Treat the settings that produce a vector as one value, and store that value with the index. js export const EMBED = { model: "onnx-community/all-MiniLM-L6-v2-ONNX", dtype: "fp32", pooling: "mean", normalize: true, queryPrefix: "", } as const; export const EMBED KEY = JSON.stringify EMBED ; export function assertSameSpace indexKey: string { if indexKey == EMBED KEY { throw new Error index built with ${indexKey}, querying with ${EMBED KEY} , ; } } Write EMBED KEY into a metadata row when you build the index. Call assertSameSpace when the query service starts. If a second service in another language reads the same index, give it the same key and the same check. dtype is in there for a reason: Transformers.js lets you load quantized variants q8 , q4 , fp16 and more, per the dtypes guide https://huggingface.co/docs/transformers.js/guides/dtypes , and a quantized model is a different set of weights, so pin it like the rest. queryPrefix is in there because the BGE model card https://huggingface.co/BAAI/bge-small-en-v1.5 suggests an instruction prefix on short retrieval queries and not on passages, which is one more asymmetric setting that is easy to forget on one side. One more trap if you move to a decoder-style embedding model that uses last token or eos pooling: the pipeline takes the final position along the sequence axis without consulting the attention mask source https://github.com/huggingface/transformers.js/blob/main/packages/transformers/src/pipelines/feature-extraction.js . Check which side your tokenizer pads on before you batch, or the "last token" of a short sentence may be padding. This post is the inside of one function. AI That Reads https://www.amazon.com/dp/B0HBNJJRD9 is the book around it: embeddings, chunking, vector search and grounding answers in your own docs, written in TypeScript for Node. If you liked seeing where the 384 floats come from, the book keeps going from there, into how to store them, search them, and tell when retrieval is quietly failing. It is book 2 of AI in TypeScript , five books from your first LLM call to agents in production: the whole series is here https://xgabriel.com/ai-in-typescript .