Fine-tuning microsoft/harrier-oss-v1-270m with SentenceTransformerTrainer — is it supported?

Microsoft's harrier-oss-v1-270m sentence-transformers model can be fine-tuned with SentenceTransformerTrainer and MultipleNegativesRankingLoss for Portuguese QA retrieval, though no public example combines that exact stack, according to a technical analysis of the model card. The model card states Harrier uses a decoder-only architecture, last-token pooling and L2-normalized embeddings, and that query instructions are part of training so omitting them degrades performance while document-side instructions are not needed. The recommended setup applies the instruction prefix to the query/anchor side during training and keeps documents unprompted to avoid a train/inference mismatch.

Seems practically supported? microsoft/harrier-oss-v1-270m with SentenceTransformerTrainer Yes — this should be a supported and reasonable setup , with one important caveat: I have not found a public example that exactly combines: microsoft/harrier-oss-v1-270m + SentenceTransformerTrainer + MultipleNegativesRankingLoss + Portuguese QA retrieval However, the evidence strongly points to this being a valid path: - microsoft/harrier-oss-v1-270m https://huggingface.co/microsoft/harrier-oss-v1-270m is packaged as a sentence-transformers model and can be loaded with SentenceTransformer "microsoft/harrier-oss-v1-270m" . - The model card explicitly shows Sentence Transformers usage and encodes queries with a prompt while encoding documents without a prompt . - The model card says Harrier uses a decoder-only architecture , last-token pooling , and L2-normalized embeddings . - The model card also says query instructions are how the model is trained and that omitting them can degrade performance; document-side instructions are not needed. - SentenceTransformerTrainingArguments https://sbert.net/docs/package reference/sentence transformer/training args.html supports training-time prompts , including column-specific prompt mappings. - MultipleNegativesRankingLoss https://sbert.net/docs/package reference/sentence transformer/losses.html multiple-negativesrankingloss is the standard Sentence Transformers loss for positive query, document / anchor, positive retrieval pairs. - Nearby public examples exist, especially a Harrier-family Vietnamese legal retrieval model using Sentence Transformers + MNRL, and SkillRet-style decoder-embedding fine-tuning using query instructions and unprompted documents. My recommendation is: training query / anchor: Instruct: ...\nQuery: <Portuguese question training document / positive: <raw Portuguese passage inference query: Instruct: ...\nQuery: <Portuguese question inference document: <raw Portuguese passage So: apply the instruction prefix to the query/anchor side during training, not only at inference. Keep documents/passages unprompted. Yes. For Harrier, the instruction is not just an inference-time decoration. It is part of the expected query-side input format. The Harrier-270M model card https://huggingface.co/microsoft/harrier-oss-v1-270m shows this Sentence Transformers pattern: query embeddings = model.encode queries, prompt name="web search query" document embeddings = model.encode documents The same model card also shows the raw Transformers pattern: php def get detailed instruct task description: str, query: str - str: return f"Instruct: {task description}\nQuery: {query}" Each query must come with a one-sentence instruction that describes the task. No need to add instruction for retrieval documents. The FAQ is especially relevant: it says query instructions are how the model is trained, omitting them causes degradation, and document-side instructions are not needed. So for Portuguese QA retrieval, I would train with: query / anchor: Instruct: Given a Portuguese question, retrieve relevant Portuguese passages that answer the question Query: <question document / positive: <passage and infer with exactly the same policy: query: Instruct: Given a Portuguese question, retrieve relevant Portuguese passages that answer the question Query: <question document: <passage This avoids a train/inference mismatch. training query: Qual é o prazo para interpor recurso administrativo? inference query: Instruct: Given a Portuguese question, retrieve relevant Portuguese passages that answer the question Query: Qual é o prazo para interpor recurso administrativo? This fine-tunes the model on raw questions but deploys it on prompted questions. For Harrier, that is probably the wrong distribution. training query: Instruct: Given a Portuguese question, retrieve relevant Portuguese passages that answer the question Query: Qual é o prazo para interpor recurso administrativo? inference query: Instruct: Given a Portuguese question, retrieve relevant Portuguese passages that answer the question Query: Qual é o prazo para interpor recurso administrativo? Documents should remain raw in both training and inference: training document: O prazo para interposição de recurso administrativo é de 10 dias úteis... indexed document: O prazo para interposição de recurso administrativo é de 10 dias úteis... SentenceTransformerTrainer If your dataset columns are named query and document , use column-specific prompts: python from sentence transformers import SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments, losses, from sentence transformers.training args import BatchSamplers model = SentenceTransformer "microsoft/harrier-oss-v1-270m", model kwargs={"dtype": "auto"}, query prompt = "Instruct: Given a Portuguese question, " "retrieve relevant Portuguese passages that answer the question\n" "Query: " args = SentenceTransformerTrainingArguments output dir="harrier-270m-pt-qa-mnrl", per device train batch size=8, gradient accumulation steps=16, effective batch size 128 on 1 GPU learning rate=5e-6, num train epochs=1, warmup ratio=0.10, lr scheduler type="cosine", bf16=True, gradient checkpointing=True, batch sampler=BatchSamplers.NO DUPLICATES, prompts={ "query": query prompt, "document": "", }, logging steps=50, save strategy="steps", save steps=500, save total limit=2, loss = losses.MultipleNegativesRankingLoss model, directions= "query to doc", , trainer = SentenceTransformerTrainer model=model, args=args, train dataset=train dataset, columns: query, document loss=loss, trainer.train trainer.save model "harrier-270m-pt-qa-mnrl/final" If your dataset columns are named anchor and positive , change only the prompt mapping: prompts={ "anchor": query prompt, "positive": "", } The important rule is simple: query-like column: prompt document-like column: no prompt I would start with the English instruction format because it matches Harrier’s public prompt style: Instruct: Given a Portuguese question, retrieve relevant Portuguese passages that answer the question Query: The query and document texts themselves should remain Portuguese. After you have a baseline, test a Portuguese instruction as an ablation: Instruct: Dada uma pergunta em português, recupere passagens em português relevantes que respondam à pergunta Query: I would not start by translating the structural markers to Instrução: and Consulta: . Keep Instruct: and Query: first, because that matches the Harrier format shown in the model card and config sentence transformers.json https://huggingface.co/microsoft/harrier-oss-v1-270m/blob/main/config sentence transformers.json . Recommended first prompt: query prompt = "Instruct: Given a Portuguese question, " "retrieve relevant Portuguese passages that answer the question\n" "Query: " MultipleNegativesRankingLoss appropriate? Yes. For QA retrieval, your data usually has the form: query: Portuguese question positive: passage that answers the question That is a natural fit for MultipleNegativesRankingLoss https://sbert.net/docs/package reference/sentence transformer/losses.html multiple-negativesrankingloss , which is designed for positive pairs such as query, response or anchor, positive . A basic version is: loss = losses.MultipleNegativesRankingLoss model For clarity in retrieval, I would write: loss = losses.MultipleNegativesRankingLoss model, directions= "query to doc", , This trains the model so that each query is closer to its matching passage than to other passages in the batch. MNRL uses other positives in the batch as negatives. That is efficient, but it can be harmful if some of those “negatives” are actually relevant. Example: query: Como solicitar a segunda via da fatura? positive A: A segunda via da fatura pode ser solicitada no portal do cliente. positive B for another query: Para emitir uma cópia da fatura, acesse Minha Conta e clique em Segunda Via. For the first query, positive B is not really negative. It probably answers the same question. If it appears in the same batch, MNRL may incorrectly push it away. This is common in QA retrieval, FAQ retrieval, legal retrieval, policy retrieval, support retrieval, and any corpus with repeated answer templates. Use: batch sampler=BatchSamplers.NO DUPLICATES The Sentence Transformers training overview https://sbert.net/docs/sentence transformer/training overview.html specifically notes that losses using in-batch negatives benefit from no duplicate samples in a batch. The loss docs https://sbert.net/docs/package reference/sentence transformer/losses.html also discuss cached / larger-batch variants of MNRL. Also deduplicate aggressively before training: - exact duplicate passages; - near-duplicate chunks; - boilerplate-heavy passages; - repeated FAQ answers; - multiple chunks from the same source document; - multiple positives that answer the same query. Harrier and BGE-M3 should not be treated as interchangeable SBERT-style encoders. microsoft/harrier-oss-v1-270m https://huggingface.co/microsoft/harrier-oss-v1-270m is: - decoder-only; - multilingual; - 270M parameters; - 640-dimensional embeddings; - up to 32,768 tokens; - last-token pooled; - L2-normalized; - instruction-sensitive on the query side. When used through SentenceTransformer , last-token pooling and normalization are handled automatically. If using raw AutoModel , you must reproduce the model-card pooling behavior yourself: python def last token pool last hidden states, attention mask : left padding = attention mask :, -1 .sum == attention mask.shape 0 if left padding: return last hidden states :, -1 sequence lengths = attention mask.sum dim=1 - 1 batch size = last hidden states.shape 0 return last hidden states torch.arange batch size , sequence lengths For this use case, I would stay with SentenceTransformer unless there is a strong reason not to. BAAI/bge-m3 https://huggingface.co/BAAI/bge-m3 is not just a dense embedding model. Its model card describes it as multi-functional, multilingual, and multi-granular: - dense retrieval; - sparse retrieval; - multi-vector retrieval; - more than 100 languages; - up to 8192 tokens. This matters for a fair comparison. Do not compare: BGE-M3 hybrid/sparse/multi-vector system vs Harrier dense-only system and call that a model-only comparison. Fairer comparisons are: BGE-M3 dense vs Harrier dense BGE-M3 hybrid vs Harrier dense + BM25 BGE-M3 + reranker vs Harrier + reranker For a first full-model Harrier-270M MNRL run, I would start conservatively. | Parameter | Recommended first value | | Base model | microsoft/harrier-oss-v1-270m | | Loss | MultipleNegativesRankingLoss | | Direction | "query to doc", | | Query prompt | yes | | Document prompt | no | | Learning rate | 5e-6 | | LR candidates | 3e-6 , 5e-6 , 1e-5 | | Epochs | 1 | | Warmup ratio | 0.10 | | Scheduler | cosine | | Precision | bf16 if supported | | Physical batch size | 4–16 , depending on GPU | | Effective batch size | 128–256 | | Batch sampler | BatchSamplers.NO DUPLICATES | | Gradient checkpointing | yes if memory-bound | | Max sequence length | 512 or 1024 first | I would not start with 5e-5 for full-model MNRL fine-tuning. Harrier is already a strong embedding model; the goal is adaptation, not overwriting its embedding geometry. A useful nearby reference is mainguyen9/vietlegal-harrier-0.6b https://huggingface.co/mainguyen9/vietlegal-harrier-0.6b , a Harrier-family Vietnamese legal retrieval model that reports Sentence Transformers training, MNRL, hard-negative mining, LR 3e-6 , batch size 256 , one epoch, warmup 10% , cosine scheduler, and bf16. It is not the same model size or language, but it is a closer reference than generic BERT/SBERT defaults. CachedMultipleNegativesRankingLoss ? Use it if your GPU memory prevents a useful effective batch size. MNRL benefits from larger batches because larger batches provide more in-batch negatives. If normal MNRL is memory-bound, try: loss = losses.CachedMultipleNegativesRankingLoss model, mini batch size=32, Then test effective batch sizes like: 256 512 1024 But I would not make cached MNRL the first experiment. First establish that the simple MNRL setup works. Run these in order. | Run | Model | Training | Query prompt | Doc prompt | LR | Effective batch | Purpose | | A | BGE-M3 | existing fine-tune | current | current | current | current | incumbent baseline | | B | Harrier-270M | none | yes | no | — | — | zero-shot baseline | | C | Harrier-270M | MNRL | yes | no | 5e-6 | 128 | main first run | | D | Harrier-270M | MNRL | yes | no | 3e-6 | 128–256 | lower-LR check | | E | Harrier-270M | MNRL | yes | no | 1e-5 | 128–256 | upper-LR check | | F | Harrier-270M | MNRL | no | no | 5e-6 | 128 | prompt ablation | | G | Harrier-270M | Cached MNRL | yes | no | 5e-6 | 256–1024 | batch-size check | | H | Harrier-270M | hard-negative stage | yes | no | 3e-6–5e-6 | task-dependent | ranking refinement | The most important comparison is: fine-tuned BGE-M3 vs zero-shot Harrier with query instruction vs fine-tuned Harrier with query instruction vs fine-tuned Harrier without query instruction Leaderboard scores are useful for model shortlisting, but the final decision should be based on your own Portuguese QA retrieval benchmark. For broader benchmark context, see MTEB https://huggingface.co/spaces/mteb/leaderboard , MMTEB https://arxiv.org/abs/2502.13595 , and the original MTEB paper https://arxiv.org/abs/2210.07316 . MTEB-style scores are useful, but they do not replace task-specific evaluation. Use the same evaluation pipeline for BGE-M3 and Harrier. Minimum retrieval metrics: nDCG@10 MRR@10 Recall@5 Recall@10 Recall@50 Recall@100 Why these metrics matter: | Metric | What it tells you | | Recall@50 / Recall@100 | Whether the retriever can put the right passage somewhere in the candidate pool | | Recall@5 / Recall@10 | Whether the retriever is good enough for direct RAG context selection | | MRR@10 | Whether the first relevant passage appears early | | nDCG@10 | Ranking quality when there are multiple relevant passages | Also track operational metrics: embedding throughput query latency index size GPU memory embedding dimension chunk length max sequence length For Portuguese-specific external sanity checks, useful resources include: Do not start with hard negatives. Start with clean query-positive MNRL. After the first baseline is stable: 1. Embed the full corpus. 2. Retrieve top 100 candidates per training query. 3. Remove known positives. 4. Skip the top few candidates if they may be unlabeled positives. 5. Sample negatives from ranks 20–100 or 50–100. 6. Train a second stage with explicit negatives or a hard-negative-aware setup. The reason to avoid the top retrieved “negative” is that it may actually be a valid answer that was not labeled. The SkillRet paper https://arxiv.org/html/2605.05726v1 is a useful related reference. It fine-tunes decoder-style embedding models using MultipleNegativesRankingLoss , applies the same task-specific query instruction to anchor queries during training, uses no document prompt for Harrier/Qwen-style embedding models, and mines hard negatives for the reranker stage. It also reports that fine-tuning Harrier-OSS-0.6B and Qwen3-Embedding-0.6B gives nearly identical performance in that task, suggesting that the training recipe matters at least as much as the exact decoder-embedding base. Bad: dataset query already contains: Instruct: ... Query: ... and TrainingArguments also uses: prompts={"query": "Instruct: ...\nQuery: "} This produces: Instruct: ... Query: Instruct: ... Query: <question Use one method: Either store raw queries and use prompts=... or store prompted queries and do not use prompts=... I recommend storing raw queries and using prompts=... . Bad: document: Instruct: Given a Portuguese question, retrieve relevant Portuguese passages that answer the question Query: <passage For Harrier retrieval, documents should be raw passages. Bad: training query: raw question inference query: prompted question Better: training query: prompted question inference query: prompted question Bad: BGE-M3 hybrid vs Harrier dense-only Better: BGE-M3 dense vs Harrier dense BGE-M3 hybrid vs Harrier dense + BM25 BGE-M3 + reranker vs Harrier + reranker Harrier supports long context, but that does not mean a first fine-tune should use 32k tokens. Start with: 512 or 1024 tokens Then test: 2048 4096 8192 only if your evaluation set shows that longer passages help. In retrieval, better chunking is often more useful than simply increasing max length. Use the same query prompt and raw documents: python from sentence transformers import SentenceTransformer model = SentenceTransformer "harrier-270m-pt-qa-mnrl/final" query prompt = "Instruct: Given a Portuguese question, " "retrieve relevant Portuguese passages that answer the question\n" "Query: " queries = "Qual é o prazo para interpor recurso administrativo?", documents = "O prazo para interposição de recurso administrativo é de 10 dias úteis...", "A segunda via da fatura pode ser solicitada no portal do cliente...", query embeddings = model.encode queries, prompt=query prompt, normalize embeddings=True, document embeddings = model.encode documents, normalize embeddings=True, scores = query embeddings @ document embeddings.T print scores Avoid passing both prompt and prompt name unless you intentionally want one to override the other. A related Qwen3 Embedding discussion https://huggingface.co/Qwen/Qwen3-Embedding-0.6B/discussions/5 notes that explicit prompt takes priority over prompt name in Sentence Transformers-style usage. For this Portuguese QA retrieval use case, I would proceed like this: 1. Keep your fine-tuned BGE-M3 model as the incumbent baseline. 2. Evaluate Harrier-270M zero-shot with the correct query instruction and raw documents. 3. Fine-tune Harrier with MNRL using query-side instruction during training. 4. Do not prompt documents. 5. Start with lr=5e-6 , one epoch, warmup 0.10 , cosine scheduler, bf16, effective batch size around 128 . 6. Run LR ablations at 3e-6 , 5e-6 , and 1e-5 . 7. Use BatchSamplers.NO DUPLICATES . 8. Deduplicate query/document pairs aggressively. 9. Try CachedMultipleNegativesRankingLoss if memory prevents larger effective batches. 10. Add hard negatives only after the clean first-stage baseline works. 11. Compare systems fairly: dense vs dense, hybrid vs hybrid, reranked vs reranked. 12. Decide based on your own held-out Portuguese QA retrieval set, not only Multilingual MTEB v2. - Supported? Yes, practically. Harrier-270M is a Sentence Transformers model and should work with SentenceTransformerTrainer . - Exact public recipe? I have not found an exact Harrier-270M + STTrainer + MNRL + Portuguese QA recipe. - Instruction during training? Yes. Apply it to the query/anchor side during training and inference. - Documents? Keep documents/passages unprompted. - Loss? MultipleNegativesRankingLoss is appropriate for query, positive passage pairs. - Main risks? Prompt mismatch, false negatives, duplicates, too-high LR, too-small effective batch, and incorrect pooling if using raw AutoModel . - Starting hyperparameters? lr=3e-6 to 1e-5 , one epoch, warmup 0.10 , cosine scheduler, bf16, effective batch 128–256 , BatchSamplers.NO DUPLICATES . - Best next experiment? Harrier zero-shot prompted vs Harrier MNRL prompted vs Harrier no-prompt ablation vs your fine-tuned BGE-M3 baseline.