{"slug": "linkup-research-releases-sparseup-a-149m-parameter-open-source-sparse-embedding", "title": "Linkup Research Releases SPARSEUP: A 149M-Parameter Open-Source Sparse Embedding Model", "summary": "Linkup released SPARSEUP, an open-source learned sparse embedding model with a 149M-parameter ModernBERT backbone, published under Apache 2.0 on Hugging Face. Linkup reports 56.4 average nDCG@10 on BEIR-13, which it calls the strongest public vocabulary-based sparse encoder under 150M parameters, though in a controlled comparison with backbone and data fixed, LateOn scored 58.9 and DenseOn 57.9. The model uses logit shifting, per-position top-k of 12, and case folding that cuts output dimensions from about 50k to about 34k, and with the Seismic inverted index reaches over 97% recall against exact search in about 380 microseconds per query.", "body_md": "Linkup research team releases **SPARSEUP**, an open-source learned sparse embedding model. The model runs on a 149M-parameter ModernBERT backbone and ships under Apache 2.0. Linkup team reports 56.4 average nDCG@10 on BEIR-13. It calls this the strongest public vocabulary-based sparse encoder it knows of under 150M parameters.\n\n**Is it deployable?** Yes. The [weights are on Hugging Face](https://huggingface.co/Linkup-Platform/linkup-sparseup-embed-v1) under Apache 2.0. The model loads through Transformers or Sentence Transformers with `trust_remote_code=True`. \n\n## **Why a Sparse Model, and Why Now**\n\nMost open retrieval models are dense: 1 vector per text. Sparse models output weights over a vocabulary instead. Each dimension maps to a real token, so vectors fit inverted indexes and humans can read them. They also tend to match rare words well.\n\nThe trigger was LightOn’s [DenseOn and LateOn release](https://huggingface.co/papers/2607.27178). LightOn published open data, a training recipe, a dense model and a late-interaction model. SPARSEUP fills the missing sparse slot. It uses the same backbone family and fine-tuning data, so all 3 retrieval styles can be compared side by side.\n\n## **How is SPARSEUP Built**\n\nTraining starts from [LateOn-unsupervised](https://huggingface.co/lightonai/LateOn-unsupervised). That checkpoint had no MLM head, so the team grafted back ModernBERT’s original one. Fine-tuning used [LightOn’s fine-tuning mixture](https://huggingface.co/datasets/lightonai/embeddings-fine-tuning) with contrastive learning only. Each query gets 7 hard negatives sampled from a pool of 50, and in-batch negatives. There is no cross-encoder distillation, and training fits on a single H100.\n\n**A vanilla SPLADE on this backbone produced huge bags full of stopwords. Linkup fixed this with 3 changes:**\n\n- **Logit shifting:** The encoder computes`log(1 + ReLU(x - 15))` . ModernBERT’s MLM logits sat too high, saturating the log and making bags dense at initialization.\n- **Per-position top-k:** Each input token keeps only its 12 strongest vocabulary dimensions before max pooling. This caps expansion per token, not total vector size.\n- **Case folding:** Byte-level BPE stores`heat` ,`Heat` ,`Ġheat` and`ĠHeat` as separate ids. SPARSEUP folds them onto 1 id and keeps the largest weight. Output dimensions drop from about 50k to about 34k.\n\nQueries and documents take `[Q]` and `[D]` prefixes, and scoring is a dot product. Evaluation max lengths are 128 tokens for queries and 512 for documents.\n\n## **Benchmark Results**\n\nAgainst other sparse encoders on BEIR-13 (nDCG@10, without MS MARCO), per the [model card](https://huggingface.co/Linkup-Platform/linkup-sparseup-embed-v1):\n\n| Model | BEIR-13 avg | \n|---|---|\n| **SPARSEUP** | **56.4** | \n| [opensearch-neural-sparse-encoding-doc-v3-gte](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte) | 54.6 | \n| opensearch-neural-sparse-encoding-v1 | 52.44 | \n| ModernBERT-VT | 52.4 | \n| splade-v3 | 51.7 | \n| [granite-embedding-30m-sparse](https://huggingface.co/ibm-granite/granite-embedding-30m-sparse) | 50.6 | \n| LACONIC-1B (1B parameters, different size class) | 58.7 | \n\nThe controlled comparison is less flattering. With backbone and data fixed, LateOn scores 58.9, DenseOn 57.9 and SPARSEUP 56.4. SPARSEUP uses approximate Seismic search, while LightOn reports exact search. SPARSEUP wins ArguAna and Touché and beats DenseOn on HotpotQA. It lags on more semantic sets, with FiQA showing the largest gap. DBPedia is another weak spot.\n\nOn decontaminated BEIR, the gap to DenseOn shrinks to 0.17 points. Linkup warns that decontaminated NQ and MS MARCO have only 21 and 46 queries, so those results are noisy.\n\n## **Speed and Sparsity**\n\nOn MS MARCO, SPARSEUP averages 47 non-zero terms per query and 190 per document. SPLADE-v3 averages 25 and 170. With the Seismic inverted index, it reaches over 97% recall against exact search in about 380 microseconds per query, single-threaded. Linkup says inflating vector size could add 1 to 2 BEIR points, but it chose to stay sparse.\n\n## **Key Takeaways**\n\n- SPARSEUP is Linkup Research’s first open model: a 149M-parameter sparse encoder under Apache 2.0.\n- It scores 56.4 nDCG@10 on BEIR-13, top among public sparse encoders under 150M, per Linkup.\n- 3 fixes drive it: a logit shift of 15, top-12 expansion per token, and case folding.\n- With identical data, it trails DenseOn by 1.52 points and LateOn by 2.5 on BEIR-13.\n- It reaches over 97% recall in about 380µs per query with Seismic on MS MARCO.\n\nCheck out the [**Model Weights**](https://huggingface.co/Linkup-Platform/linkup-sparseup-embed-v1) and [** Technical Details**](https://www.linkup.so/blog/introducing-sparseup-by-linkup). All credit goes to the researcher of this project. Also, feel free to follow us on **[Twitter](https://x.com/intent/follow?screen_name=marktechpost)** and don’t forget to join our **[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)** and Subscribe to **[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})**. Wait! are you on telegram? [now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)\n\nNeed to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/MJjjVDPS7whH8Ngs6)\n\nAsif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.", "url": "https://wpnews.pro/news/linkup-research-releases-sparseup-a-149m-parameter-open-source-sparse-embedding", "canonical_source": "https://www.marktechpost.com/2026/09/19/linkup-research-releases-sparseup/", "published_at": "2026-09-19 07:48:38+00:00", "updated_at": "2026-09-19 07:55:05.406327+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "natural-language-processing", "ai-tools"], "entities": ["Linkup", "SPARSEUP", "ModernBERT", "Hugging Face", "LightOn", "LateOn", "DenseOn", "BEIR-13"], "alternates": {"html": "https://wpnews.pro/news/linkup-research-releases-sparseup-a-149m-parameter-open-source-sparse-embedding", "markdown": "https://wpnews.pro/news/linkup-research-releases-sparseup-a-149m-parameter-open-source-sparse-embedding.md", "text": "https://wpnews.pro/news/linkup-research-releases-sparseup-a-149m-parameter-open-source-sparse-embedding.txt", "jsonld": "https://wpnews.pro/news/linkup-research-releases-sparseup-a-149m-parameter-open-source-sparse-embedding.jsonld"}}