{"slug": "papr-embed-code-sota-at-code-retrieval-open-source", "title": "Papr-embed-code SOTA at code retrieval open source", "summary": "Papr's papr-embed-code-1, a Qwen3-Embedding model further trained on code retrieval and released in open-weight 0.6B and 4B sizes, beat its base model on code it was not trained on, gaining 0.034 nDCG@10 on average for the 4B and 0.026 for the 0.6B, according to a Qdrant evaluation published November 18, 2026. The gains held on every held-out code task but varied widely, from little headroom on HumanEval (base 4B already at 0.98) to 0.070 on WikiSQL for the 4B, while prose performance slipped 0.003 for the 0.6B and 0.016 for the 4B. Qdrant also found the 4B's 2,560-dimension vectors compress to 328 bytes each at 1-bit TurboQuant, 31x smaller than the 10 KB float32 size.", "body_md": "# Does a Code-Tuned Embedding Model Pay Off? papr-embed-code-1 in Qdrant\n\nDylan Couzon, Anush\n\n·\nNovember 18, 2026\n\nOn this page:\n\nIf you search code, you’ve probably asked this: is a code-specific embedding model worth it, or is a strong general one good enough?\n\nPapr’s [papr-embed-code-1](https://paprai.substack.com/p/introducing-papr-embed-schema-native) is a good test case. It’s [Qwen3-Embedding](https://huggingface.co/Qwen/Qwen3-Embedding-4B) trained further on code retrieval, with open weights in two sizes: [0.6B](https://huggingface.co/papr-ai/papr-embed-code-1-0.6b) and [4B](https://huggingface.co/papr-ai/papr-embed-code-1-4b). Papr reports top scores on CoIR, a code retrieval benchmark. But it trained on most of CoIR’s training data, and its model card says so. Papr also [reports the 4B at #1](https://paprai.substack.com/p/introducing-papr-embed-schema-native) on MTEB’s two held-out code leaderboards, RTEB (Code) and CodeRAG.\n\nWe asked a narrower question. Does it beat the model it started from on code it never saw? And what does it cost to run in Qdrant?\n\n## How We Tested\n\nWe compared each papr-embed-code-1 model with the Qwen3-Embedding model it was trained from, with the same query instructions. The tasks come in three groups:\n\n- **Code Papr didn’t train on:**[HumanEval](https://huggingface.co/datasets/embedding-benchmark/HumanEval) and[MBPP](https://huggingface.co/datasets/embedding-benchmark/MBPP) match a task description to the Python function that solves it.[FreshStack](https://huggingface.co/datasets/embedding-benchmark/FreshStack_mteb) matches a Stack Overflow question to the files of the library it’s about.[WikiSQL](https://huggingface.co/datasets/embedding-benchmark/WikiSQL_mteb) matches a question about a table to its SQL query.\n- **Code Papr trained on:** three CoIR tasks, though the 4B skipped CodeTransOcean Contest.[APPS](https://huggingface.co/datasets/CoIR-Retrieval/apps) matches a programming problem to its solution.[CodeTransOcean Contest](https://huggingface.co/datasets/CoIR-Retrieval/codetrans-contest) matches a program to the same program in another language.[StackOverflowQA](https://huggingface.co/datasets/CoIR-Retrieval/stackoverflow-qa) matches a question to its answer.\n- **Prose:** four BEIR datasets,[NFCorpus](https://huggingface.co/datasets/mteb/nfcorpus) ,[ArguAna](https://huggingface.co/datasets/mteb/arguana) ,[SCIDOCS](https://huggingface.co/datasets/mteb/scidocs) , and[FiQA](https://huggingface.co/datasets/mteb/fiqa) .\n\nWhere Papr’s model cards score its own models, ours match to within 0.001 nDCG@10 on every task except the ones full of identical documents, marked † in the [full results](#full-results). There, the score depends on how ties break, which can change with batch size. On [SyntheticText2SQL](https://huggingface.co/datasets/CoIR-Retrieval/synthetic-text2sql), we got 0.625 and 0.650 for the two sizes, against Papr’s 0.720 and 0.723.\n\n## Better on Code It Never Saw\n\nOn code Papr didn’t train on, both sizes beat their base model. The 4B gained 0.034 nDCG@10 on average, and the 0.6B 0.026. On the CoIR tasks, the gains were similar.\n\nThe gain held on every one of those tasks, but its size varied a lot. On HumanEval, the 4B base model already scored 0.98, so there wasn’t much left to gain. On WikiSQL, the 4B gained 0.070.\n\nProse went the other way. Both sizes scored a little lower than their base model, by 0.003 for the 0.6B and 0.016 for the 4B. On TREC-COVID, a set of COVID-19 research papers, the 0.6B lost 0.065. If most of what you search is prose, keep the base model.\n\n## Compressed to 1 Bit, the 4B Beats the 0.6B\n\nCode search usually means a lot of vectors: a function, a file chunk, or a snippet each. The 4B’s vectors have 2,560 dimensions, which is 10 KB per vector in float32.\n\nThey compress very well. With 1-bit [TurboQuant](https://qdrant.tech/documentation/manage-data/quantization/#turboquant-quantization), each vector takes 328 bytes, 31x less. On our seven code tasks, 1-bit TurboQuant kept 99.7% of the 4B’s float32 score without rescoring, and all of it with rescoring at 2x oversampling.\n\nThat changes which model to pick. The 4B compressed to 1 bit scores higher than the 0.6B in float32, and takes 12x less RAM.\n\nThe 4B still costs more to run, because it embeds about four times slower than the 0.6B on the same GPU. And WikiSQL is the exception, where the 0.6B scored higher.\n\n## A Million Snippets in One Collection\n\nOur other code tasks have at most 20,000 documents, which makes compression look easy. So we also loaded [CodeSearchNet-CC](https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ccr) into one collection: about a million snippets in six languages, from Python to Ruby. Each query is the start of a function, and the right document is the rest of it. A [payload filter](https://qdrant.tech/documentation/search/filtering/) keeps the search within the query’s language.\n\nCompression held up at this size. At `hnsw_ef` 128, 1-bit TurboQuant with rescoring kept 99.5% of exact search’s score, against 99.7% for float32. It was faster too, at 1.0 ms per query against 1.8 ms, with the original vectors for rescoring already in memory.\n\nThe bigger lever was `hnsw_ef`, the number of candidates the search explores. Raising it from 128 to 256 added 0.002 to 0.004 nDCG@10 to every setup. That’s enough for rescored 1-bit vectors at 256 to beat float32 at 128. At 256, float32, 2-bit, and rescored 1-bit vectors all kept 99.8% or more, and the 1-bit search took 1.4 ms against 2.9 ms for float32.\n\nDropping the filter and searching all six languages at once cost 0.005 to 0.007.\n\n## What About BM25?\n\nCode is full of exact names, so keyword search seems like a natural partner. Qdrant 1.20 added a [code tokenizer](https://qdrant.tech/documentation/manage-data/indexing/#code) for full-text indexes. It splits `getUserName` and `get_user_name` into the same words. Qdrant can also [score full-text matches with BM25](https://qdrant.tech/documentation/search/text-search/full-text-search/), so a [hybrid query](https://qdrant.tech/documentation/search/hybrid-queries/) can fuse BM25 with the dense search.\n\nOn our tasks, BM25 on its own was far behind dense search. On APPS, where the queries are problem statements and the documents are bare solutions, it scored 0.01 nDCG@10. The two share almost no words.\n\nFusing BM25 with dense search at equal weight made every task worse, by 0.02 to 0.17 for the 4B. Down-weighting BM25 helped, but only up to break-even. With BM25’s [RRF weight](https://qdrant.tech/documentation/search/hybrid-queries/#weighted-rrf) at 0.1, the 4B averaged 0.0006 below dense search alone, and its best gain on any task was 0.002.\n\nOur queries were questions, descriptions, and code, and for those, dense search alone was the better choice.\n\n## So, Does It Pay Off?\n\nFor code search, yes. On code it never saw, papr-embed-code-1 beat the model it started from at both sizes, by about 0.03 nDCG@10. On prose it did a little worse, so if most of your corpus is prose, stay with the base model.\n\nThe bigger choice is the size. Compressed to 1 bit, the 4B scored higher than the 0.6B in float32, in a twelfth of the RAM. If you can afford to embed with the 4B, we’d store it with 1-bit [TurboQuant](https://qdrant.tech/documentation/manage-data/quantization/#turboquant-quantization), keep rescoring on, and raise `hnsw_ef` before adding bits. Compare against an unquantized collection on your own queries, once with [rescoring disabled](https://qdrant.tech/documentation/manage-data/quantization/#searching-with-quantization) and once with 2x oversampling. If you index several languages in one collection, filter by language.\n\nWe’d leave BM25 out to start. If your users search for exact identifiers, that’s the case to test it on, with a low weight.\n\nThe 0.6B is coming to [FastEmbed](https://qdrant.tech/documentation/fastembed/) in its next release.\n\n## Full Results\n\n## Exact Search\n\nEvery model got the same query instructions. † Many identical documents, so the score depends on how ties break. Left out of the averages. ‡ Papr trained on its train split. § The 4B didn’t train on it.\n\n| Task | Qwen3-Embedding-0.6B | papr-embed-code-1-0.6b | Qwen3-Embedding-4B | papr-embed-code-1-4b | \n|---|---|---|---|---|\n| *Metric* | *nDCG@10* | *nDCG@10* | *nDCG@10* | *nDCG@10* | \n| *Code, not trained on* |  |  |  |  | \n| [HumanEval](https://huggingface.co/datasets/embedding-benchmark/HumanEval) | 0.942 | 0.975 | 0.983 | 0.992 | \n| [MBPP](https://huggingface.co/datasets/embedding-benchmark/MBPP) | 0.881 | 0.908 | 0.914 | 0.936 | \n| [DS-1000](https://huggingface.co/datasets/embedding-benchmark/DS1000) † | 0.606 | 0.638 | 0.640 | 0.679 | \n| [FreshStack](https://huggingface.co/datasets/embedding-benchmark/FreshStack_mteb) | 0.378 | 0.386 | 0.433 | 0.470 | \n| [WikiSQL](https://huggingface.co/datasets/embedding-benchmark/WikiSQL_mteb) | 0.925 | 0.961 | 0.848 | 0.918 | \n| *Code, trained on* |  |  |  |  | \n| [APPS](https://huggingface.co/datasets/CoIR-Retrieval/apps) | 0.752 | 0.793 | 0.891 | 0.925 | \n| [CodeTransOcean Contest](https://huggingface.co/datasets/CoIR-Retrieval/codetrans-contest) § | 0.864 | 0.909 | 0.918 | 0.930 | \n| [CodeTransOcean DL](https://huggingface.co/datasets/CoIR-Retrieval/codetrans-dl) † | 0.320 | 0.345 | 0.344 | 0.350 | \n| [CosQA](https://huggingface.co/datasets/CoIR-Retrieval/cosqa) † | 0.398 | 0.408 | 0.414 | 0.405 | \n| [StackOverflowQA](https://huggingface.co/datasets/CoIR-Retrieval/stackoverflow-qa) | 0.900 | 0.936 | 0.940 | 0.967 | \n| [SyntheticText2SQL](https://huggingface.co/datasets/CoIR-Retrieval/synthetic-text2sql) † | 0.619 | 0.625 | 0.644 | 0.650 | \n| [CodeSearchNet-CC](https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ccr) | N/A | 0.922 | N/A | N/A | \n| *Prose* |  |  |  |  | \n| [SciFact](https://huggingface.co/datasets/mteb/scifact) ‡ | 0.699 | 0.824 | 0.786 | 0.868 | \n| [NFCorpus](https://huggingface.co/datasets/mteb/nfcorpus) | 0.363 | 0.375 | 0.410 | 0.421 | \n| [ArguAna](https://huggingface.co/datasets/mteb/arguana) | 0.693 | 0.665 | 0.751 | 0.725 | \n| [SCIDOCS](https://huggingface.co/datasets/mteb/scidocs) | 0.244 | 0.240 | 0.316 | 0.287 | \n| [FiQA](https://huggingface.co/datasets/mteb/fiqa) | 0.472 | 0.478 | 0.622 | 0.600 | \n| [TREC-COVID](https://huggingface.co/datasets/mteb/trec-covid) | 0.893 | 0.828 | N/A | N/A | \n\n## Quantization\n\nMean over seven code tasks (HumanEval, MBPP, FreshStack, WikiSQL, APPS, CodeTransOcean Contest, and StackOverflowQA), searched with HNSW. Rescoring uses 2x oversampling. Bytes count only the vectors searched in RAM.\n\n| Storage | 0.6B Bytes | 0.6B, No Rescoring | 0.6B, With Rescoring | 4B Bytes | 4B, No Rescoring | 4B, With Rescoring | \n|---|---|---|---|---|---|---|\n| *Metric* | *per vector* | *nDCG@10* | *nDCG@10* | *per vector* | *nDCG@10* | *nDCG@10* | \n| Original float32 | 4,096 | 0.838 | N/A | 10,240 | 0.877 | N/A | \n| turbo8 datatype | 1,028 | 0.838 | N/A | 2,564 | 0.877 | N/A | \n| int8 | 1,028 | 0.838 | 0.838 | 2,564 | 0.877 | 0.877 | \n| 4-bit TurboQuant | 520 | 0.838 | 0.838 | 1,288 | 0.877 | 0.877 | \n| 2-bit TurboQuant | 264 | 0.837 | 0.838 | 648 | 0.876 | 0.877 | \n| 1-bit TurboQuant | 136 | 0.833 | 0.838 | 328 | 0.875 | 0.877 | \n| 1-bit binary, 8-bit query | 128 | 0.831 | 0.838 | 320 | 0.874 | 0.877 | \n\n## A Million Snippets\n\npapr-embed-code-1-0.6b on CodeSearchNet-CC, 1,005,474 snippets, each query filtered to its language. Exact search scores 0.923. Latency is Qdrant’s server-side time.\n\n| Storage | Bytes per Vector | hnsw_ef | nDCG@10 | p50 Latency (ms) | \n|---|---|---|---|---|\n| Original float32 | 4,096 | 64 | 0.914 | 1.1 | \n| Original float32 | 4,096 | 128 | 0.920 | 1.8 | \n| Original float32 | 4,096 | 256 | 0.922 | 2.9 | \n| turbo8 datatype | 1,028 | 64 | 0.913 | 0.7 | \n| turbo8 datatype | 1,028 | 128 | 0.920 | 1.1 | \n| turbo8 datatype | 1,028 | 256 | 0.922 | 1.7 | \n| 2-bit TurboQuant | 264 | 64 | 0.910 | 0.7 | \n| 2-bit TurboQuant | 264 | 128 | 0.918 | 0.9 | \n| 2-bit TurboQuant | 264 | 256 | 0.921 | 1.4 | \n| 1-bit TurboQuant | 136 | 64 | 0.905 | 0.6 | \n| 1-bit TurboQuant | 136 | 128 | 0.913 | 0.9 | \n| 1-bit TurboQuant | 136 | 256 | 0.917 | 1.3 | \n| 1-bit TurboQuant, 2x rescoring | 136 | 64 | 0.910 | 1.3 | \n| 1-bit TurboQuant, 2x rescoring | 136 | 128 | 0.918 | 1.0 | \n| 1-bit TurboQuant, 2x rescoring | 136 | 256 | 0.922 | 1.4 | \n\n## BM25 Hybrid Search\n\npapr-embed-code-1-4b. BM25 uses the code tokenizer. RRF uses Qdrant’s default k.\n\n| Task | Dense | BM25 | RRF | RRF, BM25 ×0.1 | DBSF | \n|---|---|---|---|---|---|\n| *Metric* | *nDCG@10* | *nDCG@10* | *nDCG@10* | *nDCG@10* | *nDCG@10* | \n| [HumanEval](https://huggingface.co/datasets/embedding-benchmark/HumanEval) | 0.992 | 0.393 | 0.884 | 0.992 | 0.888 | \n| [MBPP](https://huggingface.co/datasets/embedding-benchmark/MBPP) | 0.936 | 0.385 | 0.847 | 0.936 | 0.805 | \n| [FreshStack](https://huggingface.co/datasets/embedding-benchmark/FreshStack_mteb) | 0.470 | 0.264 | 0.418 | 0.472 | 0.406 | \n| [WikiSQL](https://huggingface.co/datasets/embedding-benchmark/WikiSQL_mteb) | 0.918 | 0.513 | 0.805 | 0.917 | 0.831 | \n| [APPS](https://huggingface.co/datasets/CoIR-Retrieval/apps) | 0.925 | 0.010 | 0.756 | 0.925 | 0.854 | \n| [CodeTransOcean Contest](https://huggingface.co/datasets/CoIR-Retrieval/codetrans-contest) | 0.930 | 0.670 | 0.912 | 0.931 | 0.905 | \n| [StackOverflowQA](https://huggingface.co/datasets/CoIR-Retrieval/stackoverflow-qa) | 0.967 | 0.635 | 0.902 | 0.967 | 0.897 | \n| [SciFact](https://huggingface.co/datasets/mteb/scifact) | 0.868 | 0.668 | 0.811 | 0.868 | 0.792 | \n| [NFCorpus](https://huggingface.co/datasets/mteb/nfcorpus) | 0.421 | 0.307 | 0.395 | 0.419 | 0.398 | \n| [ArguAna](https://huggingface.co/datasets/mteb/arguana) | 0.728 | 0.411 | 0.623 | 0.726 | 0.633 | \n| [SCIDOCS](https://huggingface.co/datasets/mteb/scidocs) | 0.287 | 0.151 | 0.238 | 0.283 | 0.223 | \n| [FiQA](https://huggingface.co/datasets/mteb/fiqa) | 0.600 | 0.237 | 0.521 | 0.599 | 0.485 | \n\nThanks to Papr for releasing the code models with open weights, and for publishing exactly how they trained and evaluated them.", "url": "https://wpnews.pro/news/papr-embed-code-sota-at-code-retrieval-open-source", "canonical_source": "https://qdrant.tech/blog/papr-embed-code/", "published_at": "2026-11-20 02:27:48+00:00", "updated_at": "2026-11-20 02:44:30.728554+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "ai-tools", "ai-infrastructure"], "entities": ["Papr", "papr-embed-code-1", "Qwen3-Embedding", "Qdrant", "CoIR", "HumanEval", "WikiSQL", "TurboQuant"], "also_reported_by": [], "takeaway": "Papr's code-tuned papr-embed-code-1 beats its Qwen3-Embedding base by 0.034 nDCG@10 (4B) and 0.026 (0.6B) on unseen code but loses 0.016 and 0.003 on prose, and its 4B vectors shrink from 10 KB float32 to 328 bytes at 1-bit TurboQuant.", "recommendation": {"advice": "Use papr-embed-code-1 for code search but keep the Qwen3-Embedding base for prose-heavy corpora, and quantize 4B vectors to 1-bit TurboQuant to cut storage 31x with code retrieval intact.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/papr-embed-code-sota-at-code-retrieval-open-source", "markdown": "https://wpnews.pro/news/papr-embed-code-sota-at-code-retrieval-open-source.md", "text": "https://wpnews.pro/news/papr-embed-code-sota-at-code-retrieval-open-source.txt", "jsonld": "https://wpnews.pro/news/papr-embed-code-sota-at-code-retrieval-open-source.jsonld"}}