cd /news/ai-research/papr-embed-code-sota-at-code-retriev… · home › topics › ai-research › article
[ARTICLE · art-172974] src=qdrant.tech ↗ pub= topic=ai-research verified=true sentiment=↑ positive

Papr-embed-code SOTA at code retrieval open source

Papr's papr-embed-code-1, a Qwen3-Embedding model further trained on code retrieval and released in open-weight 0.6B and 4B sizes, beat its base model on code it was not trained on, gaining 0.034 nDCG@10 on average for the 4B and 0.026 for the 0.6B, according to a Qdrant evaluation published November 18, 2026. The gains held on every held-out code task but varied widely, from little headroom on HumanEval (base 4B already at 0.98) to 0.070 on WikiSQL for the 4B, while prose performance slipped 0.003 for the 0.6B and 0.016 for the 4B. Qdrant also found the 4B's 2,560-dimension vectors compress to 328 bytes each at 1-bit TurboQuant, 31x smaller than the 10 KB float32 size.

recommendation

Use papr-embed-code-1 for code search but keep the Qwen3-Embedding base for prose-heavy corpora, and quantize 4B vectors to 1-bit TurboQuant to cut storage 31x with code retrieval intact.

more in playbook: Retrieval and RAG
read10 min views6 publishedNov 20, 2026
Papr-embed-code SOTA at code retrieval open source
Image: Qdrant (auto-discovered)

Dylan Couzon, Anush

· November 18, 2026

On this page:

If you search code, you’ve probably asked this: is a code-specific embedding model worth it, or is a strong general one good enough? Papr’s papr-embed-code-1 is a good test case. It’s Qwen3-Embedding trained further on code retrieval, with open weights in two sizes: 0.6B and 4B. Papr reports top scores on CoIR, a code retrieval benchmark. But it trained on most of CoIR’s training data, and its model card says so. Papr also reports the 4B at #1 on MTEB’s two held-out code leaderboards, RTEB (Code) and CodeRAG.

We asked a narrower question. Does it beat the model it started from on code it never saw? And what does it cost to run in Qdrant?

How We Tested #

We compared each papr-embed-code-1 model with the Qwen3-Embedding model it was trained from, with the same query instructions. The tasks come in three groups:

  • Code Papr didn’t train on:HumanEval andMBPP match a task description to the Python function that solves it.FreshStack matches a Stack Overflow question to the files of the library it’s about.WikiSQL matches a question about a table to its SQL query.
  • Code Papr trained on: three CoIR tasks, though the 4B skipped CodeTransOcean Contest.APPS matches a programming problem to its solution.CodeTransOcean Contest matches a program to the same program in another language.StackOverflowQA matches a question to its answer.
  • Prose: four BEIR datasets,NFCorpus ,ArguAna ,SCIDOCS , andFiQA .

Where Papr’s model cards score its own models, ours match to within 0.001 nDCG@10 on every task except the ones full of identical documents, marked † in the full results. There, the score depends on how ties break, which can change with batch size. On SyntheticText2SQL, we got 0.625 and 0.650 for the two sizes, against Papr’s 0.720 and 0.723.

Better on Code It Never Saw #

On code Papr didn’t train on, both sizes beat their base model. The 4B gained 0.034 nDCG@10 on average, and the 0.6B 0.026. On the CoIR tasks, the gains were similar.

The gain held on every one of those tasks, but its size varied a lot. On HumanEval, the 4B base model already scored 0.98, so there wasn’t much left to gain. On WikiSQL, the 4B gained 0.070.

Prose went the other way. Both sizes scored a little lower than their base model, by 0.003 for the 0.6B and 0.016 for the 4B. On TREC-COVID, a set of COVID-19 research papers, the 0.6B lost 0.065. If most of what you search is prose, keep the base model.

Compressed to 1 Bit, the 4B Beats the 0.6B #

Code search usually means a lot of vectors: a function, a file chunk, or a snippet each. The 4B’s vectors have 2,560 dimensions, which is 10 KB per vector in float32.

They compress very well. With 1-bit TurboQuant, each vector takes 328 bytes, 31x less. On our seven code tasks, 1-bit TurboQuant kept 99.7% of the 4B’s float32 score without rescoring, and all of it with rescoring at 2x oversampling.

That changes which model to pick. The 4B compressed to 1 bit scores higher than the 0.6B in float32, and takes 12x less RAM.

The 4B still costs more to run, because it embeds about four times slower than the 0.6B on the same GPU. And WikiSQL is the exception, where the 0.6B scored higher.

A Million Snippets in One Collection #

Our other code tasks have at most 20,000 documents, which makes compression look easy. So we also loaded CodeSearchNet-CC into one collection: about a million snippets in six languages, from Python to Ruby. Each query is the start of a function, and the right document is the rest of it. A payload filter keeps the search within the query’s language.

Compression held up at this size. At hnsw_ef 128, 1-bit TurboQuant with rescoring kept 99.5% of exact search’s score, against 99.7% for float32. It was faster too, at 1.0 ms per query against 1.8 ms, with the original vectors for rescoring already in memory.

The bigger lever was hnsw_ef, the number of candidates the search explores. Raising it from 128 to 256 added 0.002 to 0.004 nDCG@10 to every setup. That’s enough for rescored 1-bit vectors at 256 to beat float32 at 128. At 256, float32, 2-bit, and rescored 1-bit vectors all kept 99.8% or more, and the 1-bit search took 1.4 ms against 2.9 ms for float32.

Dropping the filter and searching all six languages at once cost 0.005 to 0.007.

What About BM25? #

Code is full of exact names, so keyword search seems like a natural partner. Qdrant 1.20 added a code tokenizer for full-text indexes. It splits getUserName and get_user_name into the same words. Qdrant can also score full-text matches with BM25, so a hybrid query can fuse BM25 with the dense search.

On our tasks, BM25 on its own was far behind dense search. On APPS, where the queries are problem statements and the documents are bare solutions, it scored 0.01 nDCG@10. The two share almost no words.

Fusing BM25 with dense search at equal weight made every task worse, by 0.02 to 0.17 for the 4B. Down-weighting BM25 helped, but only up to break-even. With BM25’s RRF weight at 0.1, the 4B averaged 0.0006 below dense search alone, and its best gain on any task was 0.002.

Our queries were questions, descriptions, and code, and for those, dense search alone was the better choice.

So, Does It Pay Off? #

For code search, yes. On code it never saw, papr-embed-code-1 beat the model it started from at both sizes, by about 0.03 nDCG@10. On prose it did a little worse, so if most of your corpus is prose, stay with the base model. The bigger choice is the size. Compressed to 1 bit, the 4B scored higher than the 0.6B in float32, in a twelfth of the RAM. If you can afford to embed with the 4B, we’d store it with 1-bit TurboQuant, keep rescoring on, and raise hnsw_ef before adding bits. Compare against an unquantized collection on your own queries, once with rescoring disabled and once with 2x oversampling. If you index several languages in one collection, filter by language.

We’d leave BM25 out to start. If your users search for exact identifiers, that’s the case to test it on, with a low weight.

The 0.6B is coming to FastEmbed in its next release.

Full Results #

Every model got the same query instructions. † Many identical documents, so the score depends on how ties break. Left out of the averages. ‡ Papr trained on its train split. § The 4B didn’t train on it.

| Task | Qwen3-Embedding-0.6B | papr-embed-code-1-0.6b | Qwen3-Embedding-4B | papr-embed-code-1-4b | 
|---|---|---|---|---|

| Metric | nDCG@10 | nDCG@10 | nDCG@10 | nDCG@10 | | Code, not trained on | | | | |

| [HumanEval](https://huggingface.co/datasets/embedding-benchmark/HumanEval) | 0.942 | 0.975 | 0.983 | 0.992 | 
| [MBPP](https://huggingface.co/datasets/embedding-benchmark/MBPP) | 0.881 | 0.908 | 0.914 | 0.936 | 
| [DS-1000](https://huggingface.co/datasets/embedding-benchmark/DS1000) † | 0.606 | 0.638 | 0.640 | 0.679 | 
| [FreshStack](https://huggingface.co/datasets/embedding-benchmark/FreshStack_mteb) | 0.378 | 0.386 | 0.433 | 0.470 | 
| [WikiSQL](https://huggingface.co/datasets/embedding-benchmark/WikiSQL_mteb) | 0.925 | 0.961 | 0.848 | 0.918 | 

| Code, trained on | | | | |

| [APPS](https://huggingface.co/datasets/CoIR-Retrieval/apps) | 0.752 | 0.793 | 0.891 | 0.925 | 
| [CodeTransOcean Contest](https://huggingface.co/datasets/CoIR-Retrieval/codetrans-contest) § | 0.864 | 0.909 | 0.918 | 0.930 | 
| [CodeTransOcean DL](https://huggingface.co/datasets/CoIR-Retrieval/codetrans-dl) † | 0.320 | 0.345 | 0.344 | 0.350 | 
| [CosQA](https://huggingface.co/datasets/CoIR-Retrieval/cosqa) † | 0.398 | 0.408 | 0.414 | 0.405 | 
| [StackOverflowQA](https://huggingface.co/datasets/CoIR-Retrieval/stackoverflow-qa) | 0.900 | 0.936 | 0.940 | 0.967 | 
| [SyntheticText2SQL](https://huggingface.co/datasets/CoIR-Retrieval/synthetic-text2sql) † | 0.619 | 0.625 | 0.644 | 0.650 | 
| [CodeSearchNet-CC](https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ccr) | N/A | 0.922 | N/A | N/A | 

| Prose | | | | |

| [SciFact](https://huggingface.co/datasets/mteb/scifact) ‡ | 0.699 | 0.824 | 0.786 | 0.868 | 
| [NFCorpus](https://huggingface.co/datasets/mteb/nfcorpus) | 0.363 | 0.375 | 0.410 | 0.421 | 
| [ArguAna](https://huggingface.co/datasets/mteb/arguana) | 0.693 | 0.665 | 0.751 | 0.725 | 
| [SCIDOCS](https://huggingface.co/datasets/mteb/scidocs) | 0.244 | 0.240 | 0.316 | 0.287 | 
| [FiQA](https://huggingface.co/datasets/mteb/fiqa) | 0.472 | 0.478 | 0.622 | 0.600 | 
| [TREC-COVID](https://huggingface.co/datasets/mteb/trec-covid) | 0.893 | 0.828 | N/A | N/A | 

Quantization #

Mean over seven code tasks (HumanEval, MBPP, FreshStack, WikiSQL, APPS, CodeTransOcean Contest, and StackOverflowQA), searched with HNSW. Rescoring uses 2x oversampling. Bytes count only the vectors searched in RAM.

Storage 0.6B Bytes 0.6B, No Rescoring 0.6B, With Rescoring 4B Bytes 4B, No Rescoring 4B, With Rescoring
Metric per vector nDCG@10 nDCG@10 per vector nDCG@10 nDCG@10
Original float32 4,096 0.838 N/A 10,240 0.877 N/A
turbo8 datatype 1,028 0.838 N/A 2,564 0.877 N/A
int8 1,028 0.838 0.838 2,564 0.877 0.877
4-bit TurboQuant 520 0.838 0.838 1,288 0.877 0.877
2-bit TurboQuant 264 0.837 0.838 648 0.876 0.877
1-bit TurboQuant 136 0.833 0.838 328 0.875 0.877
1-bit binary, 8-bit query 128 0.831 0.838 320 0.874 0.877

A Million Snippets #

papr-embed-code-1-0.6b on CodeSearchNet-CC, 1,005,474 snippets, each query filtered to its language. Exact search scores 0.923. Latency is Qdrant’s server-side time.

Storage Bytes per Vector hnsw_ef nDCG@10 p50 Latency (ms)
Original float32 4,096 64 0.914 1.1
Original float32 4,096 128 0.920 1.8
Original float32 4,096 256 0.922 2.9
turbo8 datatype 1,028 64 0.913 0.7
turbo8 datatype 1,028 128 0.920 1.1
turbo8 datatype 1,028 256 0.922 1.7
2-bit TurboQuant 264 64 0.910 0.7
2-bit TurboQuant 264 128 0.918 0.9
2-bit TurboQuant 264 256 0.921 1.4
1-bit TurboQuant 136 64 0.905 0.6
1-bit TurboQuant 136 128 0.913 0.9
1-bit TurboQuant 136 256 0.917 1.3
1-bit TurboQuant, 2x rescoring 136 64 0.910 1.3
1-bit TurboQuant, 2x rescoring 136 128 0.918 1.0
1-bit TurboQuant, 2x rescoring 136 256 0.922 1.4

papr-embed-code-1-4b. BM25 uses the code tokenizer. RRF uses Qdrant’s default k.

Task Dense BM25 RRF RRF, BM25 ×0.1 DBSF
Metric nDCG@10 nDCG@10 nDCG@10 nDCG@10 nDCG@10
| [HumanEval](https://huggingface.co/datasets/embedding-benchmark/HumanEval) | 0.992 | 0.393 | 0.884 | 0.992 | 0.888 | 
| [MBPP](https://huggingface.co/datasets/embedding-benchmark/MBPP) | 0.936 | 0.385 | 0.847 | 0.936 | 0.805 | 

| FreshStack | 0.470 | 0.264 | 0.418 | 0.472 | 0.406 | | WikiSQL | 0.918 | 0.513 | 0.805 | 0.917 | 0.831 |

| [APPS](https://huggingface.co/datasets/CoIR-Retrieval/apps) | 0.925 | 0.010 | 0.756 | 0.925 | 0.854 | 
| [CodeTransOcean Contest](https://huggingface.co/datasets/CoIR-Retrieval/codetrans-contest) | 0.930 | 0.670 | 0.912 | 0.931 | 0.905 | 
| [StackOverflowQA](https://huggingface.co/datasets/CoIR-Retrieval/stackoverflow-qa) | 0.967 | 0.635 | 0.902 | 0.967 | 0.897 | 

| SciFact | 0.868 | 0.668 | 0.811 | 0.868 | 0.792 | | NFCorpus | 0.421 | 0.307 | 0.395 | 0.419 | 0.398 | | ArguAna | 0.728 | 0.411 | 0.623 | 0.726 | 0.633 | | SCIDOCS | 0.287 | 0.151 | 0.238 | 0.283 | 0.223 |

| FiQA | 0.600 | 0.237 | 0.521 | 0.599 | 0.485 | Thanks to Papr for releasing the code models with open weights, and for publishing exactly how they trained and evaluated them.

── more in #ai-research 4 stories · sorted by recency
── more on @papr 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/papr-embed-code-sota…] indexed:0 read:10min 2026-11-20 · —