Before we had users, we wrote the CVs ourselves The team behind kandidate.ai, a job-matching product for engineers in Germany, built a synthetic evaluation harness that turns job ads into test candidates to measure retrieval quality before real users existed. The pipeline fuses dense, keyword, and domain retrieval legs with reciprocal rank fusion, then re-ranks with skill bonuses, German-level and seniority multipliers, and diversity caps. The team found that a self-retrieval test scored recall@5 of 1.000 over 208 anchors, showing the test was too easy, and that on 2 July 2026 about 14% of judge-rated relevant jobs were missing from the embedding's top 150, which led them to add a keyword leg. kandidate.ai matches an engineer's CV against live tech jobs in Germany, in English or German. Early on we hit the problem every search product hits: how do you know the results are good before people use it? On day one you have job ads and nothing else. So we turned job ads into test candidates. We took a real job, built a strong-fit candidate from its own title and skills, and checked whether the matcher put that job near the top. A CV built from an ad should find that ad. Making test data without users turned out to be the easy part. Keeping the measurement tied to what we actually serve was harder, and our own numbers fooled us three times: CV ──► parsed profile titles, skills, years, German level │ ├─► dense leg: embedding, pgvector cosine top 300 ├─► keyword leg: Postgres full-text search top 100 └─► domain leg: jobs in the same tech stack top 150 │ ▼ reciprocal rank fusion k = 60 → 300 candidates │ ▼ re-rank: cosine + capped skill bonus + small boosts, clamped at 1.0 × German-level and seniority multipliers × your own 👍/👎, if any sort key adds a small recency lift │ ▼ diversity: max 3 per employer backfilled , repeated ads collapsed │ ▼ the list you see We fuse by rank because cosine similarity and Postgres's ts rank cd can't be compared on one scale. Fusion only decides which 300 rows get scored. After that, its order is supposed to be thrown away. The re-ranker is where the product's opinions live. Cosine similarity says "this ad is about the same things as this CV". It doesn't know that the ad asks for fluent German https://kandidate.ai/blog/how-much-german-engineering-job-ads-ask-for and the reader has A2, or that the reader holds 11 of the 12 tools it lists. The re-ranker does. The keyword leg came from a measurement. On 2 July 2026, for 4 test personas, about 14% of the jobs our judge rated relevant were missing from the embedding's top 150. A simple keyword match on 4 of the persona's skills found them, and for one persona the figure was 23%. No re-ranker weight can rescue a job that never enters the pool. Our bet on why: embeddings blur exact identifiers like PROFINET or EN ISO 13849, which cross from an English CV to a German ad unchanged. eval:synthetic covers every region × tech-stack cell in the corpus: /api/match calls. It starts from a ready-made profile, so CV parsing isn't under test. js // simplified function profileFromJob job: Job, rng: = number { const keepShare = 0.6 + rng 0.2; // 60–80% of must-haves return { titles: job.title , skills: ...sample job.requiredSkills, keepShare, rng , ...pickUpTo job.niceToHave, 2, rng , // 0–2 nice-to-haves , years: yearsFor job.seniority , germanLevel: meetsBar job.germanRequired , // no language penalty }; } A seeded PRNG mulberry32 makes the candidates repeatable, but the corpus changes nightly. Two same-seed runs three hours apart 23 September 2026 differed by up to 0.0045, so we treat a change under 0.005 as drift. This is a self-retrieval test. The candidate carries the anchor's exact title and is built to meet its German bar, so mostly it catches a broken pipeline, not a weak ranker. An early baseline 13 June 2026 read recall@5 = 1.000 over 208 anchors. That told us the test was easy, not that the ranker was good. Self-retrieval can't tell you whether the other 19 jobs in a list are any good. For that we wrote 14 synthetic personas: an embedded C++ engineer, a Berlin full-stack developer, an IC/ASIC designer, someone pivoting careers, and others. For each persona we build a judged pool from the embedding's top 150 plus 100 keyword hits. Gemini 2.5 Flash, at temperature 0, judges every job in it on one question: would this person be a credible applicant? It sees the persona's titles, years, skills and German level, plus the ad's title, skills, language fields and first 900 characters. It returns a verdict, a confidence and a written reason, and a job counts as relevant only at confidence ≥ 0.6. It isn't given posting date, salary or visa-related fields, which the ranker does weigh. We haven't measured how often it agrees with human raters. Later came golden cases : fixed pools of jobs with relevance labels, some written by hand and some exported from 👍/👎 ratings. npm run eval re-ranks them offline and reports P@5, nDCG@10 and MRR. | Metric | Definition | Blind to | |---|---|---| | Anchor recall@5 | Share of anchors in the top 5 of the 20 served | Whether the other 19 jobs are any good | | MRR | Mean of 1/rank of the anchor; 0 if it isn't in the 20 served | The same | | Precision@k | Share of the top k the judge rated relevant | Everything the judge isn't given | | Pool recall | Share of judged-relevant jobs that reach the 300-job pool | Jobs retrieval never returned. The labels come from retrieval's own results, so it reads high | Two harnesses were meant to rank exactly like production. Neither did. ✓ no aggregate regression . The same drift happened outside the evals. Two implementations of "which tech stack is this job?" disagreed on a third of live jobs 10 October 2026 . The home page's own search could reach only about half of a reader's results-page top 8 9 October . A copied ranker doesn't make an eval fail. It makes it blind, and nothing in the output says so. The fix: rankPool . js // simplified const STAMPS = "scoringVersion", "retrievalArm", "seed", "pool", "servedLimit", "cluster", "perDomainCap", as const; function comparable baseline: Stamps, run: Stamps { const differs = STAMPS.filter k = baseline k === undefined || baseline k == run k , ; return differs.length === 0 ? { ok: true } : { ok: false, differs }; // prints "deltas suppressed" } A missing stamp counts as a mismatch. Once the harness was fixed, this check refused to compare the new baseline with the old one, because seven scorer versions had landed in between. The 18 October 2026 run recall@5 0.970, MRR 0.932 became a fresh start. The cost wasn't a number. It was three months of numbers about a system no one was served. On 9 October 2026 we read back the lists we had served over the previous 90 days. 81.5% of served rows shared their score with another row in the same list , and so did 54.7% of rows in the top 5. The score was rounded to two decimals inside the scoring step, then sorted with a stable sort, so tied rows kept their input order: retrieval-fusion order. In 28% of adjacent tied pairs, the lower-ranked row had the higher cosine. Removing the rounding v10 made exact ties rare but not impossible. Arrival order still broke them, and the home page was about to re-rank the same pool in a different arrival order. So v12 made the order total. // before v9 : rounded during scoring, then a stable sort row.score = Math.round raw 100 / 100; rows.sort a, b = b.score - a.score ; // ties keep input order // after v10, v12 : full precision, then a total order rows.sort a, b = b.rankScore - a.rankScore || // score + recency lift v15+ b.cosine - a.cosine || a.id < b.id ? -1 : a.id b.id ? 1 : 0 , // code units ; // round only for display The tie-break avoids localeCompare on purpose, because a locale-dependent order is another hidden input. Both fixes were about determinism, not relevance for v12, the golden eval didn't move . The list is now sorted by the number we say it's sorted by. This happened twice. July 2026. The keyword leg raised pool recall, but recall@20 after re-ranking stayed flat. The cosine-anchored re-ranker re-buried what the keyword leg had rescued. November 2026. It seemed obvious that 500 candidates would beat 300. On 13 November we tested it on 8 of our 14 personas: the relabelling hit its $3 spend cap, and the other 6 had stale labels. The judge rated every job in the re-ranked top 100 at both pool sizes, so no unjudged row counted as a miss. | mean of 8 personas, 13 Nov 2026 | Pool 300 | Pool 500 | |---|---|---| | Pool recall | 0.852 | 0.930 | | Precision@20 after re-ranking | 0.638 | 0.625 | | Precision@100 after re-ranking | 0.546 | 0.510 | The top 20 barely moved. The top 100 got worse: the 169 rows that entered at pool 500 were 31% relevant, and the 169 they pushed out were 48% relevant. We kept 300. Limits: 8 personas is small, and the judge isn't given recency, salary or visa-related fields. But two earlier, cheaper methods pointed the same way. One was a sampled replay on 10 November, where jobs new to the top 100 were 37% relevant against 39% for the positions they competed with: too close to call. Judging every row removed the bias from unjudged rows. The study cost $3.68, about 80% of it thinking tokens. Gemini 2.5 Flash thinks by default unless you set a thinking budget. Both times, the bottleneck was ranking inside the pool, not the size of the pool. Every scoring change bumps SCORING VERSION , and every baseline records it. On 20 November 2026 we're on v21. Some of the most useful versions came from people reading the output, the code or the logs: Here's what the matcher looks like from the reader's side, from uploading a CV to applying 97 seconds : A question for anyone who has done it: if you've checked an LLM relevance judge against human raters, how large a sample did you need before you trusted it? If you're an engineer looking at jobs in Germany, kandidate.ai matches your CV, in English or German, and ranks the results for your German level. There's a free plan: kandidate.ai https://kandidate.ai/?utm source=devto&utm medium=post&utm campaign=matching engine eval .