{"slug": "is-your-agent-s-llm-judge-unreliable-use-logprobs-to-make-scores-comparable", "title": "Is Your Agent's LLM Judge Unreliable? Use Logprobs to Make Scores Comparable", "summary": "A developer behind langchainrust 0.30.0 has published an approach for making LLM-judge scores comparable in best-of-N agent selection by requesting logprobs and top_logprobs from the judge model instead of relying on the returned integer. The method computes an expected score and confidence from the token distribution, so two candidates that both score 8 can be distinguished by whether the judge assigned 8 a probability of 0.90 or 0.50. The implementation, in lc-evaluation's verifier and judge_reliability modules, requires only a one-line config change, with_top_logprobs(5), and the author cautions that the resulting confidence reflects the model's certainty about its token choice, not a calibrated measure of answer quality.", "body_md": "One sentence: **the model gives you an integer; a decision needs a measurement.**\n\nThis article gives the fix first (three steps), then explains why it works,\n\nand ends with the engineering constraints you cannot skip.\n\nThe code below comes from the real implementation in langchainrust 0.30.0:\n\n`lc-core/src/language_models/chat.rs`,\n\n`lc-evaluation/src/verifier.rs`,\n\n`lc-evaluation/src/judge_reliability.rs`.\n\nThis is a common agent scenario:\n\n**the agent produces several candidates at one step (answers, plans, or tool calls), then calls another LLM as a judge and picks one to continue.**\n\nIn agent systems this is called **best-of-N / verifier-guided selection**.\n\nIt shows up in:\n\nThe common factor: **the judge is itself an LLM.**\n\nSo the judge returns:\n\n```\nCandidate A: 8\nCandidate B: 8\n```\n\n**Which one do you pick?**\n\nYou can't. The integer 8 only tells you \"it chose 8\".\n\nIt does not tell you \"how confident it was\".\n\nThe distribution behind that same call looks like this:\n\n```\nCandidate A: 8 @ 0.90, 7 @ 0.08, 9 @ 0.02\nCandidate B: 8 @ 0.50, 7 @ 0.40, 9 @ 0.10\n```\n\nA is very sure about 8. B only picked 8 by a 0.50-to-0.40 margin over 7 —\n\nit had a 40% chance of actually wanting to say 7.\n\n**Two 8s are not the same 8.**\n\nYou do not need a different model, and you do not need to retrain anything.\n\nYou only need to take back the information the model already computed\n\nbut does not return by default:\n\n`logprobs` and `top_logprobs`.\nSame two candidates, two selection methods:\n\n| Candidate | Judge output | Integer only | Verifier (expected / confidence) | \n|---|---|---|---|\n| A | 8 | 8 | **7.94 / 0.90** | \n| B | 8 | 8 | **7.70 / 0.50** | \n\nThat is the whole fix. In code it is one extra line:\n\n``` js\nlet config = OpenAIConfig::from_env_result()?.with_top_logprobs(5);\n```\n\nThe rest of the article unpacks this:\n\nThe first reaction is usually: isn't the judge \"just directly giving a score\"?\n\nWhy would there be a probability distribution?\n\nBecause the model has no \"scoring module\" and no regression head that emits\n\na continuous value. It does exactly what it does when writing a poem or code:\n\n**predict the next token.**\n\nThe prompt looks like this:\n\n```\nScore the answer below on a 0-9 scale.\nOutput exactly one digit.\nAnswer: ...\nScore:\n```\n\nThe model's job is to complete the token after `Score:`.\n\nIt computes one softmax over the whole vocabulary; `8` has the highest\n\nprobability, so it emits `8`.\n\nSo:\n\n```\nWhat you see:      8\nWhat the model did: computed a probability over the vocabulary,\n                   8 ranked first, emitted 8\n```\n\n**\"Direct scoring\" is an analogy to human behavior, not the model's operation.**\n\nBecause of that, there is always a distribution behind the chosen token.\n\nThe default API response simply does not return it.\n\n| Layer | Content | Who provides it | \n|---|---|---|\n| Choice | It emitted 8 | The model | \n| Confidence | Whether 8 had probability 0.9 or 0.5 | The model ( `logprobs` ) | \n| True quality | How good an 8 actually is | **The model does not measure this** | \n\nThe third layer is the easiest to misread:\n\n`0.90` does not mean \"this answer has a 90% chance of being an 8-quality answer\".\n\nIt means \"the model is very sure the next token is 8\".\n\nThe model gives you an **opinion and the strength of that opinion**,\n\nnot a calibrated quality measurement.\n\nThis is not a thermometer. It is a multiple-choice answer.\n\nYou ask a person, \"Rate this answer from 1 to 10.\" They say \"8\".\n\nYou see the answer; you do not see whether they said it instantly\n\nor hesitated between 7 and 8.\n\nThe model is different: its hesitation is digitized and readable.\n\n`0.90` is an instant answer; `0.50` is a hesitant one.\n\nStep 2 says \"take the distribution back\". This section explains exactly\n\nwhat you are taking, what the fields are called, and what the response looks like.\n\nThe model does not generate characters or words. It generates **tokens**.\n\nA token can be:\n\n```\n8\n10\npoints\n(a space)\nthe\n```\n\nAt each step the model does one thing: **given the text so far, predict the next token.** It computes a probability over the entire\n\n```\nnext token \"8\"      probability 0.90\nnext token \"7\"      probability 0.08\nnext token \"9\"      probability 0.02\nnext token \"points\" probability 0.001\n...\n```\n\n`logprob` is\nIf the probability is `p`, the `logprob` is `ln(p)`.\n\n| Probability p | `logprob = ln(p)` | \n|---|---|\n| 0.90 | **-0.105** | \n| 0.50 | **-0.693** | \n| 0.08 | **-2.526** | \n| 0.02 | **-3.912** | \n| 0.001 | **-6.908** | \n\nWhy logarithms instead of probabilities?\n\nConverting back is one step:\n\n```\nprobability = exp(logprob)\n```\n\n`exp(-0.105) ≈ 0.90`.\n\nIn the request:\n\n```\n{ \"logprobs\": true }\n```\n\nThis means:\n\nReturn the log probability of **the token you actually generated**.\n\nIf the model generated `8`, the response includes:\n\n```\n{ \"token\": \"8\", \"logprob\": -0.105 }\n```\n\nWithout it you only see `8` and have no idea how sure the model was.\n\n```\n{ \"logprobs\": true, \"top_logprobs\": 5 }\n```\n\nAt each generated position, in addition to the generated token,\n\nreturn the **5 most likely candidate tokens**.\n\nThe response looks like this:\n\n```\n{\n  \"token\": \"8\",\n  \"logprob\": -0.105,\n  \"top_logprobs\": [\n    { \"token\": \"8\", \"logprob\": -0.105 },\n    { \"token\": \"7\", \"logprob\": -2.526 },\n    { \"token\": \"9\", \"logprob\": -3.912 },\n    { \"token\": \"6\", \"logprob\": -4.605 },\n    { \"token\": \"1\", \"logprob\": -5.298 }\n  ]\n}\n```\n\n`8@0.90, 7@0.08, 9@0.02` comes from.\nOne naming quirk:\n\nIn the code, `entry.top_logprobs` refers to the response array.\n\n`ln(p) ≤ 0`).\nCloser to 0 means more likely.\nSection 2 gave the result. This section fills in the derivation so you can\n\ncheck it yourself.\n\nTaking the argmax throws away both at the same instant.\n\n| Candidate | Distribution (score @ probability) | Expected | Confidence | argmax | \n|---|---|---|---|---|\n| A | [8@0.90](mailto:8@0.90) ,[7@0.08](mailto:7@0.08) ,[9@0.02](mailto:9@0.02) | **7.94** | 0.90 | 8 | \n| B | [8@0.50](mailto:8@0.50) ,[7@0.40](mailto:7@0.40) ,[9@0.10](mailto:9@0.10) | **7.70** | 0.50 | 8 | \n| C | [9@0.50](mailto:9@0.50) ,[6@0.40](mailto:6@0.40) ,[7@0.10](mailto:7@0.10) | **7.60** | 0.50 | 9 | \n\nThere are only two formulas:\n\n```\nexpected score = Σ (probability_i × score_i)\nconfidence     = max probability / total probability\n```\n\n(In the example the probabilities already sum to 1, so the denominator is 1.\n\nIn real calls the top-k probabilities usually sum to less than 1,\n\nso you must normalize — Section 7 covers this.)\n\n**Method A: integer only (argmax)**\n\n```\nA = 8, B = 8, C = 9   → A and B are indistinguishable; C ranks first\n```\n\n**Method B: expected score**\n\n```\nA 7.94 > B 7.70 > C 7.60   → select A\n```\n\n**Method C: expected score + confidence**\n\n```\nA (7.94, 0.90)   → better on average and more certain\n```\n\nThe three methods disagree (A selects C; B and C select A).\n\nThis is not a contrived coincidence:\n\nThe problem is not \"whether the judge is accurate\".\n\nIt is that **the information argmax discards is exactly the information you need to judge accuracy.**\n\nThe verifier fixes \"scores are not comparable\".\n\nIt does not fix \"the judge itself is biased\".\n\nEven with a distribution, you cannot rely on a single judgment.\n\nThese are not occasional mistakes. They are repeatable:\n\nThere is also a statistical trap: **raw agreement is inflated**.\n\nIt includes \"random chance agreement\"; subtract that and the real\n\nconsistency drops noticeably. Do not treat \"85% agreement\" as a conclusion —\n\nlook at kappa first.\n\n`lc-evaluation/src/judge_reliability.rs` provides:\n\n**Order swap**\n\n``` php\npub fn order_swap_consistent(original: Verdict, swapped: Verdict) -> Verdict {\n    match (original, swapped) {\n        (Verdict::AWins, Verdict::BWins) => Verdict::AWins,\n        (Verdict::BWins, Verdict::AWins) => Verdict::BWins,\n        _ => Verdict::Tie,\n    }\n}\n```\n\nIn the swapped round, \"B wins\" means the original A won.\n\n**A candidate wins only if both rounds point to the same original answer.**\n\n**Cohen's kappa**\n\n``` js\nlet kappa = cohen_kappa(&judge_a, &judge_b);\n```\n\nHow much agreement remains after subtracting chance agreement.\n\nDegenerate samples where everything is positive or everything is negative\n\nreturn `None` instead of pretending to have a conclusion.\n\n**Same-family detection**\n\n```\nassert!(same_family(\"gpt-5\", \"gpt-4o\"));\nassert!(!same_family(\"gpt-5\", \"claude-3\"));\n```\n\nEither reject same-family judging or mark it explicitly in the report.\n\n**Summary report**\n\n```\npub struct ReliabilityReport {\n    pub n: usize,\n    pub agreement: f64,\n    pub kappa: Option<f64>,\n    pub ties: usize,\n}\n```\n\nYou need to know how many pairs became ties because the order disagreed.\n\n**A high tie rate means the judge has no discriminative power; no matter how good the scores look, do not use them.**\n\nDo not let the model freely emit `\"8/10\"`, `\"8 points\"`, or `\"eight\"`.\n\nMake it emit exactly one character from 0 to 9.\n\nWhy this matters:\n\n`1` and `0` by some providers;\n\n``` js\nlet config = OpenAIConfig::from_env_result()?.with_top_logprobs(5);\nlet llm = OpenAIChat::new(config);\n```\n\nThe request body gains two fields:\n\n```\n{ \"logprobs\": true, \"top_logprobs\": 5 }\n```\n\n(The field semantics and response shape are covered in Section 4;\n\nhere we only care about how to turn them on.)\n\nThey are off by default so that callers who do not need this feature\n\nkeep a byte-identical request body.\n\nThe core function lives in `lc-evaluation/src/verifier.rs`:\n\n```\npub fn score_from_logprobs(\n    token_logprobs: &[TokenLogprob],\n    max_score: u32,\n) -> Option<VerificationScore> {\n    for entry in token_logprobs {\n        let mut mass = vec![0.0f64; (max_score + 1) as usize];\n\n        // The generated token is often included in the candidate list;\n        // adding it separately would double-count its probability.\n        let mut add = |token: &str, logprob: f64| {\n            if let Some(score) = parse_score_token(token, max_score) {\n                mass[score as usize] += logprob.exp();\n            }\n        };\n        let chosen_listed = entry.top_logprobs.iter().any(|alt| alt.token == entry.token);\n        if !chosen_listed {\n            add(&entry.token, entry.logprob);\n        }\n        for alt in &entry.top_logprobs {\n            add(&alt.token, alt.logprob);\n        }\n\n        // The top-k probabilities do not sum to 1; normalize.\n        let total: f64 = mass.iter().sum();\n        if total <= 0.0 || !total.is_finite() {\n            continue;\n        }\n\n        let expected: f64 = mass.iter().enumerate()\n            .map(|(score, p)| score as f64 * (p / total))\n            .sum();\n        let confidence = mass.iter().cloned().fold(0.0f64, f64::max) / total;\n\n        return Some(VerificationScore {\n            score: expected,\n            max_score: f64::from(max_score),\n            confidence: confidence.clamp(0.0, 1.0),\n            source: ScoreSource::Logprob,\n        });\n    }\n    None\n}\n```\n\nThere are three traps in this snippet. Get any one wrong and the score\n\nwill lie to you at some boundary:\n\n`The`, `score`, and `42` must not\ncontribute score mass. `parse_score_token` trims and rejects out-of-range values.\nKeep two outputs separate:\n\n`confidence` answers \"how concentrated this judgment is\".\n**A score without a confidence is just the distribution flattened again.**\n\nThree common integration points, ordered by cost:\n\n| Hook point | What it does | Recommendation | \n|---|---|---|\n| Before returning the final answer | N candidates, return one | On by default | \n| Before a tool call | Candidate arguments/actions, execute one | Only for high-risk tools | \n| Mid multi-step task | Prune branches, keep the best k | Only at key branch points | \n\nThe corresponding framework call (`lc-evaluation`):\n\n``` js\nlet verifier = LogprobVerifier::new(llm)\n    .with_max_score(9)\n    .with_top_logprobs(5);\n\nlet result = best_of_n(&verifier, task, &candidates).await?;\n\nfor (i, score) in result.scores.iter().enumerate() {\n    println!(\n        \"{i}: score={:.2} confidence={:.2} source={:?}\",\n        score.score, score.confidence, score.source\n    );\n}\n```\n\nDo **not** run best-of-N at every step.\n\nAn agent with M steps and N candidates per step costs\n\nM × N generations plus M × N judgments.\n\nTurn it on only where \"choosing wrong is expensive\".\n\nCost-control rules:\n\n| Scenario | Preferred | \n|---|---|\n| Tests / compilation / static checks available | **Execution verification** (hardest signal) | \n| Logprobs available and you need selection | Logprob verifier + best-of-N | \n| No logprobs | Integer judge + order swap + kappa (fallback) | \n| Very low budget | Single judge + manual spot checks | \n| High-risk decision | Execution verification + verifier + human review | \n\nbest-of-N has a subtle trap: `Iterator::max_by` returns the **last**\n\nelement on a tie.\n\n```\n// Wrong: returns the last one on ties\nlet (best_index, _) = scores\n    .iter()\n    .enumerate()\n    .max_by(|(_, a), (_, b)| a.score.partial_cmp(&b.score).unwrap())\n    .unwrap();\n```\n\nThe same input can select different candidates on different runs.\n\nIn regression tests this appears as an intermittent failure that is very\n\nhard to trace.\n\nThe correct version uses a strict greater-than and keeps the earliest:\n\n``` js\nlet mut best_index = 0usize;\nfor (i, score) in scores.iter().enumerate() {\n    if score.score > scores[best_index].score {\n        best_index = i;\n    }\n}\n```\n\n**Failure mode**: unstable best-of-N results; evaluation reports that\n\ncannot be reproduced for the same candidate set.\n\nNot every provider exposes token log probabilities\n\n(for example, Anthropic's public API).\n\nThe verifier needs two implementations:\n\n```\npub struct LogprobVerifier<M> { ... }   // distribution available; use expectation\npub struct JudgeVerifier<M> { ... }     // no distribution; parse a single integer\n```\n\n`JudgeVerifier` fixes the confidence at 0.25:\n\n```\nOk(VerificationScore {\n    score: f64::from(score),\n    max_score: f64::from(self.max_score),\n    confidence: 0.25,\n    source: ScoreSource::Judge,\n})\n```\n\nThis is not pretending it is as good as the logprob verifier.\n\nIt **explicitly marks the result as low-confidence** so downstream\n\nconsumers can decide whether to trust it.\n\n**If execution results exist, do not use model scores.**\n\nThe verifier belongs where there is no objective signal:\n\nwriting, analysis, option selection, open-ended QA.\n\n| Symptom | Root cause | Fix | \n|---|---|---|\n| Scores are systematically high | Generated token double-counted | Deduplicate first | \n| best-of-N changes between runs | Tie resolved to the last element | Strict greater-than | \n| Score parsing fails | Multi-digit / free text | Allow only one token | \n| Scores exceed the scale | No normalization | Divide by total probability | \n| Same-family scores inflated | Judge and candidate share a family | Mark or reject | \n| A/B verdict flips | No order swap | Merge both orders | \n| Kappa is `None` | All-positive or all-negative sample | Change the sample; do not fake a conclusion | \n\nIf you are plugging model scores into an agent's automated decisions,\n\ndo these eight things in order:\n\n`JudgeVerifier` with confidence 0.25\nand mark it explicitly.\n**A judge model gives you an opinion. A verifier gives you a comparable measurement.**\n\nThe difference is one sentence:\n\nIf you do only one thing, do this:\n\nStop taking just the integer. Take the distribution, and compute\n\nthe expected score and its confidence.\n\n*Next: \"From the Most Likely Token to the Expected Value: Building a Verifier\"*\n\n*— the implementation retrospective: the two small bugs that taught us the hard lessons.*", "url": "https://wpnews.pro/news/is-your-agent-s-llm-judge-unreliable-use-logprobs-to-make-scores-comparable", "canonical_source": "https://dev.to/lili_at_32369546c2947bd24/is-your-agents-llm-judge-unreliable-use-logprobs-to-make-scores-comparable-1j6m", "published_at": "2026-11-16 03:44:45+00:00", "updated_at": "2026-11-16 03:50:57.466653+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "developer-tools"], "entities": ["langchainrust", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/is-your-agent-s-llm-judge-unreliable-use-logprobs-to-make-scores-comparable", "markdown": "https://wpnews.pro/news/is-your-agent-s-llm-judge-unreliable-use-logprobs-to-make-scores-comparable.md", "text": "https://wpnews.pro/news/is-your-agent-s-llm-judge-unreliable-use-logprobs-to-make-scores-comparable.txt", "jsonld": "https://wpnews.pro/news/is-your-agent-s-llm-judge-unreliable-use-logprobs-to-make-scores-comparable.jsonld"}}