# Is Your Agent's LLM Judge Unreliable? Use Logprobs to Make Scores Comparable

> Source: <https://dev.to/lili_at_32369546c2947bd24/is-your-agents-llm-judge-unreliable-use-logprobs-to-make-scores-comparable-1j6m>
> Published: 2026-11-16 03:44:45+00:00

One sentence: **the model gives you an integer; a decision needs a measurement.**

This article gives the fix first (three steps), then explains why it works,

and ends with the engineering constraints you cannot skip.

The code below comes from the real implementation in langchainrust 0.30.0:

`lc-core/src/language_models/chat.rs`,

`lc-evaluation/src/verifier.rs`,

`lc-evaluation/src/judge_reliability.rs`.

This is a common agent scenario:

**the agent produces several candidates at one step (answers, plans, or tool calls), then calls another LLM as a judge and picks one to continue.**

In agent systems this is called **best-of-N / verifier-guided selection**.

It shows up in:

The common factor: **the judge is itself an LLM.**

So the judge returns:

```
Candidate A: 8
Candidate B: 8
```

**Which one do you pick?**

You can't. The integer 8 only tells you "it chose 8".

It does not tell you "how confident it was".

The distribution behind that same call looks like this:

```
Candidate A: 8 @ 0.90, 7 @ 0.08, 9 @ 0.02
Candidate B: 8 @ 0.50, 7 @ 0.40, 9 @ 0.10
```

A is very sure about 8. B only picked 8 by a 0.50-to-0.40 margin over 7 —

it had a 40% chance of actually wanting to say 7.

**Two 8s are not the same 8.**

You do not need a different model, and you do not need to retrain anything.

You only need to take back the information the model already computed

but does not return by default:

`logprobs` and `top_logprobs`.
Same two candidates, two selection methods:

| Candidate | Judge output | Integer only | Verifier (expected / confidence) | 
|---|---|---|---|
| A | 8 | 8 | **7.94 / 0.90** | 
| B | 8 | 8 | **7.70 / 0.50** | 

That is the whole fix. In code it is one extra line:

``` js
let config = OpenAIConfig::from_env_result()?.with_top_logprobs(5);
```

The rest of the article unpacks this:

The first reaction is usually: isn't the judge "just directly giving a score"?

Why would there be a probability distribution?

Because the model has no "scoring module" and no regression head that emits

a continuous value. It does exactly what it does when writing a poem or code:

**predict the next token.**

The prompt looks like this:

```
Score the answer below on a 0-9 scale.
Output exactly one digit.
Answer: ...
Score:
```

The model's job is to complete the token after `Score:`.

It computes one softmax over the whole vocabulary; `8` has the highest

probability, so it emits `8`.

So:

```
What you see:      8
What the model did: computed a probability over the vocabulary,
                   8 ranked first, emitted 8
```

**"Direct scoring" is an analogy to human behavior, not the model's operation.**

Because of that, there is always a distribution behind the chosen token.

The default API response simply does not return it.

| Layer | Content | Who provides it | 
|---|---|---|
| Choice | It emitted 8 | The model | 
| Confidence | Whether 8 had probability 0.9 or 0.5 | The model ( `logprobs` ) | 
| True quality | How good an 8 actually is | **The model does not measure this** | 

The third layer is the easiest to misread:

`0.90` does not mean "this answer has a 90% chance of being an 8-quality answer".

It means "the model is very sure the next token is 8".

The model gives you an **opinion and the strength of that opinion**,

not a calibrated quality measurement.

This is not a thermometer. It is a multiple-choice answer.

You ask a person, "Rate this answer from 1 to 10." They say "8".

You see the answer; you do not see whether they said it instantly

or hesitated between 7 and 8.

The model is different: its hesitation is digitized and readable.

`0.90` is an instant answer; `0.50` is a hesitant one.

Step 2 says "take the distribution back". This section explains exactly

what you are taking, what the fields are called, and what the response looks like.

The model does not generate characters or words. It generates **tokens**.

A token can be:

```
8
10
points
(a space)
the
```

At each step the model does one thing: **given the text so far, predict the next token.** It computes a probability over the entire

```
next token "8"      probability 0.90
next token "7"      probability 0.08
next token "9"      probability 0.02
next token "points" probability 0.001
...
```

`logprob` is
If the probability is `p`, the `logprob` is `ln(p)`.

| Probability p | `logprob = ln(p)` | 
|---|---|
| 0.90 | **-0.105** | 
| 0.50 | **-0.693** | 
| 0.08 | **-2.526** | 
| 0.02 | **-3.912** | 
| 0.001 | **-6.908** | 

Why logarithms instead of probabilities?

Converting back is one step:

```
probability = exp(logprob)
```

`exp(-0.105) ≈ 0.90`.

In the request:

```
{ "logprobs": true }
```

This means:

Return the log probability of **the token you actually generated**.

If the model generated `8`, the response includes:

```
{ "token": "8", "logprob": -0.105 }
```

Without it you only see `8` and have no idea how sure the model was.

```
{ "logprobs": true, "top_logprobs": 5 }
```

At each generated position, in addition to the generated token,

return the **5 most likely candidate tokens**.

The response looks like this:

```
{
  "token": "8",
  "logprob": -0.105,
  "top_logprobs": [
    { "token": "8", "logprob": -0.105 },
    { "token": "7", "logprob": -2.526 },
    { "token": "9", "logprob": -3.912 },
    { "token": "6", "logprob": -4.605 },
    { "token": "1", "logprob": -5.298 }
  ]
}
```

`8@0.90, 7@0.08, 9@0.02` comes from.
One naming quirk:

In the code, `entry.top_logprobs` refers to the response array.

`ln(p) ≤ 0`).
Closer to 0 means more likely.
Section 2 gave the result. This section fills in the derivation so you can

check it yourself.

Taking the argmax throws away both at the same instant.

| Candidate | Distribution (score @ probability) | Expected | Confidence | argmax | 
|---|---|---|---|---|
| A | [8@0.90](mailto:8@0.90) ,[7@0.08](mailto:7@0.08) ,[9@0.02](mailto:9@0.02) | **7.94** | 0.90 | 8 | 
| B | [8@0.50](mailto:8@0.50) ,[7@0.40](mailto:7@0.40) ,[9@0.10](mailto:9@0.10) | **7.70** | 0.50 | 8 | 
| C | [9@0.50](mailto:9@0.50) ,[6@0.40](mailto:6@0.40) ,[7@0.10](mailto:7@0.10) | **7.60** | 0.50 | 9 | 

There are only two formulas:

```
expected score = Σ (probability_i × score_i)
confidence     = max probability / total probability
```

(In the example the probabilities already sum to 1, so the denominator is 1.

In real calls the top-k probabilities usually sum to less than 1,

so you must normalize — Section 7 covers this.)

**Method A: integer only (argmax)**

```
A = 8, B = 8, C = 9   → A and B are indistinguishable; C ranks first
```

**Method B: expected score**

```
A 7.94 > B 7.70 > C 7.60   → select A
```

**Method C: expected score + confidence**

```
A (7.94, 0.90)   → better on average and more certain
```

The three methods disagree (A selects C; B and C select A).

This is not a contrived coincidence:

The problem is not "whether the judge is accurate".

It is that **the information argmax discards is exactly the information you need to judge accuracy.**

The verifier fixes "scores are not comparable".

It does not fix "the judge itself is biased".

Even with a distribution, you cannot rely on a single judgment.

These are not occasional mistakes. They are repeatable:

There is also a statistical trap: **raw agreement is inflated**.

It includes "random chance agreement"; subtract that and the real

consistency drops noticeably. Do not treat "85% agreement" as a conclusion —

look at kappa first.

`lc-evaluation/src/judge_reliability.rs` provides:

**Order swap**

``` php
pub fn order_swap_consistent(original: Verdict, swapped: Verdict) -> Verdict {
    match (original, swapped) {
        (Verdict::AWins, Verdict::BWins) => Verdict::AWins,
        (Verdict::BWins, Verdict::AWins) => Verdict::BWins,
        _ => Verdict::Tie,
    }
}
```

In the swapped round, "B wins" means the original A won.

**A candidate wins only if both rounds point to the same original answer.**

**Cohen's kappa**

``` js
let kappa = cohen_kappa(&judge_a, &judge_b);
```

How much agreement remains after subtracting chance agreement.

Degenerate samples where everything is positive or everything is negative

return `None` instead of pretending to have a conclusion.

**Same-family detection**

```
assert!(same_family("gpt-5", "gpt-4o"));
assert!(!same_family("gpt-5", "claude-3"));
```

Either reject same-family judging or mark it explicitly in the report.

**Summary report**

```
pub struct ReliabilityReport {
    pub n: usize,
    pub agreement: f64,
    pub kappa: Option<f64>,
    pub ties: usize,
}
```

You need to know how many pairs became ties because the order disagreed.

**A high tie rate means the judge has no discriminative power; no matter how good the scores look, do not use them.**

Do not let the model freely emit `"8/10"`, `"8 points"`, or `"eight"`.

Make it emit exactly one character from 0 to 9.

Why this matters:

`1` and `0` by some providers;

``` js
let config = OpenAIConfig::from_env_result()?.with_top_logprobs(5);
let llm = OpenAIChat::new(config);
```

The request body gains two fields:

```
{ "logprobs": true, "top_logprobs": 5 }
```

(The field semantics and response shape are covered in Section 4;

here we only care about how to turn them on.)

They are off by default so that callers who do not need this feature

keep a byte-identical request body.

The core function lives in `lc-evaluation/src/verifier.rs`:

```
pub fn score_from_logprobs(
    token_logprobs: &[TokenLogprob],
    max_score: u32,
) -> Option<VerificationScore> {
    for entry in token_logprobs {
        let mut mass = vec![0.0f64; (max_score + 1) as usize];

        // The generated token is often included in the candidate list;
        // adding it separately would double-count its probability.
        let mut add = |token: &str, logprob: f64| {
            if let Some(score) = parse_score_token(token, max_score) {
                mass[score as usize] += logprob.exp();
            }
        };
        let chosen_listed = entry.top_logprobs.iter().any(|alt| alt.token == entry.token);
        if !chosen_listed {
            add(&entry.token, entry.logprob);
        }
        for alt in &entry.top_logprobs {
            add(&alt.token, alt.logprob);
        }

        // The top-k probabilities do not sum to 1; normalize.
        let total: f64 = mass.iter().sum();
        if total <= 0.0 || !total.is_finite() {
            continue;
        }

        let expected: f64 = mass.iter().enumerate()
            .map(|(score, p)| score as f64 * (p / total))
            .sum();
        let confidence = mass.iter().cloned().fold(0.0f64, f64::max) / total;

        return Some(VerificationScore {
            score: expected,
            max_score: f64::from(max_score),
            confidence: confidence.clamp(0.0, 1.0),
            source: ScoreSource::Logprob,
        });
    }
    None
}
```

There are three traps in this snippet. Get any one wrong and the score

will lie to you at some boundary:

`The`, `score`, and `42` must not
contribute score mass. `parse_score_token` trims and rejects out-of-range values.
Keep two outputs separate:

`confidence` answers "how concentrated this judgment is".
**A score without a confidence is just the distribution flattened again.**

Three common integration points, ordered by cost:

| Hook point | What it does | Recommendation | 
|---|---|---|
| Before returning the final answer | N candidates, return one | On by default | 
| Before a tool call | Candidate arguments/actions, execute one | Only for high-risk tools | 
| Mid multi-step task | Prune branches, keep the best k | Only at key branch points | 

The corresponding framework call (`lc-evaluation`):

``` js
let verifier = LogprobVerifier::new(llm)
    .with_max_score(9)
    .with_top_logprobs(5);

let result = best_of_n(&verifier, task, &candidates).await?;

for (i, score) in result.scores.iter().enumerate() {
    println!(
        "{i}: score={:.2} confidence={:.2} source={:?}",
        score.score, score.confidence, score.source
    );
}
```

Do **not** run best-of-N at every step.

An agent with M steps and N candidates per step costs

M × N generations plus M × N judgments.

Turn it on only where "choosing wrong is expensive".

Cost-control rules:

| Scenario | Preferred | 
|---|---|
| Tests / compilation / static checks available | **Execution verification** (hardest signal) | 
| Logprobs available and you need selection | Logprob verifier + best-of-N | 
| No logprobs | Integer judge + order swap + kappa (fallback) | 
| Very low budget | Single judge + manual spot checks | 
| High-risk decision | Execution verification + verifier + human review | 

best-of-N has a subtle trap: `Iterator::max_by` returns the **last**

element on a tie.

```
// Wrong: returns the last one on ties
let (best_index, _) = scores
    .iter()
    .enumerate()
    .max_by(|(_, a), (_, b)| a.score.partial_cmp(&b.score).unwrap())
    .unwrap();
```

The same input can select different candidates on different runs.

In regression tests this appears as an intermittent failure that is very

hard to trace.

The correct version uses a strict greater-than and keeps the earliest:

``` js
let mut best_index = 0usize;
for (i, score) in scores.iter().enumerate() {
    if score.score > scores[best_index].score {
        best_index = i;
    }
}
```

**Failure mode**: unstable best-of-N results; evaluation reports that

cannot be reproduced for the same candidate set.

Not every provider exposes token log probabilities

(for example, Anthropic's public API).

The verifier needs two implementations:

```
pub struct LogprobVerifier<M> { ... }   // distribution available; use expectation
pub struct JudgeVerifier<M> { ... }     // no distribution; parse a single integer
```

`JudgeVerifier` fixes the confidence at 0.25:

```
Ok(VerificationScore {
    score: f64::from(score),
    max_score: f64::from(self.max_score),
    confidence: 0.25,
    source: ScoreSource::Judge,
})
```

This is not pretending it is as good as the logprob verifier.

It **explicitly marks the result as low-confidence** so downstream

consumers can decide whether to trust it.

**If execution results exist, do not use model scores.**

The verifier belongs where there is no objective signal:

writing, analysis, option selection, open-ended QA.

| Symptom | Root cause | Fix | 
|---|---|---|
| Scores are systematically high | Generated token double-counted | Deduplicate first | 
| best-of-N changes between runs | Tie resolved to the last element | Strict greater-than | 
| Score parsing fails | Multi-digit / free text | Allow only one token | 
| Scores exceed the scale | No normalization | Divide by total probability | 
| Same-family scores inflated | Judge and candidate share a family | Mark or reject | 
| A/B verdict flips | No order swap | Merge both orders | 
| Kappa is `None` | All-positive or all-negative sample | Change the sample; do not fake a conclusion | 

If you are plugging model scores into an agent's automated decisions,

do these eight things in order:

`JudgeVerifier` with confidence 0.25
and mark it explicitly.
**A judge model gives you an opinion. A verifier gives you a comparable measurement.**

The difference is one sentence:

If you do only one thing, do this:

Stop taking just the integer. Take the distribution, and compute

the expected score and its confidence.

*Next: "From the Most Likely Token to the Expected Value: Building a Verifier"*

*— the implementation retrospective: the two small bugs that taught us the hard lessons.*
