# PIN v5 - Substantial upgrade hence the new thread

> Source: <https://discuss.huggingface.co/t/pin-v5-substantial-upgrade-hence-the-new-thread/178745#post_3>
> Published: 2026-08-17 18:29:12+00:00

Hi. Thanks to the dataset/code release, I was able to turn this into an experiment:

I tried a small control around the **“knowing you are unsure” vs “knowing you are out of your depth”** result in §4.6, using the public [ confidence_member.py](https://huggingface.co/opticalfibre/PIN/blob/main/confidence_member.py) as the starting point.

The short version is that the result held up better than I expected.

I first reproduced the original-style setup closely enough to use it as a gate: with the probe fitted on the same unseen class identities that appear at test time, I got a scaled-confidence unseen/seen AUROC of about **0.839**, close to the reported **0.8286**. I then removed one unseen class identity entirely from the 200-example probe fit and tested on that class.

The feature-level jurisdiction score still separated the held-out unseen class from the familiar classes quite strongly:

| Probe sees as “unseen” | Completely held-out unseen class | Direct jurisdiction AUROC |
|---|---|---|
| 8 + 9 | 7 | 0.891 |
| 7 + 9 | 8 | 0.867 |
| 7 + 8 | 9 | 0.961 |

So, at least inside this Fashion-MNIST setup, the 200-example probe does **not** seem to be working merely because all three unseen class identities were represented in its supervision.

The more interesting result came from asking a different question, though.

The score that was best at answering **“is this unfamiliar?”** was not necessarily the score that was best at answering **“which predictions should I actually keep?”**

That distinction produced a pattern that looks surprisingly close to the existing **selective classification with OOD data (SCOD)** literature.

For the three class-held-out conditions:

| Held-out | Score | Unseen AUROC ↑ | AURC ↓ |
|---|---|---|---|
| 7 | base confidence | 0.181 | 0.246 |
| 7 | direct jurisdiction | 0.891 |
0.156 |
| 7 | scaled confidence | 0.462 | 0.162 |
| 8 | base confidence | 0.806 | 0.072 |
| 8 | direct jurisdiction | 0.867 |
0.175 |
| 8 | scaled confidence | 0.877 |
0.074 |
| 9 | base confidence | 0.276 | 0.202 |
| 9 | direct jurisdiction | 0.961 |
0.147 |
| 9 | scaled confidence | 0.711 | 0.095 |

So I would now separate three things fairly sharply:

The direct jurisdiction score was the most consistent detector of the held-out unseen identity, but the scaled score was often the better *selective-prediction* score. In other words, the scaling step does not look like a useless or merely lossy conversion of the jurisdiction probe. It seems to preserve useful information from the original classifier while injecting information about unfamiliarity.

That was the part I found most interesting.

Exact control I ranPutting all of this together, the strongest result I would carry forward from my little control is probably:

```
1. The feature-level jurisdiction signal survives an unseen-class-identity holdout
   in this Fashion-MNIST regime.

2. That signal and ordinary prediction confidence are doing different jobs.

3. A score that is excellent at unfamiliarity detection is not automatically the
   best score for abstention.

4. The existing PIN scaling appears to preserve useful base-confidence information
   while injecting jurisdiction information, and in these runs that gave the best
   overall risk–coverage behavior.

5. This maps quite naturally onto the SCOD picture:
   classification-risk information and OOD/jurisdiction information are distinct
   inputs to a useful selector.
```

So if I were choosing the interpretation by use case rather than trying to collapse everything into one “confidence” number:

``` php
If the question is:
    “Is this outside the learned jurisdiction?”
        -> the feature-level jurisdiction score is the clean object to inspect.

If the question is:
    “Which predictions should I actually keep or abstain on?”
        -> ordinary prediction confidence still matters;
           risk–coverage is more informative than OOD AUROC alone.

If the question is:
    “Does this generalize to arbitrary unknowns?”
        -> this experiment does not establish that;
           it establishes class-held-out transfer inside a small semantic regime.

If the question is:
    “What would be the next boundary if this branch is worth extending?”
        -> changing semantic distance / near-vs-far unfamiliarity while keeping
           the frozen base fixed looks more informative than simply adding more
           probe draws.
```

That last branch does not seem urgent to me. The current result already says something useful: the class-held-out control strengthened the jurisdiction result, and the selective-prediction control exposed a clean reason **not** to replace the base confidence with the direct jurisdiction score.

The code release made that distinction much easier to see.