PIN v5 - Substantial upgrade hence the new thread

Hi. Thanks to the dataset/code release, I was able to turn this into an experiment:

I tried a small control around the “knowing you are unsure” vs “knowing you are out of your depth” result in §4.6, using the public confidence_member.py as the starting point.

The short version is that the result held up better than I expected.

I first reproduced the original-style setup closely enough to use it as a gate: with the probe fitted on the same unseen class identities that appear at test time, I got a scaled-confidence unseen/seen AUROC of about 0.839, close to the reported 0.8286. I then removed one unseen class identity entirely from the 200-example probe fit and tested on that class.

The feature-level jurisdiction score still separated the held-out unseen class from the familiar classes quite strongly:

So, at least inside this Fashion-MNIST setup, the 200-example probe does not seem to be working merely because all three unseen class identities were represented in its supervision.

The more interesting result came from asking a different question, though.

The score that was best at answering “is this unfamiliar?” was not necessarily the score that was best at answering “which predictions should I actually keep?”

That distinction produced a pattern that looks surprisingly close to the existing selective classification with OOD data (SCOD) literature.

For the three class-held-out conditions:

So I would now separate three things fairly sharply:

The direct jurisdiction score was the most consistent detector of the held-out unseen identity, but the scaled score was often the better selective-prediction score. In other words, the scaling step does not look like a useless or merely lossy conversion of the jurisdiction probe. It seems to preserve useful information from the original classifier while injecting information about unfamiliarity.

That was the part I found most interesting.

Exact control I ranPutting all of this together, the strongest result I would carry forward from my little control is probably:

1. The feature-level jurisdiction signal survives an unseen-class-identity holdout
   in this Fashion-MNIST regime.

2. That signal and ordinary prediction confidence are doing different jobs.

3. A score that is excellent at unfamiliarity detection is not automatically the
   best score for abstention.

4. The existing PIN scaling appears to preserve useful base-confidence information
   while injecting jurisdiction information, and in these runs that gave the best
   overall risk–coverage behavior.

5. This maps quite naturally onto the SCOD picture:
   classification-risk information and OOD/jurisdiction information are distinct
   inputs to a useful selector.

So if I were choosing the interpretation by use case rather than trying to collapse everything into one “confidence” number:

If the question is:
    “Is this outside the learned jurisdiction?”
        -> the feature-level jurisdiction score is the clean object to inspect.

If the question is:
    “Which predictions should I actually keep or abstain on?”
        -> ordinary prediction confidence still matters;
           risk–coverage is more informative than OOD AUROC alone.

If the question is:
    “Does this generalize to arbitrary unknowns?”
        -> this experiment does not establish that;
           it establishes class-held-out transfer inside a small semantic regime.

If the question is:
    “What would be the next boundary if this branch is worth extending?”
        -> changing semantic distance / near-vs-far unfamiliarity while keeping
           the frozen base fixed looks more informative than simply adding more
           probe draws.

That last branch does not seem urgent to me. The current result already says something useful: the class-held-out control strengthened the jurisdiction result, and the selective-prediction control exposed a clean reason not to replace the base confidence with the direct jurisdiction score.

The code release made that distinction much easier to see.