PIN v5 - Substantial upgrade hence the new thread

A researcher's independent control experiment on the PIN v5 dataset found that the feature-level jurisdiction signal survives a held-out unseen class identity in Fashion-MNIST, with direct jurisdiction AUROC scores of 0.891, 0.867, and 0.961 for held-out classes 7, 8, and 9, respectively. The study also revealed that the scaled confidence score often outperformed the direct jurisdiction score for selective prediction, aligning with the selective classification with OOD data (SCOD) literature, and that the PIN scaling preserves useful base-confidence information while injecting jurisdiction information.

Hi. Thanks to the dataset/code release, I was able to turn this into an experiment: I tried a small control around the “knowing you are unsure” vs “knowing you are out of your depth” result in §4.6, using the public confidence member.py https://huggingface.co/opticalfibre/PIN/blob/main/confidence member.py as the starting point. The short version is that the result held up better than I expected. I first reproduced the original-style setup closely enough to use it as a gate: with the probe fitted on the same unseen class identities that appear at test time, I got a scaled-confidence unseen/seen AUROC of about 0.839 , close to the reported 0.8286 . I then removed one unseen class identity entirely from the 200-example probe fit and tested on that class. The feature-level jurisdiction score still separated the held-out unseen class from the familiar classes quite strongly: | Probe sees as “unseen” | Completely held-out unseen class | Direct jurisdiction AUROC | |---|---|---| | 8 + 9 | 7 | 0.891 | | 7 + 9 | 8 | 0.867 | | 7 + 8 | 9 | 0.961 | So, at least inside this Fashion-MNIST setup, the 200-example probe does not seem to be working merely because all three unseen class identities were represented in its supervision. The more interesting result came from asking a different question, though. The score that was best at answering “is this unfamiliar?” was not necessarily the score that was best at answering “which predictions should I actually keep?” That distinction produced a pattern that looks surprisingly close to the existing selective classification with OOD data SCOD literature. For the three class-held-out conditions: | Held-out | Score | Unseen AUROC ↑ | AURC ↓ | |---|---|---|---| | 7 | base confidence | 0.181 | 0.246 | | 7 | direct jurisdiction | 0.891 | 0.156 | | 7 | scaled confidence | 0.462 | 0.162 | | 8 | base confidence | 0.806 | 0.072 | | 8 | direct jurisdiction | 0.867 | 0.175 | | 8 | scaled confidence | 0.877 | 0.074 | | 9 | base confidence | 0.276 | 0.202 | | 9 | direct jurisdiction | 0.961 | 0.147 | | 9 | scaled confidence | 0.711 | 0.095 | So I would now separate three things fairly sharply: The direct jurisdiction score was the most consistent detector of the held-out unseen identity, but the scaled score was often the better selective-prediction score. In other words, the scaling step does not look like a useless or merely lossy conversion of the jurisdiction probe. It seems to preserve useful information from the original classifier while injecting information about unfamiliarity. That was the part I found most interesting. Exact control I ranPutting all of this together, the strongest result I would carry forward from my little control is probably: 1. The feature-level jurisdiction signal survives an unseen-class-identity holdout in this Fashion-MNIST regime. 2. That signal and ordinary prediction confidence are doing different jobs. 3. A score that is excellent at unfamiliarity detection is not automatically the best score for abstention. 4. The existing PIN scaling appears to preserve useful base-confidence information while injecting jurisdiction information, and in these runs that gave the best overall risk–coverage behavior. 5. This maps quite naturally onto the SCOD picture: classification-risk information and OOD/jurisdiction information are distinct inputs to a useful selector. So if I were choosing the interpretation by use case rather than trying to collapse everything into one “confidence” number: php If the question is: “Is this outside the learned jurisdiction?” - the feature-level jurisdiction score is the clean object to inspect. If the question is: “Which predictions should I actually keep or abstain on?” - ordinary prediction confidence still matters; risk–coverage is more informative than OOD AUROC alone. If the question is: “Does this generalize to arbitrary unknowns?” - this experiment does not establish that; it establishes class-held-out transfer inside a small semantic regime. If the question is: “What would be the next boundary if this branch is worth extending?” - changing semantic distance / near-vs-far unfamiliarity while keeping the frozen base fixed looks more informative than simply adding more probe draws. That last branch does not seem urgent to me. The current result already says something useful: the class-held-out control strengthened the jurisdiction result, and the selective-prediction control exposed a clean reason not to replace the base confidence with the direct jurisdiction score. The code release made that distinction much easier to see.