Rank Portability Does Not Imply Feasibility Portability: Target-Specific Evaluation of Joint Hardware Constraints

A study of 10,000 GPT architectures across 13 HW-GPT-Bench devices found that cross-device rank correlation with an RTX3080 proxy ranges from 0.951 to 0.996 in latency SRCC, yet proxy-reuse violation risk under matched joint latency-energy constraints ranges from 33.3% to 100%. On NAS-Bench-201, cross-device rank correlations were moderate while target-comparable feasible-set overlap remained incomplete, and an exact finite-sample split-conformal analysis showed a one-sided 90% threshold requires at least nine calibration observations. The authors conclude that rank portability, feasibility portability, and target-specific decision support are distinct evaluation objects, and that cross-device evaluations should report which target environments actually support the claimed operating point.

arXiv:2609.22122v1 Announce Type: new Abstract: Cross-device hardware evaluation often assumes that if architecture rankings transfer across devices, a proxy device can support target-side model selection. We stress-test this assumption for joint latency-energy feasibility across two public architecture families. On NAS-Bench-201, cross-device rank correlations are moderate, while target-comparable feasible-set overlap remains incomplete. A faithful AdaProxy diagnostic substantially improves latency ranking, showing that the observed boundary failures are not simply due to weak adaptation. Exact finite-sample split-conformal analysis also exposes an evidence bottleneck: a finite one-sided 90% threshold requires at least nine calibration observations. We then replicate the phenomenon on 10,000 GPT architectures across 13 HW-GPT-Bench devices. Relative to an RTX3080 proxy, target latency SRCC ranges from 0.951 to 0.996, yet proxy-reuse violation risk ranges from 33.3% to 100% under matched joint constraints. These results show that rank portability, feasibility portability, and target-specific decision support are distinct evaluation objects. Cross-device evaluations should therefore report which target environments actually support the operating point being claimed.