Distinguishing among reasoning levels of GPT 5.5 Pro via a grammar lens benchmark

A formal grammar-lens benchmark derived from computation theory found that GPT-5.5 and its Pro variants accepted progressively more novel test data as test-time reasoning effort rose, with acceptance rates of 16%, 40%, 43% and 52% for GPT-5.5, GPT-5.5-Pro-Medium, -High and -XHigh respectively, according to the arXiv paper 2611.00555v1. The authors report the benchmark can discern fine-grained differences among models' added test-time reasoning levels, and that as test-time compute effort increases, model responses diverge from human responses to the same data. The paper suggests this growing disagreement may be broadly characteristic of more capable language models, raising the potential of a general capability-versus-alignment conflict.

arXiv:2611.00555v1 Announce Type: new Abstract: Does the level of test-time processing effort affect a language model's computational capability? A formal grammar-lens benchmark method, derived from the theory of computation, generates novel data at specified levels, each successively more computationally powerful than the levels below. Using this data, we tested GPT-5.5, and GPT-5.5-Pro-Medium, -High, and -XHigh, showing incrementally increased acceptance of test data: 16, 40, 43, and 52\%, respectively, illustrating the benchmark's ability to discern fine-grained differences among models' added test-time reasoning levels. Additionally, as test-time compute effort increases, model responses diverge from human responses to the same data. This increasing disagreement may turn out to be broadly characteristic of more capable language models, introducing the potential of a general capability-versus-alignment conflict.