Tracing Knowledge and Reasoning in LLM A new arXiv paper (2611.17461v1) proposes a task-centric framework that uses attention- and activation-based signals plus representation-based measures to characterize how large language models process benchmarks internally, rather than reporting only dataset- or task-level scores. The authors evaluated 16 LLMs across 24 benchmarks and ran targeted internal-state analyses on representative models and tasks, finding systematic differences in processing between knowledge-centric and reasoning-centric tasks. The code is open sourced at github.com/extraordinarylab/trace-kr, and the authors say the approach could support more targeted benchmark selection and more informed model choice. arXiv:2611.17461v1 Announce Type: new Abstract: Current evaluations of Large Language Models LLMs report performance at the dataset or task level, revealing what a model achieves but offering little insight into the internal processing associated with its performance. We propose a task-centric framework for characterising the model-internal processing elicited by a benchmark. It combines attention- and activation-based signals with representation-based measures to characterise benchmarks spanning a continuum from retrieval-dominant to reasoning-dominant and to surface failure modes that task-level scores conceal. We evaluate 16 LLMs across 24 diverse benchmarks and conduct targeted internal-state analyses on representative models and tasks, observing systematic differences in processing between knowledge-centric and reasoning-centric tasks. We shift evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, potentially supporting more targeted benchmark selection and more informed model choice. We open source our code at https://github.com/extraordinarylab/trace-kr.