{"slug": "tracing-knowledge-and-reasoning-in-llm", "title": "Tracing Knowledge and Reasoning in LLM", "summary": "A new arXiv paper (2611.17461v1) proposes a task-centric framework that uses attention- and activation-based signals plus representation-based measures to characterize how large language models process benchmarks internally, rather than reporting only dataset- or task-level scores. The authors evaluated 16 LLMs across 24 benchmarks and ran targeted internal-state analyses on representative models and tasks, finding systematic differences in processing between knowledge-centric and reasoning-centric tasks. The code is open sourced at github.com/extraordinarylab/trace-kr, and the authors say the approach could support more targeted benchmark selection and more informed model choice.", "body_md": "arXiv:2611.17461v1 Announce Type: new \nAbstract: Current evaluations of Large Language Models (LLMs) report performance at the dataset or task level, revealing what a model achieves but offering little insight into the internal processing associated with its performance. We propose a task-centric framework for characterising the model-internal processing elicited by a benchmark. It combines attention- and activation-based signals with representation-based measures to characterise benchmarks spanning a continuum from retrieval-dominant to reasoning-dominant and to surface failure modes that task-level scores conceal. We evaluate 16 LLMs across 24 diverse benchmarks and conduct targeted internal-state analyses on representative models and tasks, observing systematic differences in processing between knowledge-centric and reasoning-centric tasks. We shift evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, potentially supporting more targeted benchmark selection and more informed model choice. We open source our code at https://github.com/extraordinarylab/trace-kr.", "url": "https://wpnews.pro/news/tracing-knowledge-and-reasoning-in-llm", "canonical_source": "https://www.machinebrief.com/news/tracing-knowledge-and-reasoning-in-llm-ps1g", "published_at": "2026-11-16 05:00:00+00:00", "updated_at": "2026-11-16 05:24:33.714746+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["arXiv", "extraordinarylab", "trace-kr"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tracing-knowledge-and-reasoning-in-llm", "markdown": "https://wpnews.pro/news/tracing-knowledge-and-reasoning-in-llm.md", "text": "https://wpnews.pro/news/tracing-knowledge-and-reasoning-in-llm.txt", "jsonld": "https://wpnews.pro/news/tracing-knowledge-and-reasoning-in-llm.jsonld"}}