cd /news/large-language-models/tracing-knowledge-and-reasoning-in-l… · home › topics › large-language-models › article
[ARTICLE · art-170331] src=machinebrief.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Tracing Knowledge and Reasoning in LLM

A new arXiv paper (2611.17461v1) proposes a task-centric framework that uses attention- and activation-based signals plus representation-based measures to characterize how large language models process benchmarks internally, rather than reporting only dataset- or task-level scores. The authors evaluated 16 LLMs across 24 benchmarks and ran targeted internal-state analyses on representative models and tasks, finding systematic differences in processing between knowledge-centric and reasoning-centric tasks. The code is open sourced at github.com/extraordinarylab/trace-kr, and the authors say the approach could support more targeted benchmark selection and more informed model choice.

by read1 min views1 publishedNov 16, 2026

arXiv:2611.17461v1 Announce Type: new Abstract: Current evaluations of Large Language Models (LLMs) report performance at the dataset or task level, revealing what a model achieves but offering little insight into the internal processing associated with its performance. We propose a task-centric framework for characterising the model-internal processing elicited by a benchmark. It combines attention- and activation-based signals with representation-based measures to characterise benchmarks spanning a continuum from retrieval-dominant to reasoning-dominant and to surface failure modes that task-level scores conceal. We evaluate 16 LLMs across 24 diverse benchmarks and conduct targeted internal-state analyses on representative models and tasks, observing systematic differences in processing between knowledge-centric and reasoning-centric tasks. We shift evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, potentially supporting more targeted benchmark selection and more informed model choice. We open source our code at https://github.com/extraordinarylab/trace-kr.

── more in #large-language-models 4 stories · sorted by recency
── more on @arxiv 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/tracing-knowledge-an…] indexed:0 read:1min 2026-11-16 · —