What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs

Researchers have developed a method to detect and track concepts within large language models (LLMs) by creating datasets that delineate specific ideas, training linear probes to identify those concepts across model layers, and demonstrating the probes' ability to follow concepts through extended contexts. The approach, tested on four concepts across three LLMs, aims to provide a low-cost, scalable way to monitor what models are "thinking" during normal operation. This capability could enable broader oversight of LLM decision-making as the technique is expanded to many more concepts.

arXiv:2605.28823v1 Announce Type: new Abstract: As the influence of LLMs expands, it is imperative to gain insight into their decisions. One way to do that is to develop probes that detect the presence or absence of a broad set of concepts within the embeddings computed in an LLM - which is what we might say a model is "thinking" about. Such probes should be low-cost and easily applicable to any LLM, so that monitoring for many concepts is possible during normal operation. In this paper, we take the first steps towards developing the capability of creating many such probes by defining and executing examples of the key tasks needed: first, the careful delineation of a concept through the creation of a dataset with the concept both present and then absent. Then, the training and testing of a set of linear probes to detect the concept on any layer of an LLM, including an exploration of the complexity of the probe needed. Finally, we show that such probes can track concepts across larger contexts. This is done with four separate concepts and three different LLMs. When this process is scaled to many more concepts, it will create the ability to easily monitor new models.