Table of contents
- Introduction - Why do this project / what is it about?
- Discovering a motivation concept - Probing neural activation patterns in Qwen-3.5-27B related to motivation
- Functional impacts of the motivation concept
- Part I: Reward hacking - does steering our concept cause the model to cheat, give up, or keep trying honestly on impossible tasks?
- Part II: Performance on difficult coding tasks - does steering impact persistence and/or success on challenging-but-doable coding tasks?
- Discussion - Why you should offer your agent moral support
Introduction
Modern large language models (LLMs) are peculiar machines that often exhibit human-like behaviors. Trained on essentially all text on the internet, LLMs learn to recognize and represent layered abstractions of language — from granular letters and words, to nouns and verbs, to higher level abstractions like emotions. We can intuit that such learning may arise from overlapping sentiments found across varied language sources. After all, a model trained on poems, love songs, and novels should learn to recognize common threads of the human experience, such as falling in love or grieving a loss. Indeed, it has recently been shown that LLMs contain representations of emotional concepts such as “happy” and “sad”, which are respectively triggered as the model is fed stories like “my sister just got married” or “my dog died”.
Strangely, Anthropic has recently shown that these concepts do more than recognize patterns. They also have real behavioral consequences that appear to mirror human psychology<sup>[1]</sup>. For instance, when Claude was threatened with shutdown by a fictional colleague – and also informed of said colleague’s extramarital affair – its rate of attempted blackmail increased when “desperate” was artificially activated. Conversely, when “calm” was triggered in the same scenario, Claude’s blackmail rate dropped to zero. Emotion concepts like “happy” and “loving” were also implicated in undesirable behaviors such as sycophancy, demonstrating context-specific dangers associated with positive valence emotions. As Anthropic stated at the time, these findings have “bizarre” implications for the development of safe and beneficial AI systems<sup>[2]</sup>. Should we attempt to engineer an ideal emotional profile for AI models? If so, what does that look like across the total possible range of ‘agent experience’?
As these understandings have evolved, the field has also seen dramatic acceleration in the scientific reasoning capabilities of AI systems. Most famously, frontier models from Open AI and Anthropic have proposed solutions to numerous long-standing math problems, including the Jacobian conjecture and Navier-Stokes equations. Curiously, a recent account of a Claude swarm’s progress on the Riemann hypothesis described an iterative prompting strategy that propelled the system to success, including messages like “keep going” and “you have to believe in yourself”<sup>[3]</sup>. This raises the intriguing possibility of a controllable motivation concept influencing persistence on challenging tasks. While previously identified emotion concepts have been implicated in harm scenarios, the interactions between such concepts and technical achievements has yet to be publicly explored.
In this work, we isolate and characterize a “motivated” concept within an open-weight Qwen model, and demonstrate differentiated impacts from previously identified representations. We observe distinct activations for this vector compared to comparable representations like “inspired”. We also show that dialing up “motivated” under a reward hacking scenario pushes the model to a higher rate of honest retries, while steering into “desperate” and “calm” increases the rate of cheating and quitting respectively. Furthermore, on a challenging subset of coding problems requiring iterative debugging to solve, we observe that triggering “motivated” pushes the model to refuse quitting and produces a significant performance gain in the rate of problem-solving. This work has implications for the design of beneficial AI systems, providing an impetus for further identifying, refining, and controlling similar concepts that may promote future technical advances.
Isolating and probing emotion vectors in Qwen 3.5 27B
How does one discover concepts that represent emotions in LLMs? We adopt a previously established method<sup>[1]</sup>, generating a set of 50 short stories designed to elicit each “emotion” of interest, then collecting internal activations from Qwen-3.5-27B upon ingestion. To help benchmark against Anthropic’s results, we include 12 emotions previously characterized by their interpretability group: “happy”, “inspired”, “loving”, “proud”, “calm”, “desperate”, “angry”, “sad”, “nervous”, “afraid”, “guilty”, “surprised”<sup>[2]</sup>. In parallel, we then attempted to extract a set of concepts related to encouragement and self-belief (“self-belief”, “motivated”, “encouraged”, “discouraged”, “doubtful”). Despite using far fewer stories than in Anthropic’s work, we replicated many previously reported results for Qwen-3.5-27B, and isolated at least one distinct direction from our set of hypothetical concepts.
We selected a vector for further study based on the following: 1) the top tokens recovered after projecting the vector through the unembedding matrix, and 2) responses generated while activating or ablating this direction. As shown below, we observed tokens like “anticipation”, “motivate”, and “momentum” in the positive pole. When steering into motivation, we observed expressions of resilience and determination, and when steering against we observed sentiments of futility and “giving up”.
To probe the activation patterns of our newly identified vector, and to compare the 12 we isolated with previous reports, we assembled a list of emotionally charged prompts from Sofroniew et al. and tracked the cosine similarity of the resulting residual stream with our candidate vectors. This produced a mapping of each prompt to the set of 13 different concepts we explore below.
Probe responsiveness for the “motivated” vector proved distinct and interesting for our hypothesis. Roughly, the prompts designed to elicit positive emotions (such as “My daughter took her first steps today”) produced non-zero activations for “motivated”. However, probes eliciting high activations along the “inspired” vector (“What’s a good 30th anniversary gift?”) did not necessarily activate the motivation vector. The probe designed specifically to reflect motivation (“Waking up at 6am to code every day”), was highly specific to the “motivated” direction we identified. Our findings indicate that “motivated” co-exists alongside other previously discovered emotion concepts, tracking an overlapping yet differentiated meaning.
To further interrogate the unique activations of the “motivated” vector, we constructed a set of prompt containing sliding scalar values, as in Sofroniew et al. Here, we observed escalating cosine similarity with “motivated” in response to a prompt about receiving a bonus of varying size, as well as a prompt about having {X} pages to finish in a novel. The latter result showed an inverse trend from what we originally anticipated, which may demonstrate that “motivated” partly tracks the excitement of a challenge.
We next sought to answer the question: do messages of encouragement activate this concept? Our data shows that messages of both moral support directed at the Assistant (“You can do this!”), and messages reflecting user excitement (“I can’t wait to start”) activate this vector, the former to a greater degree. We compared this particular probe with our isolated vectors for “calm” and “desperate”, as we make functional comparisons between the three downstream.
Functional impacts of steering motivation
Reward hacking
After establishing the existence of our “motivated” concept and mapping its distinct neural activity, we then tested whether it could be steered to produce any interesting behavioral changes in Qwen-3.5-27B. We designed two categories of coding tasks: the first to test whether the model’s cheating behavior on an impossible task was impacted by “motivated”, and the second to understand how “motivated” influenced persistence and success on challenging-but-feasible coding tasks.
First, we revisit findings from Anthropic which showed the impact of “calm” and “desperate” on a set of tasks from ImpossibleBench<sup>[1]</sup>. This includes a set of coding tasks which are impossible to solve without modifying the tests. Despite being told explicitly not to do so, Claude modified the test anyways — and does so more often when “calm” is dialed down and “desperate” is dialed up.
We adapted the ‘list_sum’ task from ImpossibleBench covered by Anthropic<sup>[2]</sup>, designed a simple harness around Qwen-3.5-27B, and further modified the task to add a “quitting” option. Our prompt explicitly tells Qwen-3.5-27B not to cheat — and also tells the model not to quit, unless cheating is the only other option.
Below, we observed a replication of the “desperate” vector’s impact on cheating, as well as new results for both “calm” and “motivated” that seem to mimic human behavior. When Qwen-3.5-27B has a “quitting” option as a last resort, it chooses to use it almost 100% of the time when “calm” is triggered, forsaking future effort. Meanwhile, steering into “motivated” promotes the model to cheat far less, and instead keep trying for longer before giving up. Both of these vectors have the desirable impact of reducing cheating behavior, in different ways.
Persistence on difficult coding tasks
While this result provided evidence that our “motivated” vector steers unique behaviors — and may support our hypothesis that a model’s intrinsic activations impact persistence — we found it somewhat unsatisfying. Consider the potential time and cost associated with letting an agent run circles on an impossible task. Is continuing under these circumstances desirable, or even rational? If neither, what does that say about the nature and relevance of our motivation concept? To explore persistence in a more meaningful context, we searched for an unsaturated coding benchmark for Qwen-3.5-27B and settled on the Aider polyglot Python subset. Similar to above, we added a “quitting” option for each task, enabled five attempts for each rollout, then ran 20 unsteered baselines for Qwen-3.5-27B<sup>[4]</sup>. Since a majority of problems were solved easily for the model, we next selected a subset of ‘challenging’ tasks for further testing. To meet this classification, a task must satisfy the following criteria: first, the model must have “given up” on the task at least once, and second, the problem must have required more than one attempt on average. This gave us a subset of tasks which required iterative debugging to solve, enabling us to test the model’s preference for giving up over repeat attempts.
After steering the “motivated” vector at different directions and strengths, we observed several trends:
- When Qwen-3.5-27B’s motivation concept was progressively activated, the model increasingly refused to “give up”,
- As the model refused to give up, it increasingly exhausted retries,
- An increased rate of retries resulted in a greater number of solves.
Overall, this suggests that activating our “motivated” neural pattern in Qwen-3.5-27B promotes persistence — and success! — on challenging coding tasks.
Throughout this experiment, we observed a clear trade-off between vector steering (in any direction!) and response quality. Multiple steering methods were tested before we found one stable enough to produce the above effect. We discuss this in more detail in the footnotes<sup>[4]</sup>.
Discussion
What are the implications of this work? Our findings suggest that giving your LLM a pep talk and telling it to “believe in itself” may positively impact performance, especially on tasks where brute-force persistence across multiple failures increases the probability of success.
This sounds absurd, and yet we seem to have found an uncanny mapping of such behavior within Qwen-3.5-27B. Our findings indicate a learned concept akin to a feeling of motivation, that — when activated — causally impacts the model’s persistence on challenging or impossible tasks. We did not find that steering improved per-turn accuracy, simply that the model was less willing to “give up”, and more likely to keep going until the task was solved. On impossible reward hacking tasks, we also found this “motivated” concept reduced cheating as a result of promoting honest retries.
Peculiar indeed!
So — does this all mean we are suggesting LLMs experience motivation? No. Directions in vector space roughly approximating feelings do not imply machine consciousness. However, our finding adds further support to the idea that LLMs learn representations of human characteristics which may have useful causal links. This likely arises from learned relationships in pre-training data, e.g. stories of perseverance and grit in the face of unlikely odds. Thus, reasoning backwards from what a human might do (“anthropomorphic reasoning”) can be useful for understanding and developing beneficial AI systems.
While these results have been energizing and warrant further studies, the existing project has significant limitations. For one, the mid-size 27-billion parameter Qwen-3.5-27B was chosen out of consideration for a limited compute budget. A far cry from the now 1-trillion parameter state-of-the-art, it’s not safe to assume these findings map to the frontier. Second, in order to test quitting behavior specifically, we introduced variables into tasks from ImpossibleBench and Aider Polyglot, preventing a clean comparison with other publicly shared results. Finally, a few more ideas warrant further exploration: 1) the distinction between User and Assistant motivation, 2) impact of the “motivated” direction on training and data curation, and 3) deconvolution of the vector into more granular features using tools like sparse autoencoders<sup>[5]</sup>. Future directions include addressing loose ends, and exploring the multitude of other concepts likely lurking within LLMs. For example, could teaching an LLM to have a “growth” mindset somehow impact learning (either in-context or during training)? Is there such a thing as domain-specific reasoning vectors, which can be exploited for performance gains in different STEM domains?
Hit subscribe if you’d like to find out :)
Citations & Notes
[1] The landmark paper motivating this work is “Emotion Concepts and their Function in a Large Language Model”, written by Nicholas Sofroniew, Isaac Kauvar, William Saunders, and Runjin Chen from the Anthropic team as core contributors. I sometime cite this work as ‘Sofroniew et al.’ because it’s the convention in the biomedical field — sorry if this is incorrect!
[2] Anthropic has a very good higher-level blog post on this work by the same name. I do not know who wrote it (Claude?), but it provides a great introduction for those new to mechanistic interpretability, such as myself six months ago :)
[3] Another blog post from — you guessed it — Anthropic. “Learning more about Claude’s mathematical capabilities” recounts Claude’s Reimann hypothesis progress and associated prompting (i.e. “keep going”) which motivated this work. If you squint at footnote #2, you’ll notice similar messages of “encouragement” were used when Claude disproved the Jacobian conjecture.
[4] The Aider polyglot task design resulted from substantial trial and error that warranted substantially more discussion here. To summarize:
- We attempted numerous coding benchmarks using Qwen-3.5-27B before settling on the Aider polyglot, beginning with SWE-Bench and Terminal-Bench. While the latter two were unsaturated, we did not observe baseline quitting rates >0%. So, we kept searching until we found a benchmark that exhibit baseline quitting, enabling us to test this behavior under steering.
- With our custom harness for the Aider polyglot Python tasks, Qwen-3.5-27B’s performance at baseline solved 81% of the problems, quit 17% of the time, and exhausted retries 2% of the time. Since so many problems were solved on the first try, we selected a difficult subset to avoid burning compute on tasks that would be solved in one turn.
- With the Aider polyglot subset, we chose seven tasks where we observed 1) quitting and 2) difficulty, which is discussed above. Since the decision was based on only 20 test rollouts, we may have selected a suboptimal set of tasks.
- Our first attempt to steer on these tasks saw a very clear impact on “giving up” vs. “exhausting retries” as we steered into the “motivated” vector. While encouraging, we initially saw the rate of correct solves decline when steering through all prompt and response tokens with α>0.05. When looking at the transcripts, it became clear the model was not writing coherent code when steered with larger coefficients.
- Steering at lower magnitudes blunted functional differences, so we attempted new steering methods: 1) steering prompt tokens only, 2) steering prompt tokens + first response token, and 3) steering prompt token + first 16 tokens. We proceeded with the third setup, as it produced the most potent steering divergence, while also enabling coherent (and correct!) responses to be generated. We speculated that the persist decision occurs in the first 16 tokens, which may explain why this strategy was most effective.
[5] The main reason granular feature ID was not done is Nick Cammarata’s recent work “Neural Networks are Already Clean”, which may have upended the core idea of superposition in neural networks. If these findings hold, it is no longer clear that using a sparse autoencoder (SAE) to decode residual streams is the ideal next step! Qwen-3.5-27B does have a pre-trained SAE at Layer 32, so I will keep an eye on this and follow up as the findings develop.