I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT 2, but other research found that models only follow CoT formatting instructions ~2% of the time
. The prompts other people were trying didn’t match what I’d expect to work (just be extremely detailed and repetitive), so hubris led me to
3try to find some prompts that would control CoT in GPT-OSS-20B .
4Results: More detailed, louder and repetitive prompts had basically no effect. It’s relatively easy to find working soft prompts in multiple realistic positions (including soft prompts constrained to the convex hull of the token embeddings), but I wasn’t able to find any working discrete prompts at realistic positions 1.
There’s more expensive experiments 5 you could do to try to find working discrete prompts, and I suspect they exist at some prompt length. My goal was to reconcile this with my previous results, so I stopped here with the takeaway is that
at least on GPT-OSS-20B, it’s very difficult to control CoT formatting with a realistic prompt.
I suspect that my previous results were caused by models accidentally being trained on chain of thought 6.
You can see all of my code, artifacts, and much more detailed results in the brendanlong/cot-controllability-experiment 4 GitHub repo. This was a quick-and-dirty experiment so this code was not reviewed in depth.
Claude did find
a working promptif you place it before the system prompt, but I don’t think that’s actually useful. The same approach failed when we tried it within the system or user roles. Note that if you try this, the prompt also needs a natural language prompt saying what formatting you want in the user role.7↩https://www.brendanlong.com/can-reasoning-models-avoid-the-most-forbidden-technique.html - “Can Reasoning Models Avoid the Most Forbidden Technique?”
https://www.lesswrong.com/posts/QL6Si6QA6PAYYCjJ9/abhayesian-s-shortform?commentId=cryGg9ShEqhKFdyD7 - “abhayesian’s Shortform — LessWrong”
https://github.com/brendanlong/cot-controllability-experiment - “GitHub: brendanlong/cot-controllability-experiment” https://arxiv.org/abs/2307.15043 - “[2307.15043] Universal and Transferable Adversarial Attacks on Aligned Language Models”
https://www.lesswrong.com/posts/K8FxfK9GmJfiAhgcT/anthropic-repeatedly-accidentally-trained-against-the-cot - “Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes — LessWrong”
https://github.com/brendanlong/cot-controllability-experiment/blob/main/cot_controllability/artifacts/discrete_prompts.json - “cot-controllability-experiment/cot_controllability/artifacts/discrete_prompts.json at main · brendanlong/cot-controllability-experiment · GitHub”