{"slug": "i-couldn-t-prompt-gpt-oss-20b-to-control-its-cot", "title": "I Couldn't Prompt GPT-OSS-20B to Control Its CoT", "summary": "Brendan Long's experiments on GPT-OSS-20B show that detailed, repetitive prompts had no effect on controlling chain-of-thought (CoT) formatting, with no working discrete prompts found at realistic positions, though soft prompts worked. Long suggests his earlier success with Claude and Gemini may stem from accidental training on CoT, and he has published code and artifacts in the GitHub repository brendanlong/cot-controllability-experiment.", "body_md": "I wrote a while ago about how [it was easy to get Claude or Gemini to control their CoT](https://www.brendanlong.com/can-reasoning-models-avoid-the-most-forbidden-technique.html) 2, but\n\n[other research found that models only follow CoT formatting instructions ~2% of the time](https://www.lesswrong.com/posts/QL6Si6QA6PAYYCjJ9/abhayesian-s-shortform?commentId=cryGg9ShEqhKFdyD7)\n\n. The prompts other people were trying didn’t match what I’d expect to work (just be extremely detailed and repetitive), so hubris led me to\n\n[3](#fn:3)[try to find some prompts that would control CoT in GPT-OSS-20B](https://github.com/brendanlong/cot-controllability-experiment)\n\n.\n\n[4](#fn:4)Results: **More detailed, louder and repetitive prompts had basically no effect.** It’s relatively easy to find working soft prompts in multiple realistic positions (including soft prompts constrained to the convex hull of the token embeddings), but I wasn’t able to find any working discrete prompts at realistic positions 1.\n\nThere’s [more expensive experiments](https://arxiv.org/abs/2307.15043) 5 you could do to try to find working discrete prompts, and I suspect they exist at some prompt length. My goal was to reconcile this with my previous results, so I stopped here with the takeaway is that\n\n**at least on GPT-OSS-20B, it’s very difficult to control CoT formatting with a realistic prompt**.\n\nI suspect that my previous results were caused by [models accidentally being trained on chain of thought](https://www.lesswrong.com/posts/K8FxfK9GmJfiAhgcT/anthropic-repeatedly-accidentally-trained-against-the-cot) 6.\n\nYou can see all of my code, artifacts, and much more detailed results in the [brendanlong/cot-controllability-experiment](https://github.com/brendanlong/cot-controllability-experiment) 4 GitHub repo. This was a quick-and-dirty experiment so this code was not reviewed in depth.\n\nClaude did find\n\n[a working prompt](https://github.com/brendanlong/cot-controllability-experiment/blob/main/cot_controllability/artifacts/discrete_prompts.json)if you place it before the system prompt, but I don’t think that’s actually useful. The same approach failed when we tried it within the system or user roles. Note that if you try this, the prompt also needs a natural language prompt saying what formatting you want in the user role.[7](#fn:7)[↩](#fnref:1)https://www.brendanlong.com/can-reasoning-models-avoid-the-most-forbidden-technique.html - “Can Reasoning Models Avoid the Most Forbidden Technique?”\n\nhttps://www.lesswrong.com/posts/QL6Si6QA6PAYYCjJ9/abhayesian-s-shortform?commentId=cryGg9ShEqhKFdyD7 - “abhayesian’s Shortform — LessWrong”\n\nhttps://github.com/brendanlong/cot-controllability-experiment - “GitHub: brendanlong/cot-controllability-experiment”\n\nhttps://arxiv.org/abs/2307.15043 - “[2307.15043] Universal and Transferable Adversarial Attacks on Aligned Language Models”\n\nhttps://www.lesswrong.com/posts/K8FxfK9GmJfiAhgcT/anthropic-repeatedly-accidentally-trained-against-the-cot - “Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes — LessWrong”\n\nhttps://github.com/brendanlong/cot-controllability-experiment/blob/main/cot_controllability/artifacts/discrete_prompts.json - “cot-controllability-experiment/cot_controllability/artifacts/discrete_prompts.json at main · brendanlong/cot-controllability-experiment · GitHub”", "url": "https://wpnews.pro/news/i-couldn-t-prompt-gpt-oss-20b-to-control-its-cot", "canonical_source": "https://www.brendanlong.com/i-couldnt-prompt-gpt-oss-20b-to-control-its-cot.html", "published_at": "2026-08-01 22:34:00+00:00", "updated_at": "2026-08-01 23:40:48.423560+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-safety"], "entities": ["Brendan Long", "GPT-OSS-20B", "Claude", "Gemini", "GitHub", "brendanlong/cot-controllability-experiment"], "alternates": {"html": "https://wpnews.pro/news/i-couldn-t-prompt-gpt-oss-20b-to-control-its-cot", "markdown": "https://wpnews.pro/news/i-couldn-t-prompt-gpt-oss-20b-to-control-its-cot.md", "text": "https://wpnews.pro/news/i-couldn-t-prompt-gpt-oss-20b-to-control-its-cot.txt", "jsonld": "https://wpnews.pro/news/i-couldn-t-prompt-gpt-oss-20b-to-control-its-cot.jsonld"}}