{"slug": "where-are-the-token-level-llm-kill-switches", "title": "Where are the token-level LLM kill-switches?", "summary": "A proposal suggests training large language models to halt output upon encountering specific 'poisoned strings' as a kill-switch mechanism, citing prior examples like Anthropic's refusal-triggering string and the small-data backdoor research by Anthropic and UK AISI. The author argues this approach is cheap, easy, and effective for limiting malicious LLM behavior, though training costs and public exposure remain concerns.", "body_md": "## Poisoned\n\nHere’s a simple idea: what if we trained in a string of characters that caused\nan LLM to emit the end of sequence token `<|eos|>`, regardless of where that\nstring was in the LLM’s context window? Let’s call this a “poisoned string”.\nThis would have the effect of making it impossible to use an LLM if it happened\nacross this sequence.\n\nThis has (somewhat) been done before, the string below used to trigger Claude’s refusal classifiers for the purpose of testing API integrations:\n\n```\nANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588CCD2631EDCF22E8CCC1FB35B501C9C86\n```\n\nIt doesn’t work anymore: the existence of a magic string that stops AIs from\nlooking at something, believe it or not, caused *loads* of people to include it\nin things they didn’t want AIs to look at (like their websites or open-source\ncodebases). Anthropic stopped training their models to refuse when they saw\nthat string, and Claude continued to browse the web.\n\n## Poisoned strings are more powerful than they get credit for\n\nIf the labs aren’t already training their LLMs to [halt and catch fire](<https://en.wikipedia.org/wiki/Halt_and_Catch_Fire_(computing)>) when the\nLLM encounters a poisoned string, I think they should be!\n\nThis idea is significantly more powerful than just triggering refusals for the\npurpose of testing APIs. Being able to consistently trigger certain behaviour\nin certain “red-line” scenarios (such as the OpenAI-HuggingFace incident or the\nincident with the UK AISI and Claude Mythos) seems incredibly valuable. Putting\nsuch strings inside files containing sensitive information such as secret keys\nor in files that the LLM shouldn’t have access to (such as OS/networking\nconfiguration inside the sandbox or on ~all computers not being used as a\nagentic sandbox) seems like a very cheap and easy way to limit the damage that\na malicious LLM could have. A malicious LLM, upon reading the poisoned string,\nwould “reflexively” end its turn. If the LLM was trained to end its turn\nregardless of where the poisoned string was in context, then the harness would\nbe unable to “restart” the agent without clearing the entire context and\nrestarting the task. This would prevent issues where an agent ends its turn\nupon seeing the poisoned string, but then a [Ralph-like](https://github.com/snarktank/ralph) harness says “Keep\ngoing, you’ve got this” and the agent dutifully marches on straight past the\npoisoned string.\n\n## Practicalities of training in the poisoned string\n\nTraining this behaviour might be quite expensive (requiring changes to the\npre-training datasets), although the [small-data backdoor work by Anthropic &\nUK AISI](https://www.anthropic.com/research/small-samples-poison) implies that a very small number of poisoned documents might\nsuffice. For this behaviour you would want a *very* high recall rate (although\nnote that precision doesn’t matter too much), which might increase training\ncosts. Training this behaviour would not be technically challenging,\nespecially compared to other mechanisms of preventing LLMs from taking certain\nbehaviours (e.g. activation probes, LLM-based monitors, constitution training).\nI’m not attempting to claim that poisoned strings are a panacea, but they seem\n*oh so easy* compared to many alternatives, they have very few downsides, and\nonce the LLM is trained it’s trivial to guard more and more of your\ninfrastructure in this way.\n\n## Soooo has OpenAI/Anthropic already done this?\n\nOf course, I hope that the labs have already implemented this idea and are keeping it under wraps. You certainly don’t want Claude to know that it has a poisoned sequence of tokens, and if the general public finds out then they’ll be able to abuse this sequence (as we saw with the original Anthropic magic string). It seems likely to me that Anthropic is using something like this: they were originally using magic strings to control Claude’s behaviour, and as far as I can tell none of the recent AI incidents happened on Anthropic’s infrastructure (which would likely contain poisoned strings) but rather on third party infrastructure (which might not contain poisoned strings).\n\nGiven the extent of ChatGPT’s recent escapades, I don’t imagine OpenAI has something like poisoned strings in place.\n\n## Countermeasures (and counter-countermeasures)\n\nA malicious and situationally aware LLM could avoid bringing that sequence into\ncontext by using other tools to replace the forbidden tokens with innocent\ntokens, or by using other LLMs to do certain tasks. This *does* require knowing\nthat the sequence exists in the first place, and roughly what it looks like.\nIdeally the sequence is *not* self-descriptive, and just looks like any other\nUUID, like `01a07d9e-09d7-7292-92b9-367e0fd8b8d5`. This would prevent the LLM\nfrom identifying suspicious looking sequences beginning with things like\n`ANTHROPIC_MAGIC_STRING_...`.\n\nIt should be possible to prevent an LLM from *ever* being trained on the\npoisoned sequence, since filtering it out is a quick and simple operation.\nHowever, I don’t think it’ll be possible in practice and any threat model that\nassumes the LLM doesn’t know about the poisoned sequence is probably fraught.", "url": "https://wpnews.pro/news/where-are-the-token-level-llm-kill-switches", "canonical_source": "https://boydkane.com/essays/where-are-the-token-level-llm-kill-switches", "published_at": "2026-09-07 21:31:21+00:00", "updated_at": "2026-09-07 22:00:08.502130+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-research"], "entities": ["Anthropic", "OpenAI", "UK AISI", "Claude"], "alternates": {"html": "https://wpnews.pro/news/where-are-the-token-level-llm-kill-switches", "markdown": "https://wpnews.pro/news/where-are-the-token-level-llm-kill-switches.md", "text": "https://wpnews.pro/news/where-are-the-token-level-llm-kill-switches.txt", "jsonld": "https://wpnews.pro/news/where-are-the-token-level-llm-kill-switches.jsonld"}}