The research looked at two specific areas: the last-layer-last-token embedding space and the top-50 next-token probability space. The results are pretty telling for anyone interested in LLM security and prompt engineering.
Internal Representation Analysis #
Embedding Space: The last-layer embeddings tend to group prompts based on their surface-level characteristics—things like spelling errors, weird formatting, or specific character perturbations. It's more about the "look" of the prompt than the intent.Probability Space: The top-50 next-token space is functionally one-dimensional but way messier to cluster. It tracks what the model is likely to say next.
The most interesting finding is the lack of a "behavioral hyperplane." In simpler terms, there isn't a clean, mathematical line in these spaces that separates "refusal" from "compliance."
However, a few specific tokens act as massive tells for a successful bypass. In the Qwen-2.5-1.5B model, the token "Sure" is heavily associated with a compliant response. For Llama models, the comma (",") and the double newline ("ĊĊ") show a similar correlation.
This suggests that while the model might be "tricked" by the perturbation, the transition from refusal to compliance often hinges on a few critical pivot tokens that signal the model has entered a "helpful" state rather than a "safe" state. For those building a real-world AI workflow or researching LLM agents, this highlights just how fragile the boundary between a guarded response and a jailbroken one actually is.
Next Video LLMs are failing a basic logic test →