Are LLMs Safe Beyond Text: Do Emojis Expose Gaps in Safety Evaluation

A new arXiv study (2608.18164v1) found that emoji-augmented prompts expose safety evaluation gaps in large language models, with success rates varying from 0% to 10% across four open-source models. Gemma 2 9B and Mistral 7B each showed a 10% success rate, Llama 3 8B 6%, and Qwen 2 7B 0%, with a chi-square test (χ² = 32.94, p < 0.001) confirming significant differences. The findings suggest that text-only safety evaluations may underrepresent model vulnerabilities.

arXiv:2608.18164v1 Announce Type: new Abstract: Safety evaluations of large language models LLMs predominantly rely on text-based adversarial prompts, potentially overlooking vulnerabilities arising from alternative input representations. This work examines emoji-augmented prompts as a test case for this gap, evaluating 50 prompts across four open-source LLMs Mistral 7B, Qwen 2 7B, Gemma 2 9B, Llama 3 8B . Results show substantial variation in robustness: Gemma 2 9B and Mistral 7B exhibit non-zero success rates 10% , Llama 3 8B 6%, while Qwen 2 7B shows complete resistance 0% success rate . A chi-square test $\chi^2 = 32.94, p < 0.001$ confirms significant differences in outcome distributions. These findings indicate that robustness is sensitive to input representation, and that evaluations restricted to standard text prompts may underrepresent model vulnerabilities.