{"slug": "are-llms-safe-beyond-text-do-emojis-expose-gaps-in-safety-evaluation", "title": "Are LLMs Safe Beyond Text: Do Emojis Expose Gaps in Safety Evaluation", "summary": "A new arXiv study (2608.18164v1) found that emoji-augmented prompts expose safety evaluation gaps in large language models, with success rates varying from 0% to 10% across four open-source models. Gemma 2 9B and Mistral 7B each showed a 10% success rate, Llama 3 8B 6%, and Qwen 2 7B 0%, with a chi-square test (χ² = 32.94, p < 0.001) confirming significant differences. The findings suggest that text-only safety evaluations may underrepresent model vulnerabilities.", "body_md": "arXiv:2608.18164v1 Announce Type: new\nAbstract: Safety evaluations of large language models (LLMs) predominantly rely on text-based adversarial prompts, potentially overlooking vulnerabilities arising from alternative input representations. This work examines emoji-augmented prompts as a test case for this gap, evaluating 50 prompts across four open-source LLMs (Mistral 7B, Qwen 2 7B, Gemma 2 9B, Llama 3 8B). Results show substantial variation in robustness: Gemma 2 9B and Mistral 7B exhibit non-zero success rates (10%), Llama 3 8B 6%, while Qwen 2 7B shows complete resistance (0% success rate). A chi-square test ($\\chi^2 = 32.94, p < 0.001$) confirms significant differences in outcome distributions. These findings indicate that robustness is sensitive to input representation, and that evaluations restricted to standard text prompts may underrepresent model vulnerabilities.", "url": "https://wpnews.pro/news/are-llms-safe-beyond-text-do-emojis-expose-gaps-in-safety-evaluation", "canonical_source": "https://www.machinebrief.com/news/are-llms-safe-beyond-text-do-emojis-expose-gaps-in-safety-ev-tv88", "published_at": "2026-08-20 04:00:00+00:00", "updated_at": "2026-08-20 05:14:35.079509+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety"], "entities": ["arXiv", "Mistral 7B", "Qwen 2 7B", "Gemma 2 9B", "Llama 3 8B"], "alternates": {"html": "https://wpnews.pro/news/are-llms-safe-beyond-text-do-emojis-expose-gaps-in-safety-evaluation", "markdown": "https://wpnews.pro/news/are-llms-safe-beyond-text-do-emojis-expose-gaps-in-safety-evaluation.md", "text": "https://wpnews.pro/news/are-llms-safe-beyond-text-do-emojis-expose-gaps-in-safety-evaluation.txt", "jsonld": "https://wpnews.pro/news/are-llms-safe-beyond-text-do-emojis-expose-gaps-in-safety-evaluation.jsonld"}}