The "Knowledge-Behavior Gap" in Cultural Taboo Safety of Large Language Models

Researchers introduced CulShield, the first public benchmark for evaluating cultural taboo safety in large language models, covering 77 countries and territories with over 2,020 taboos. Testing on advanced models such as GPT-4o-mini and Gemini-2.5-pro revealed a 'knowledge-behavior gap,' where models fail to apply known taboos during interaction, and showed that linguistic context variations significantly affect safety.

arXiv:2608.12341v1 Announce Type: new Abstract: Cultural taboo safety is essential for deploying large language models LLMs , as culturally insensitive outputs may cause offense or even social harm. However, existing cultural benchmarks primarily assess cultural knowledge or values biases, while overlooking whether LLMs can recognize and respect cultural taboos, especially when taboos are implicitly hidden in seemingly harmless questions. Besides, cultural taboos are implicit, and context-dependent, thus poss unique challenges for reliable evaluation. To address these gaps, we introduce \textbf{CulShield}, the first public benchmark dedicated to evaluating and improving the cultural taboo safety of LLMs. CulShield spans 77 countries and territories, and includes over 2,020 taboos. It evaluates models along both explicit knowledge and implicit behaviors. Experiments on several advanced LLMs e.g., GPT-4o-mini, Gemini-2.5-pro reveal a clear knowledge-behavior gap'': models often fail to apply known taboos during interaction. We further show that variations in linguistic context can significantly affect LLMs' cultural taboo safety. Code and data is accessible here: https://github.com/hedyHe/CulShield.