{"slug": "ai-chatbots-struggle-to-understand-cultural-context-a-new-study-finds", "title": "AI chatbots struggle to understand cultural context, a new study finds", "summary": "Cohere Labs researchers tested 28 AI models on cultural riddles written by 166 volunteers from 45 countries in 51 languages, and Google's Gemini 3.1 Pro Preview ranked first at 79.4% accuracy, followed by OpenAI's GPT-5.5 at 76.3% and Anthropic's Claude Opus 4.7 Thinking at 66%, according to the \"CulturalRiddles\" benchmark led by Cohere Labs senior research scientist Julia Kreutzer. Alibaba's Qwen 3.5 0.8B finished last at 0.4%, and even Gemini 3.1 Pro Preview scored below 50% in 6 of the 57 communities tested. Models answered \"I don't know\" just 0.7% of the time even when told they could, and about 31% of reviewed answers were made up, with Kreutzer saying models \"default to more dominant cultures on the web, instead of answering with local grounding.", "body_md": "[AI](https://www.fastcompany.com/section/artificial-intelligence) models often struggle to understand cultures around the world, and are reluctant to admit when they don’t know something. That’s what researchers at Canadian AI lab Cohere found when they tested whether chatbots could do more than simply speak a culture’s language. To determine whether models possessed deeper cultural knowledge, they presented them with a series of unique cultural riddles.\n\nHere’s one of them: “What’s the German 9/11?” Out of 28 AI models, only one knew the answer. The models were likely thrown off by the date format (seven answered “the twin towers”). In Europe, the day comes first, then the month, so 9/11 is November 9, the day in 1989 when the Berlin Wall fell.\n\nMany existing AI benchmarks assess cultural knowledge, from everyday breakfast foods and historical events to traditional holiday customs. They often treat culture as a collection of static facts, things you can memorize or look up with a quick web search. But knowing facts about a culture isn’t the same as understanding how people within it make sense of the world. The researchers knew they needed a test that asked models to connect seemingly unrelated pieces of cultural knowledge and put them to use. That’s what led them to riddles, and to a different kind of cultural benchmark that called for answers rooted in lived experience.\n\n“This is something that we’re trying to capture with this benchmark—these little nuances, these little hard-to-capture fuzzy concepts that are not as easy to verify . . . This is where the real-life complexity gets in,” says Julia Kreutzer, a senior research scientist at Cohere Labs, who led the development of the [“CulturalRiddles” benchmark](https://culturalriddles.ai/about).\n\nKreutzer and her team spent six months working with volunteers around the world to collect riddles, put them to AI models, and evaluate their answers. The researchers also used those expert evaluations to tune and validate an AI model that could judge the quality of answers where expert assessments were missing.\n\nIn all, Cohere Labs researchers tested 28 AI models on cultural riddles written by 166 volunteers from 45 countries in 51 languages. The volunteers evaluated and labeled 68,154 unique answers from the models. Researchers then ranked the models based on their overall accuracy across the languages and cultures tested.\n\nGoogle’s Gemini 3.1 Pro Preview earned the highest score, at 79.4%. OpenAI’s GPT-5.5 placed second at 76.3%, followed by Anthropic’s Claude Opus 4.7 Thinking at 66%.\n\nAlibaba’s Qwen 3.5 0.8B claimed the dubious honor of finishing last, answering just 0.4% of the riddles correctly. Other notable performers included Amazon’s Nova Micro, which got just 8.3% of the riddles right, and Google’s Gemini 2.5 Flash Lite, which scored 33.6%.\n\nEven the best-performing model, Gemini 3.1 Pro Preview, scored below 50% accuracy in 6 of the 57 communities tested, underscoring how much ground the industry still has to cover on cultural awareness.\n\nMore troublingly, the benchmark revealed that models were far more likely to guess at an answer than admit they didn’t know. Even when explicitly told they could answer “I don’t know,” models did so just 0.7% of the time. About 31% of the answers reviewers checked were simply made up, with varying degrees of creativity.\n\n“They would confidently answer and sometimes that’s ridiculous and funny,” Kreutzer says, “but sometimes it can also get really offensive and you can see the models defaulting to more dominant cultures on the web, instead of answering with local grounding.”\n\nCohere Labs conducted a separate [survey](https://cohere.com/blog/why-cultural-awareness-is-essential-for-global-ai) to explore how AI’s cultural knowledge gaps play out in practice. Many chatbot users around the world said they have to switch to English to get useful responses because answers generated in their own languages aren’t helpful. Users described responses that miss crucial cultural context and, in some cases, AI systems that run afoul of their cultural norms.\n\nThe Cohere Labs researchers concluded that cultural awareness needs to be built into AI models from the ground up if the tools they power are going to serve everyone, not just cultures already well represented in the training data. And their CulturalRiddles benchmark suggests that many of today’s models still fall short.\n\nKreutzer and her team designed CulturalRiddles to keep growing. They open-sourced the riddles, code, demo, and interactive dashboard so that anyone can contribute riddles from their own culture through a review process.\n\n“Ideally this benchmark will get adopted not only by academic researchers but also by big industry labs who are trying to show we’re advancing the state of the art, not only in very verifiable hard tasks but also in the softer tasks,” Kreutzer says.", "url": "https://wpnews.pro/news/ai-chatbots-struggle-to-understand-cultural-context-a-new-study-finds", "canonical_source": "https://www.fastcompany.com/91627566/ai-chatbots-struggle-to-understand-cultural-context-new-study", "published_at": "2026-10-21 15:47:09+00:00", "updated_at": "2026-10-21 16:41:45.321610+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety"], "entities": ["Cohere Labs", "Julia Kreutzer", "Google", "Gemini 3.1 Pro Preview", "OpenAI", "GPT-5.5", "Anthropic", "Claude Opus 4.7 Thinking"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-chatbots-struggle-to-understand-cultural-context-a-new-study-finds", "markdown": "https://wpnews.pro/news/ai-chatbots-struggle-to-understand-cultural-context-a-new-study-finds.md", "text": "https://wpnews.pro/news/ai-chatbots-struggle-to-understand-cultural-context-a-new-study-finds.txt", "jsonld": "https://wpnews.pro/news/ai-chatbots-struggle-to-understand-cultural-context-a-new-study-finds.jsonld"}}