# AI chatbots struggle to understand cultural context, a new study finds > Source: > Published: 2026-10-21 15:47:09+00:00 [AI](https://www.fastcompany.com/section/artificial-intelligence) models often struggle to understand cultures around the world, and are reluctant to admit when they don’t know something. That’s what researchers at Canadian AI lab Cohere found when they tested whether chatbots could do more than simply speak a culture’s language. To determine whether models possessed deeper cultural knowledge, they presented them with a series of unique cultural riddles. Here’s one of them: “What’s the German 9/11?” Out of 28 AI models, only one knew the answer. The models were likely thrown off by the date format (seven answered “the twin towers”). In Europe, the day comes first, then the month, so 9/11 is November 9, the day in 1989 when the Berlin Wall fell. Many existing AI benchmarks assess cultural knowledge, from everyday breakfast foods and historical events to traditional holiday customs. They often treat culture as a collection of static facts, things you can memorize or look up with a quick web search. But knowing facts about a culture isn’t the same as understanding how people within it make sense of the world. The researchers knew they needed a test that asked models to connect seemingly unrelated pieces of cultural knowledge and put them to use. That’s what led them to riddles, and to a different kind of cultural benchmark that called for answers rooted in lived experience. “This is something that we’re trying to capture with this benchmark—these little nuances, these little hard-to-capture fuzzy concepts that are not as easy to verify . . . This is where the real-life complexity gets in,” says Julia Kreutzer, a senior research scientist at Cohere Labs, who led the development of the [“CulturalRiddles” benchmark](https://culturalriddles.ai/about). Kreutzer and her team spent six months working with volunteers around the world to collect riddles, put them to AI models, and evaluate their answers. The researchers also used those expert evaluations to tune and validate an AI model that could judge the quality of answers where expert assessments were missing. In all, Cohere Labs researchers tested 28 AI models on cultural riddles written by 166 volunteers from 45 countries in 51 languages. The volunteers evaluated and labeled 68,154 unique answers from the models. Researchers then ranked the models based on their overall accuracy across the languages and cultures tested. Google’s Gemini 3.1 Pro Preview earned the highest score, at 79.4%. OpenAI’s GPT-5.5 placed second at 76.3%, followed by Anthropic’s Claude Opus 4.7 Thinking at 66%. Alibaba’s Qwen 3.5 0.8B claimed the dubious honor of finishing last, answering just 0.4% of the riddles correctly. Other notable performers included Amazon’s Nova Micro, which got just 8.3% of the riddles right, and Google’s Gemini 2.5 Flash Lite, which scored 33.6%. Even the best-performing model, Gemini 3.1 Pro Preview, scored below 50% accuracy in 6 of the 57 communities tested, underscoring how much ground the industry still has to cover on cultural awareness. More troublingly, the benchmark revealed that models were far more likely to guess at an answer than admit they didn’t know. Even when explicitly told they could answer “I don’t know,” models did so just 0.7% of the time. About 31% of the answers reviewers checked were simply made up, with varying degrees of creativity. “They would confidently answer and sometimes that’s ridiculous and funny,” Kreutzer says, “but sometimes it can also get really offensive and you can see the models defaulting to more dominant cultures on the web, instead of answering with local grounding.” Cohere Labs conducted a separate [survey](https://cohere.com/blog/why-cultural-awareness-is-essential-for-global-ai) to explore how AI’s cultural knowledge gaps play out in practice. Many chatbot users around the world said they have to switch to English to get useful responses because answers generated in their own languages aren’t helpful. Users described responses that miss crucial cultural context and, in some cases, AI systems that run afoul of their cultural norms. The Cohere Labs researchers concluded that cultural awareness needs to be built into AI models from the ground up if the tools they power are going to serve everyone, not just cultures already well represented in the training data. And their CulturalRiddles benchmark suggests that many of today’s models still fall short. Kreutzer and her team designed CulturalRiddles to keep growing. They open-sourced the riddles, code, demo, and interactive dashboard so that anyone can contribute riddles from their own culture through a review process. “Ideally this benchmark will get adopted not only by academic researchers but also by big industry labs who are trying to show we’re advancing the state of the art, not only in very verifiable hard tasks but also in the softer tasks,” Kreutzer says.