{"slug": "why-ai-understands-formal-arabic-but-gets-lost-in-dialects", "title": "Why AI understands formal Arabic but gets lost in dialects", "summary": "A developer building an Arabic AI newsletter explains that large language models struggle with Arabic dialects due to a lack of training data, inefficient tokenization, and the dominance of formal Arabic (Fusha) in text corpora. The developer notes that open models offer a path to better dialect support through local fine-tuning.", "body_md": "Ask an AI chatbot a question in formal Arabic (Fusha) and it answers beautifully. Ask it the same thing in Gazan, Egyptian, or Moroccan dialect — and it stumbles, mistranslates, or misses the point entirely.\n\nAs someone building an Arabic AI newsletter, I run into this every day. Here's what's actually going on under the hood — and why it's a great window into how large language models really work.\n\nAn LLM learns language by reading enormous amounts of text. For English, that's a huge chunk of the internet. For **formal Arabic**, there's a decent amount (news, books, Wikipedia). But for **dialects**? Very little is written down — dialects live in speech, voice notes, and casual chats, not in the formal text models train on.\n\nLess data for a language variety = weaker performance. This is the well-known **\"low-resource language\"** problem, and Arabic dialects are a textbook case.\n\nArabic isn't one thing. Fusha is the shared formal standard, but everyday speech splits into dialects that can differ more than some European languages do from each other. A model trained mostly on Fusha treats a dialect almost like a foreign language it barely saw.\n\nLLMs don't read words — they break text into **tokens** (chunks). Tokenizers are optimized for the languages they saw most. Arabic script, with its rich morphology and prefixes/suffixes, often gets chopped inefficiently — and dialectal spellings (which aren't standardized) fragment even more. More tokens = more cost and more room for the model to lose meaning.\n\nThis isn't just an Arabic problem. It's a preview of what happens to **any** under-represented language or community when AI is trained mostly on high-resource data: the technology works brilliantly for some and poorly for others.\n\nThe encouraging part: **open models** are changing this. When powerful models are released openly, any local team can fine-tune them for their own language and dialects — instead of waiting for a big lab to care. That's real digital sovereignty.\n\nI write simple Arabic explainers about AI (tokens, embeddings, RAG, and more) over at ** ai.salamthabet.com** — making machine learning make sense, one term at a time. If the Arabic-AI intersection interests you, come say hi 👋", "url": "https://wpnews.pro/news/why-ai-understands-formal-arabic-but-gets-lost-in-dialects", "canonical_source": "https://dev.to/salamthabet/why-ai-understands-formal-arabic-but-gets-lost-in-dialects-29cm", "published_at": "2026-07-25 11:53:30+00:00", "updated_at": "2026-07-25 12:33:35.830192+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/why-ai-understands-formal-arabic-but-gets-lost-in-dialects", "markdown": "https://wpnews.pro/news/why-ai-understands-formal-arabic-but-gets-lost-in-dialects.md", "text": "https://wpnews.pro/news/why-ai-understands-formal-arabic-but-gets-lost-in-dialects.txt", "jsonld": "https://wpnews.pro/news/why-ai-understands-formal-arabic-but-gets-lost-in-dialects.jsonld"}}