Why AI understands formal Arabic but gets lost in dialects

A developer building an Arabic AI newsletter explains that large language models struggle with Arabic dialects due to a lack of training data, inefficient tokenization, and the dominance of formal Arabic (Fusha) in text corpora. The developer notes that open models offer a path to better dialect support through local fine-tuning.

Ask an AI chatbot a question in formal Arabic Fusha and it answers beautifully. Ask it the same thing in Gazan, Egyptian, or Moroccan dialect — and it stumbles, mistranslates, or misses the point entirely. As someone building an Arabic AI newsletter, I run into this every day. Here's what's actually going on under the hood — and why it's a great window into how large language models really work. An LLM learns language by reading enormous amounts of text. For English, that's a huge chunk of the internet. For formal Arabic , there's a decent amount news, books, Wikipedia . But for dialects ? Very little is written down — dialects live in speech, voice notes, and casual chats, not in the formal text models train on. Less data for a language variety = weaker performance. This is the well-known "low-resource language" problem, and Arabic dialects are a textbook case. Arabic isn't one thing. Fusha is the shared formal standard, but everyday speech splits into dialects that can differ more than some European languages do from each other. A model trained mostly on Fusha treats a dialect almost like a foreign language it barely saw. LLMs don't read words — they break text into tokens chunks . Tokenizers are optimized for the languages they saw most. Arabic script, with its rich morphology and prefixes/suffixes, often gets chopped inefficiently — and dialectal spellings which aren't standardized fragment even more. More tokens = more cost and more room for the model to lose meaning. This isn't just an Arabic problem. It's a preview of what happens to any under-represented language or community when AI is trained mostly on high-resource data: the technology works brilliantly for some and poorly for others. The encouraging part: open models are changing this. When powerful models are released openly, any local team can fine-tune them for their own language and dialects — instead of waiting for a big lab to care. That's real digital sovereignty. I write simple Arabic explainers about AI tokens, embeddings, RAG, and more over at ai.salamthabet.com — making machine learning make sense, one term at a time. If the Arabic-AI intersection interests you, come say hi 👋