Arabic (Egyptian dialect) veterinary diagnosis assistant: dialect -> clinical-English gap and low accuracy on real owner messages

A solo founder building an Arabic veterinary diagnosis assistant for livestock and pets in Egypt reported that retrieval-first matching of Egyptian colloquial owner messages against 530,000 chunks from roughly 216 English vet textbooks reached only about 40% exact agreement and about 72% exact-or-related agreement on real owner messages from Arabic Q&A sites. The founder measured that giving the LLM the correct passage yields about 92% top-1 accuracy, and that an LLM-proposes-10-candidate-diseases-then-search-by-disease-name pipeline reaches 92% top-3 on clean Modern Standard Arabic case descriptions, while LLM rewriting of questions into English clinical signs scored 38.3% versus 40.4% without and BGE-M3 was not better than paraphrase-multilingual-mpnet and ran 7x slower. The founder plans self-consistency over candidate lists, 2-3 follow-up questions to separate the top-3, a canonical disease list with hierarchical scoring, and a small paid vet audit.

I’m a solo founder building an Arabic veterinary diagnosis assistant for livestock and pets in Egypt poultry, cattle, buffalo, horses, cats, dogs . Setup: ~216 English vet textbooks → 530k chunks Postgres + pgvector, paraphrase-multilingual-mpnet 768-d with a 128-token window, plus lexical search ; Claude Haiku as the LLM. What we measured: - Retrieval-first embed the Arabic symptoms → top-10 chunks → LLM is weak. - Given the correct passage, the LLM gets ~92% top-1, so the reasoning works. - “LLM proposes 10 candidate diseases → search the books by disease name → LLM picks top-3” reaches 92% top-3 on clean Arabic MSA case descriptions. - On real owner messages from Arabic Q&A sites, agreement is only ~40% exact and ~72% exact or related labels are the replier’s answers, not vet-reviewed . The translation/dialect problem: owners write in Egyptian colloquial Arabic “الفراخ بتنهج وفاتحة بوقها”, “الجاموسة مانعة الأكل” , while all the evidence is English and organised by disease name. What we tried: 1. AGROVOC Arabic–English glossary: +1 case, and generic terms “heat”, “symptoms” diluted queries. 2. A 287-term Egyptian field-term dictionary: fixed 3 cases, broke 3. 3. A 410-term controlled clinical vocabulary for poultry MSA + Egyptian : did not meet our gate. 4. LLM rewriting of the whole question into English clinical signs before diagnosis: 38.3% vs 40.4% without, so no gain. 5. BGE-M3 instead of mpnet: not better on our pilot, and 7x slower. Planned next: self-consistency over candidate lists, 2–3 follow-up questions to separate the top-3, a canonical disease list with hierarchical scoring, and a small paid vet audit. Questions: 1. What has worked for you mapping dialectal or lay symptom text to clinical concepts? Is the right move to normalise the dialect, or to ask follow-up questions? 2. Do you know Egyptian/Arabic medical NLP resources dialect symptom lexicons, annotated corpora or veterinary ontologies VeNom, SNOMED vet that can be used? 3. Any pitfalls with LLM-simulated owners for evaluation? Thanks I think the biggest gap is embedding Egyptian Arabic symptoms straight against English textbook chunks. I’d try a step before retrieval: have the LLM rewrite the owner’s message into a short clinical English symptom list, then retrieve on that. 128 tokens is also pretty tight for textbook text, I’d test 300-500 side by side on your eval set. keeping each chunking + model variant as its own index on the same table makes those comparisons cheap: github.com/pixeltable/pixeltable/blob/main/docs/release/platform/embedding-indexes.ipynb https://github.com/pixeltable/pixeltable/blob/main/docs/release/platform/embedding-indexes.ipynb