I’m a solo founder building an Arabic veterinary diagnosis assistant for livestock and pets in Egypt (poultry, cattle, buffalo, horses, cats, dogs).
Setup: ~216 English vet textbooks → 530k chunks (Postgres + pgvector, paraphrase-multilingual-mpnet 768-d with a 128-token window, plus lexical search); Claude Haiku as the LLM.
What we measured:
- Retrieval-first (embed the Arabic symptoms → top-10 chunks → LLM) is weak.
- Given the correct passage, the LLM gets ~92% top-1, so the reasoning works.
- “LLM proposes 10 candidate diseases → search the books by disease name → LLM picks top-3” reaches 92% top-3 on clean Arabic (MSA) case descriptions.
- On real owner messages from Arabic Q&A sites, agreement is only ~40% exact and ~72% exact or related (labels are the replier’s answers, not vet-reviewed).
The translation/dialect problem: owners write in Egyptian colloquial Arabic (“الفراخ بتنهج وفاتحة بوقها”, “الجاموسة مانعة الأكل”), while all the evidence is English and organised by disease name. What we tried:
- AGROVOC Arabic–English glossary: +1 case, and generic terms (“heat”, “symptoms”) diluted queries.
- A 287-term Egyptian field-term dictionary: fixed 3 cases, broke 3.
- A 410-term controlled clinical vocabulary for poultry (MSA + Egyptian): did not meet our gate.
- LLM rewriting of the whole question into English clinical signs before diagnosis: 38.3% vs 40.4% without, so no gain.
- BGE-M3 instead of mpnet: not better on our pilot, and 7x slower.
Planned next: self-consistency over candidate lists, 2–3 follow-up questions to separate the top-3, a canonical disease list with hierarchical scoring, and a small paid vet audit.
Questions:
- What has worked for you mapping dialectal or lay symptom text to clinical concepts? Is the right move to normalise the dialect, or to ask follow-up questions?
- Do you know Egyptian/Arabic medical NLP resources (dialect symptom lexicons, annotated corpora) or veterinary ontologies (VeNom, SNOMED vet) that can be used?
- Any pitfalls with LLM-simulated owners for evaluation?
Thanks!
I think the biggest gap is embedding Egyptian Arabic symptoms straight against English textbook chunks. I’d try a step before retrieval: have the LLM rewrite the owner’s message into a short clinical English symptom list, then retrieve on that. 128 tokens is also pretty tight for textbook text, I’d test 300-500 side by side on your eval set. keeping each chunking + model variant as its own index on the same table makes those comparisons cheap: github.com/pixeltable/pixeltable/blob/main/docs/release/platform/embedding-indexes.ipynb