{"slug": "reading-ingredient-lists-from-phone-photos-a-crop-bug-a-model-bench-and-why-the", "title": "Reading ingredient lists from phone photos: a crop bug, a model bench, and why the AI never gives the verdict", "summary": "A developer built Halal Food Scanner, an Expo/React Native app that uses a vision model only to read ingredient names from pack photos and maps them to a human-reviewed table of 401 E-numbers and named ingredients, keeping the halal verdict out of the model's hands after bench testing showed some models invented ingredients. A crop-to-text-box optimization raised empty-read rates from about 40% to 48%, so the developer removed cropping and now uploads the full frame at 1600 px, adding roughly 0.8 s per read, and later dropped the on-device ML Kit readability check that forced retakes on a third of scans. Users who got an answer on their first day returned 38% of the time versus 10% for those who got none.", "body_md": "I build Halal Food Scanner, an iPhone and Android app (Expo / React Native) that tells you if a food is halal. You scan the barcode, and when the product isn't in Open Food Facts or has no ingredient list there, you take a photo of the ingredients instead. The app answers Halal, Haram or Uncertain, with the reason for each ingredient.\n\nThe photo path is where most of the hard work is. This is what we learned building it, including a bug I shipped myself.\n\nThe first decision was the most important one: the AI does not decide if something is halal.\n\nA photo goes to a vision model in one call. The model has one job: find the ingredient names on the pack and map each one to an id in our own table (\"Gelatine\" -> gelatin, \"E471\" / \"mono- and diglycerides of fatty acids\" -> e471, \"Lab\" -> rennet). The verdict comes from the table, not from the model.\n\nThe table is a human-reviewed list of 401 E-numbers (261 halal, 138 uncertain, 2 haram) plus named ingredients like gelatin, rennet, whey and natural flavourings. Each row is sourced to halal bodies and standards (JAKIM MS 1500, IFANCA, SANHA, MUI and others). Where they disagree, the app follows the most cautious reviewed position and shows the others.\n\nTwo rules fall out of this:\n\nWhy so strict? Because when we benched models on real photos, some of them invented ingredients. On a 30-photo bench of Japanese and French packs, one model added ingredients that weren't on the pack at all. A small, fast model did the same on other labels. In a halal app an invented ingredient is a wrong answer, so the model only gets the part of the job where a mistake is visible and checkable: reading.\n\nWe now send photos to Gemini Flash first and only fall back to other models when it fails or times out. For blurry, sideways Arabic labels, an empty read gets one retry with the stronger model.\n\nAt the time, ML Kit ran a quick text check on the phone before we uploaded anything. It told us if the photo had readable text at all, so we could ask for a retake right away instead of after a network round trip.\n\nOn October 15 I made what looked like a smart change: ML Kit already knows where the text is, so crop the upload to that text box. Smaller image, bigger letters, better reads. We were sending the photo at 1024 px on the long edge, so the crop should help small print.\n\nIt didn't. After the change, the share of ingredient photos that came back with no ingredients went from about 40% to 48%.\n\nThe reason only showed up when I looked at the photos. Since mid-October the app keeps failed ingredient photos for 30 days in a private bucket, so we can check failures by hand. Two things were in there:\n\nThe fix was to stop cropping. The whole frame now goes up at 1600 px instead of 1024. On a bench of 8 full-resolution Open Food Facts photos, 7 read the same and one small multilingual label went from 4 lumped strings to 18 separate ingredients. The cost is about 0.8 s more per read (median ~3.8 s -> ~4.6 s), which is fine for a photo you take once.\n\nTwo weeks later the same kind of look at the data killed the ML Kit check completely. A third of ingredients scans took 2 or 3 shots because ML Kit judged the photo unreadable, and Arabic and Japanese labels always used all three, even though the cloud model reads them fine. Median capture time was 4.6 s with retakes against 1.9 s for one shot. Now the app sends the first photo straight away, and the native module is gone from the app.\n\nThe old app showed an error alert when a photo came back empty. Now it shows a small picture sheet: wrong side of the pack, too blurry, and from the first miss a button to scan the barcode instead. Showing a picture of a good ingredients photo works better than any sentence we wrote.\n\nThis matters more than it looks. People whose first day gave them an answer came back on another day 38% of the time. People who got no answer came back 10% of the time.\n\nIf you want to see the rulings the table uses, they're public on the site, for example [is E471 halal?](https://halalfoodscanner.app/ingredients/e471/) and the [full E-number list](https://halalfoodscanner.app/ingredients/). The app is at [halalfoodscanner.app](https://halalfoodscanner.app/).", "url": "https://wpnews.pro/news/reading-ingredient-lists-from-phone-photos-a-crop-bug-a-model-bench-and-why-the", "canonical_source": "https://dev.to/ahmedashourdev/reading-ingredient-lists-from-phone-photos-a-crop-bug-a-model-bench-and-why-the-ai-never-gives-4bc9", "published_at": "2026-11-02 19:38:50+00:00", "updated_at": "2026-11-02 19:53:40.091767+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "ai-products", "ai-tools"], "entities": ["Halal Food Scanner", "Gemini Flash", "ML Kit", "Open Food Facts", "JAKIM MS 1500", "IFANCA", "SANHA", "MUI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/reading-ingredient-lists-from-phone-photos-a-crop-bug-a-model-bench-and-why-the", "markdown": "https://wpnews.pro/news/reading-ingredient-lists-from-phone-photos-a-crop-bug-a-model-bench-and-why-the.md", "text": "https://wpnews.pro/news/reading-ingredient-lists-from-phone-photos-a-crop-bug-a-model-bench-and-why-the.txt", "jsonld": "https://wpnews.pro/news/reading-ingredient-lists-from-phone-photos-a-crop-bug-a-model-bench-and-why-the.jsonld"}}