{"slug": "visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam", "title": "Visual-Pill-ID: Building an AI Pharmacist with GPT-4o and SAM 💊", "summary": "A developer built Visual-Pill-ID, a computer vision pipeline that combines the Segment Anything Model (SAM) for instance segmentation with GPT-4o's multimodal reasoning to identify loose pills from a single photo. The system segments each pill, crops and preprocesses it with OpenCV, then uses GPT-4o to perform OCR on curved surfaces and cross-check identified medications against the user's prescription. The developer notes that production-grade medical vision systems would still need to handle edge cases like glare on blister packs and HIPAA-compliant data handling.", "body_md": "Ever stared at a handful of loose pills and wondered, \"Wait, was the blue one for my allergies or my blood pressure?\" 😅 You're not alone. Medication errors are a massive global health challenge.\n\nIn this tutorial, we are building **Visual-Pill-ID**, a cutting-edge computer vision pipeline that solves the \"multi-pill confusion\" problem. By combining the geometric precision of the **Segment Anything Model (SAM)** with the multimodal reasoning of **GPT-4o**, we can transform a messy photo of mixed medication into a structured, verified prescription list.\n\nWe'll be diving deep into **Computer Vision**, **Instance Segmentation**, and **Multimodal LLMs** to create a production-ready OCR and identification system.\n\nThe biggest challenge in pill identification isn't just \"seeing\" the pill; it's isolating it from a crowded background and understanding its specific markings. Our pipeline follows a \"Segment-then-Analyze\" pattern.\n\n``` php\ngraph TD\n    A[Raw Image of Multiple Pills] --> B[SAM: Segment Anything Model]\n    B --> C{Instance Masks}\n    C --> D[OpenCV: Crop & Preprocess]\n    D --> E[GPT-4o Vision: Multi-modal Analysis]\n    E --> F[OCR & Pill Identification]\n    F --> G[Prescription Validation & Safety Logic]\n    G --> H[Final Structured JSON Output]\n```\n\nTo follow along, you’ll need:\n\n`vit_h` or `vit_b` checkpoints.\nTraditional bounding boxes often overlap when pills are touching. We need **Instance Segmentation** to get the exact pixels of each pill.\n\n``` python\nimport numpy as np\nimport torch\nimport cv2\nfrom segment_anything import sam_model_registry, SamAutomaticMaskGenerator\n\n# Load the SAM model\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nsam = sam_model_registry[\"vit_h\"](checkpoint=\"sam_vit_h_4b8939.pth\").to(device)\n\n# Generate masks automatically\nmask_generator = SamAutomaticMaskGenerator(sam)\n\ndef get_pill_masks(image_path):\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    masks = mask_generator.generate(image)\n\n    # Filter by area to remove tiny artifacts\n    filtered_masks = [m for m in masks if m['area'] > 500]\n    return filtered_masks, image\n\nprint(f\"🚀 Detected {len(get_pill_masks('pills.jpg')[0])} potential pills!\")\n```\n\nOnce we have the masks, we crop each pill. However, sending 10 separate images to GPT-4o is expensive. Instead, we create a \"Collage of Interest\" or send them in a structured batch. GPT-4o is incredible at **OCR on curved surfaces**, which is typical for medication.\n\n``` python\nimport base64\nimport requests\n\ndef encode_image(image_np):\n    _, buffer = cv2.imencode('.jpg', image_np)\n    return base64.b64encode(buffer).decode('utf-8')\n\ndef identify_pills(pill_crops):\n    # Constructing the multimodal prompt\n    prompt_content = [\n        {\"type\": \"text\", \"text\": \"Identify each pill in these images. Extract markings, color, and shape. Compare with standard medical databases.\"}\n    ]\n\n    for crop in pill_crops:\n        base64_image = encode_image(crop)\n        prompt_content.append({\n            \"type\": \"image_url\",\n            \"image_url\": {\"url\": f\"data:image/jpeg;base64,{base64_image}\"}\n        })\n\n    # Call GPT-4o\n    # API implementation details...\n    # Return structured JSON\n```\n\nIt's not enough to know it's \"Ibuprofen 200mg.\" We need to know if it matches the user's prescription. By feeding the OCR text from the medicine bottle (also captured in the pipeline) and the identified pills into GPT-4o, we can perform a cross-check.\n\nWhile this DIY pipeline is great for prototyping, building a production-grade medical vision system requires handling edge cases like glare on blister packs and HIPAA-compliant data handling.\n\nFor more production-ready examples and advanced patterns on integrating LLMs with specialized computer vision models, check out the detailed guides at **[WellAlly Tech Blog](https://www.wellally.tech/blog)**. They cover deep-dives into AI reliability that are crucial for healthcare applications. 🥑\n\nHere is a snippet showing how we combine the SAM mask with an OpenCV crop to feed the vision model:\n\n``` python\ndef process_pipeline(img_path):\n    masks, original_img = get_pill_masks(img_path)\n    pill_data = []\n\n    for i, mask in enumerate(masks):\n        # Create a bounding box from the mask\n        x, y, w, h = mask['bbox']\n        crop = original_img[y:y+h, x:x+w]\n\n        # In a real app, you'd send this to GPT-4o\n        # id_result = call_gpt4o_vision(crop)\n\n        pill_data.append({\n            \"id\": i,\n            \"position\": mask['point_coords'],\n            \"confidence\": mask['stability_score']\n        })\n\n    return pill_data\n\n# Example output structure\n# [\n#   {\"id\": 1, \"label\": \"Metformin\", \"color\": \"white\", \"shape\": \"oblong\"},\n#   {\"id\": 2, \"label\": \"Lisinopril\", \"color\": \"pink\", \"shape\": \"round\"}\n# ]\n```\n\nBy combining **SAM's spatial awareness** with **GPT-4o's semantic intelligence**, we've built a pipeline that understands both the *where* and the *what*. This multi-stage approach is much more robust than using a single \"end-to-end\" model which might hallucinate pill counts.\n\n**What's next for Visual-Pill-ID?**\n\nAre you working on Multimodal AI? Drop a comment below or share your thoughts on medical AI safety! 👇\n\n*If you enjoyed this technical deep dive, don't forget to visit [wellally.tech/blog](https://www.wellally.tech/blog) for more insights on high-performance AI architectures!*", "url": "https://wpnews.pro/news/visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam", "canonical_source": "https://dev.to/wellallytech/visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam-431j", "published_at": "2026-09-17 01:10:00+00:00", "updated_at": "2026-09-17 01:22:59.252438+00:00", "lang": "en", "topics": ["computer-vision", "large-language-models", "generative-ai", "ai-tools", "ai-products"], "entities": ["GPT-4o", "Segment Anything Model", "SAM", "OpenCV", "WellAlly Tech Blog", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam", "markdown": "https://wpnews.pro/news/visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam.md", "text": "https://wpnews.pro/news/visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam.txt", "jsonld": "https://wpnews.pro/news/visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam.jsonld"}}