Visual-Pill-ID: Building an AI Pharmacist with GPT-4o and SAM 💊

A developer built Visual-Pill-ID, a computer vision pipeline that combines the Segment Anything Model (SAM) for instance segmentation with GPT-4o's multimodal reasoning to identify loose pills from a single photo. The system segments each pill, crops and preprocesses it with OpenCV, then uses GPT-4o to perform OCR on curved surfaces and cross-check identified medications against the user's prescription. The developer notes that production-grade medical vision systems would still need to handle edge cases like glare on blister packs and HIPAA-compliant data handling.

Ever stared at a handful of loose pills and wondered, "Wait, was the blue one for my allergies or my blood pressure?" 😅 You're not alone. Medication errors are a massive global health challenge. In this tutorial, we are building Visual-Pill-ID , a cutting-edge computer vision pipeline that solves the "multi-pill confusion" problem. By combining the geometric precision of the Segment Anything Model SAM with the multimodal reasoning of GPT-4o , we can transform a messy photo of mixed medication into a structured, verified prescription list. We'll be diving deep into Computer Vision , Instance Segmentation , and Multimodal LLMs to create a production-ready OCR and identification system. The biggest challenge in pill identification isn't just "seeing" the pill; it's isolating it from a crowded background and understanding its specific markings. Our pipeline follows a "Segment-then-Analyze" pattern. php graph TD A Raw Image of Multiple Pills -- B SAM: Segment Anything Model B -- C{Instance Masks} C -- D OpenCV: Crop & Preprocess D -- E GPT-4o Vision: Multi-modal Analysis E -- F OCR & Pill Identification F -- G Prescription Validation & Safety Logic G -- H Final Structured JSON Output To follow along, you’ll need: vit h or vit b checkpoints. Traditional bounding boxes often overlap when pills are touching. We need Instance Segmentation to get the exact pixels of each pill. python import numpy as np import torch import cv2 from segment anything import sam model registry, SamAutomaticMaskGenerator Load the SAM model device = "cuda" if torch.cuda.is available else "cpu" sam = sam model registry "vit h" checkpoint="sam vit h 4b8939.pth" .to device Generate masks automatically mask generator = SamAutomaticMaskGenerator sam def get pill masks image path : image = cv2.imread image path image = cv2.cvtColor image, cv2.COLOR BGR2RGB masks = mask generator.generate image Filter by area to remove tiny artifacts filtered masks = m for m in masks if m 'area' 500 return filtered masks, image print f"🚀 Detected {len get pill masks 'pills.jpg' 0 } potential pills " Once we have the masks, we crop each pill. However, sending 10 separate images to GPT-4o is expensive. Instead, we create a "Collage of Interest" or send them in a structured batch. GPT-4o is incredible at OCR on curved surfaces , which is typical for medication. python import base64 import requests def encode image image np : , buffer = cv2.imencode '.jpg', image np return base64.b64encode buffer .decode 'utf-8' def identify pills pill crops : Constructing the multimodal prompt prompt content = {"type": "text", "text": "Identify each pill in these images. Extract markings, color, and shape. Compare with standard medical databases."} for crop in pill crops: base64 image = encode image crop prompt content.append { "type": "image url", "image url": {"url": f"data:image/jpeg;base64,{base64 image}"} } Call GPT-4o API implementation details... Return structured JSON It's not enough to know it's "Ibuprofen 200mg." We need to know if it matches the user's prescription. By feeding the OCR text from the medicine bottle also captured in the pipeline and the identified pills into GPT-4o, we can perform a cross-check. While this DIY pipeline is great for prototyping, building a production-grade medical vision system requires handling edge cases like glare on blister packs and HIPAA-compliant data handling. For more production-ready examples and advanced patterns on integrating LLMs with specialized computer vision models, check out the detailed guides at WellAlly Tech Blog https://www.wellally.tech/blog . They cover deep-dives into AI reliability that are crucial for healthcare applications. 🥑 Here is a snippet showing how we combine the SAM mask with an OpenCV crop to feed the vision model: python def process pipeline img path : masks, original img = get pill masks img path pill data = for i, mask in enumerate masks : Create a bounding box from the mask x, y, w, h = mask 'bbox' crop = original img y:y+h, x:x+w In a real app, you'd send this to GPT-4o id result = call gpt4o vision crop pill data.append { "id": i, "position": mask 'point coords' , "confidence": mask 'stability score' } return pill data Example output structure {"id": 1, "label": "Metformin", "color": "white", "shape": "oblong"}, {"id": 2, "label": "Lisinopril", "color": "pink", "shape": "round"} By combining SAM's spatial awareness with GPT-4o's semantic intelligence , we've built a pipeline that understands both the where and the what . This multi-stage approach is much more robust than using a single "end-to-end" model which might hallucinate pill counts. What's next for Visual-Pill-ID? Are you working on Multimodal AI? Drop a comment below or share your thoughts on medical AI safety 👇 If you enjoyed this technical deep dive, don't forget to visit wellally.tech/blog https://www.wellally.tech/blog for more insights on high-performance AI architectures