Visual-Pill-ID: Building an AI Pharmacist with GPT-4o and SAM 💊

Ever stared at a handful of loose pills and wondered, "Wait, was the blue one for my allergies or my blood pressure?" 😅 You're not alone. Medication errors are a massive global health challenge.

In this tutorial, we are building Visual-Pill-ID, a cutting-edge computer vision pipeline that solves the "multi-pill confusion" problem. By combining the geometric precision of the Segment Anything Model (SAM) with the multimodal reasoning of GPT-4o, we can transform a messy photo of mixed medication into a structured, verified prescription list.

We'll be diving deep into Computer Vision, Instance Segmentation, and Multimodal LLMs to create a production-ready OCR and identification system.

The biggest challenge in pill identification isn't just "seeing" the pill; it's isolating it from a crowded background and understanding its specific markings. Our pipeline follows a "Segment-then-Analyze" pattern.

graph TD
    A[Raw Image of Multiple Pills] --> B[SAM: Segment Anything Model]
    B --> C{Instance Masks}
    C --> D[OpenCV: Crop & Preprocess]
    D --> E[GPT-4o Vision: Multi-modal Analysis]
    E --> F[OCR & Pill Identification]
    F --> G[Prescription Validation & Safety Logic]
    G --> H[Final Structured JSON Output]

To follow along, you’ll need:

vit_h or vit_b checkpoints. Traditional bounding boxes often overlap when pills are touching. We need Instance Segmentation to get the exact pixels of each pill.

import numpy as np
import torch
import cv2
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

device = "cuda" if torch.cuda.is_available() else "cpu"
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to(device)

mask_generator = SamAutomaticMaskGenerator(sam)

def get_pill_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    masks = mask_generator.generate(image)

    filtered_masks = [m for m in masks if m['area'] > 500]
    return filtered_masks, image

print(f"🚀 Detected {len(get_pill_masks('pills.jpg')[0])} potential pills!")

Once we have the masks, we crop each pill. However, sending 10 separate images to GPT-4o is expensive. Instead, we create a "Collage of Interest" or send them in a structured batch. GPT-4o is incredible at OCR on curved surfaces, which is typical for medication.

import base64
import requests

def encode_image(image_np):
    _, buffer = cv2.imencode('.jpg', image_np)
    return base64.b64encode(buffer).decode('utf-8')

def identify_pills(pill_crops):
    prompt_content = [
        {"type": "text", "text": "Identify each pill in these images. Extract markings, color, and shape. Compare with standard medical databases."}
    ]

    for crop in pill_crops:
        base64_image = encode_image(crop)
        prompt_content.append({
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
        })

It's not enough to know it's "Ibuprofen 200mg." We need to know if it matches the user's prescription. By feeding the OCR text from the medicine bottle (also captured in the pipeline) and the identified pills into GPT-4o, we can perform a cross-check.

While this DIY pipeline is great for prototyping, building a production-grade medical vision system requires handling edge cases like glare on blister packs and HIPAA-compliant data handling.

For more production-ready examples and advanced patterns on integrating LLMs with specialized computer vision models, check out the detailed guides at WellAlly Tech Blog. They cover deep-dives into AI reliability that are crucial for healthcare applications. 🥑

Here is a snippet showing how we combine the SAM mask with an OpenCV crop to feed the vision model:

def process_pipeline(img_path):
    masks, original_img = get_pill_masks(img_path)
    pill_data = []

    for i, mask in enumerate(masks):
        x, y, w, h = mask['bbox']
        crop = original_img[y:y+h, x:x+w]


        pill_data.append({
            "id": i,
            "position": mask['point_coords'],
            "confidence": mask['stability_score']
        })

    return pill_data

By combining SAM's spatial awareness with GPT-4o's semantic intelligence, we've built a pipeline that understands both the where and the what. This multi-stage approach is much more robust than using a single "end-to-end" model which might hallucinate pill counts.

What's next for Visual-Pill-ID?

Are you working on Multimodal AI? Drop a comment below or share your thoughts on medical AI safety! 👇

If you enjoyed this technical deep dive, don't forget to visit wellally.tech/blog for more insights on high-performance AI architectures!