# Visual-Pill-ID: Building an AI Pharmacist with GPT-4o and SAM 💊

> Source: <https://dev.to/wellallytech/visual-pill-id-building-an-ai-pharmacist-with-gpt-4o-and-sam-431j>
> Published: 2026-09-17 01:10:00+00:00

Ever stared at a handful of loose pills and wondered, "Wait, was the blue one for my allergies or my blood pressure?" 😅 You're not alone. Medication errors are a massive global health challenge.

In this tutorial, we are building **Visual-Pill-ID**, a cutting-edge computer vision pipeline that solves the "multi-pill confusion" problem. By combining the geometric precision of the **Segment Anything Model (SAM)** with the multimodal reasoning of **GPT-4o**, we can transform a messy photo of mixed medication into a structured, verified prescription list.

We'll be diving deep into **Computer Vision**, **Instance Segmentation**, and **Multimodal LLMs** to create a production-ready OCR and identification system.

The biggest challenge in pill identification isn't just "seeing" the pill; it's isolating it from a crowded background and understanding its specific markings. Our pipeline follows a "Segment-then-Analyze" pattern.

``` php
graph TD
    A[Raw Image of Multiple Pills] --> B[SAM: Segment Anything Model]
    B --> C{Instance Masks}
    C --> D[OpenCV: Crop & Preprocess]
    D --> E[GPT-4o Vision: Multi-modal Analysis]
    E --> F[OCR & Pill Identification]
    F --> G[Prescription Validation & Safety Logic]
    G --> H[Final Structured JSON Output]
```

To follow along, you’ll need:

`vit_h` or `vit_b` checkpoints.
Traditional bounding boxes often overlap when pills are touching. We need **Instance Segmentation** to get the exact pixels of each pill.

``` python
import numpy as np
import torch
import cv2
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

# Load the SAM model
device = "cuda" if torch.cuda.is_available() else "cpu"
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to(device)

# Generate masks automatically
mask_generator = SamAutomaticMaskGenerator(sam)

def get_pill_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    masks = mask_generator.generate(image)

    # Filter by area to remove tiny artifacts
    filtered_masks = [m for m in masks if m['area'] > 500]
    return filtered_masks, image

print(f"🚀 Detected {len(get_pill_masks('pills.jpg')[0])} potential pills!")
```

Once we have the masks, we crop each pill. However, sending 10 separate images to GPT-4o is expensive. Instead, we create a "Collage of Interest" or send them in a structured batch. GPT-4o is incredible at **OCR on curved surfaces**, which is typical for medication.

``` python
import base64
import requests

def encode_image(image_np):
    _, buffer = cv2.imencode('.jpg', image_np)
    return base64.b64encode(buffer).decode('utf-8')

def identify_pills(pill_crops):
    # Constructing the multimodal prompt
    prompt_content = [
        {"type": "text", "text": "Identify each pill in these images. Extract markings, color, and shape. Compare with standard medical databases."}
    ]

    for crop in pill_crops:
        base64_image = encode_image(crop)
        prompt_content.append({
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
        })

    # Call GPT-4o
    # API implementation details...
    # Return structured JSON
```

It's not enough to know it's "Ibuprofen 200mg." We need to know if it matches the user's prescription. By feeding the OCR text from the medicine bottle (also captured in the pipeline) and the identified pills into GPT-4o, we can perform a cross-check.

While this DIY pipeline is great for prototyping, building a production-grade medical vision system requires handling edge cases like glare on blister packs and HIPAA-compliant data handling.

For more production-ready examples and advanced patterns on integrating LLMs with specialized computer vision models, check out the detailed guides at **[WellAlly Tech Blog](https://www.wellally.tech/blog)**. They cover deep-dives into AI reliability that are crucial for healthcare applications. 🥑

Here is a snippet showing how we combine the SAM mask with an OpenCV crop to feed the vision model:

``` python
def process_pipeline(img_path):
    masks, original_img = get_pill_masks(img_path)
    pill_data = []

    for i, mask in enumerate(masks):
        # Create a bounding box from the mask
        x, y, w, h = mask['bbox']
        crop = original_img[y:y+h, x:x+w]

        # In a real app, you'd send this to GPT-4o
        # id_result = call_gpt4o_vision(crop)

        pill_data.append({
            "id": i,
            "position": mask['point_coords'],
            "confidence": mask['stability_score']
        })

    return pill_data

# Example output structure
# [
#   {"id": 1, "label": "Metformin", "color": "white", "shape": "oblong"},
#   {"id": 2, "label": "Lisinopril", "color": "pink", "shape": "round"}
# ]
```

By combining **SAM's spatial awareness** with **GPT-4o's semantic intelligence**, we've built a pipeline that understands both the *where* and the *what*. This multi-stage approach is much more robust than using a single "end-to-end" model which might hallucinate pill counts.

**What's next for Visual-Pill-ID?**

Are you working on Multimodal AI? Drop a comment below or share your thoughts on medical AI safety! 👇

*If you enjoyed this technical deep dive, don't forget to visit [wellally.tech/blog](https://www.wellally.tech/blog) for more insights on high-performance AI architectures!*