From Pixels to Prescriptions: Building an AI Pharmacist with YOLOv10 and RAG

We’ve all been there: staring at a cluttered medicine cabinet, holding two different blister packs, and wondering, "Can I take these together?" In the age of AI, "googling it" isn't just slow—it's potentially dangerous.

Today, we are building a Smart Home Medicine Assistant. By combining computer vision for healthcare with YOLOv10, and leveraging Retrieval-Augmented Generation (RAG) through Function Calling, we can create a system that identifies medicine packaging and performs real-time Drug-Drug Interaction (DDI) risk detection. Whether you are interested in AI-driven healthcare automation or advanced object detection, this guide will show you how to turn a camera feed into a life-saving advisor.

Building a reliable medical assistant requires more than just a chatbox. We need a robust pipeline that can identify physical objects and cross-reference them with validated medical databases.

graph TD
    A[User Uploads Image/Video] --> B{YOLOv10 Inference}
    B -->|Detected Label| C[Drug Identification Agent]
    C --> D{Redis History Check}
    D -->|Existing Meds Found| E[LLM Function Calling]
    E --> F[DrugBank API / Knowledge Base]
    F --> G[Conflict Detection Logic]
    G --> H[Final Safety Report & Guidance]
    H --> I[Store Current Med in Redis]

To follow this tutorial, you'll need the following stack:

YOLOv10 is a game-changer because it eliminates the need for Non-Maximum Suppression (NMS), significantly reducing latency. This is perfect for edge devices like a smart mirror or a mobile app.

from ultralytics import YOLOv10

model = YOLOv10('weights/yolov10n_medicine.pt')

def identify_medication(image_path):
    results = model.predict(source=image_path, conf=0.25)
    detected_drugs = []

    for result in results:
        for box in result.boxes:
            label = model.names[int(box.cls)]
            detected_drugs.append(label)

    return list(set(detected_drugs)) # Return unique meds found

A single pill doesn't tell the whole story. To detect Drug-Drug Interactions (DDI), the system needs to remember what you’ve already scanned or what you are currently taking. We use Redis as a fast, volatile memory for the "current session."

import redis

cache = redis.Redis(host='localhost', port=6379, db=0)

def sync_medication_history(user_id, new_drug):
    history = cache.get(user_id)
    meds = history.decode('utf-8').split(',') if history else []

    if new_drug not in meds:
        meds.append(new_drug)
        cache.set(user_id, ",".join(meds))

    return meds

Now for the "brain." We use OpenAI's Function Calling to bridge the gap between the vision model and the DrugBank API. Instead of the LLM "hallucinating" side effects, it fetches real data.

tools = [
    {
        "type": "function",
        "function": {
            "name": "check_drug_interaction",
            "description": "Checks for adverse interactions between two or more drugs.",
            "parameters": {
                "type": "object",
                "properties": {
                    "drugs": {"type": "array", "items": {"type": "string"}}
                },
                "required": ["drugs"]
            }
        }
    }
]

def check_drug_interaction(drugs):
    interactions = call_drugbank_api(drugs)
    return interactions

Building a proof-of-concept is easy, but deploying AI in a healthcare context requires strict adherence to safety and data privacy patterns.

💡

Looking for deeper insights?For advanced patterns on securing medical data in RAG pipelines and optimizing Vision Transformers for mobile, check out the deep-dive articles at. They cover production-grade AI implementations that go beyond the basics of this tutorial.[WellAlly Tech Blog]

Finally, we wrap everything in a FastAPI endpoint that takes an image and returns a safety score.

from fastapi import FastAPI, UploadFile

app = FastAPI()

@app.post("/scan-medication")
async def scan_medication(user_id: str, file: UploadFile):
    drug_names = identify_medication(file.file)

    all_meds = []
    for drug in drug_names:
        all_meds = sync_medication_history(user_id, drug)

    if len(all_meds) > 1:
        report = agent.run(f"Check interactions for these drugs: {all_meds}")
        return {"status": "warning", "data": report}

    return {"status": "safe", "detected": drug_names}

By combining YOLOv10 for lightning-fast recognition and RAG/Function Calling for grounded medical knowledge, we've built a prototype that solves a real-world problem. However, remember: AI is an assistant, not a replacement for a doctor.

What’s next? You could expand this by adding:

Did you find this helpful? Drop a comment below if you have questions about the YOLOv10 training process or how to structure your RAG medical knowledge base! 🥑🚀

Stay curious, keep building.