Stop Guessing Your Meds: Building a Multimodal RAG Assistant with LLaVA and ChromaDB

Ever stared at a cryptic medicine bottle, wondering if it interacts with your morning coffee or that other pill you're taking? For the elderly or those with visual impairments, reading tiny labels on medication packaging is more than a nuisance—it’s a safety hazard.

In this tutorial, we are building a Medication Safety Assistant. This isn't just a simple OCR tool; we are implementing a Multimodal Retrieval-Augmented Generation (RAG) pipeline. We'll use LLaVA (Large Language-and-Vision Assistant) to "see" the medicine box, ChromaDB to store and retrieve detailed medical instructions, and Ollama to run everything locally and privately.

By the end of this guide, you'll understand how to bridge the gap between computer vision and structured knowledge retrieval to build life-saving AI applications. 🚀

Traditional RAG handles text. Multimodal RAG allows our system to process an image, convert the visual features into a query, and then fetch the relevant "truth" from a local vector database.

graph TD
    A[User Uploads Photo of Medicine] --> B[LLaVA via Ollama]
    B --> C{Identify Brand & Active Ingredients}
    C --> D[Generate Search Query]
    D --> E[(ChromaDB - Medical Knowledge)]
    E --> F[Retrieve Safety Guidelines & Dosage]
    F --> G[LLaVA Reasoning + Context]
    G --> H[Final Safety Instructions & UI]
    style B fill:#f96,stroke:#333,stroke-width:2px
    style E fill:#69f,stroke:#333,stroke-width:2px

To follow along, ensure you have the following installed:

LLaVA

, Ollama

, ChromaDB

, Gradio

.

pip install chromadb ollama gradio sentence-transformers

Before we can identify medicine, we need a "brain" containing the actual medical instructions. We'll use ChromaDB to store embeddings of medicine names and their corresponding contraindications.

import chromadb
from chromadb.utils import embedding_functions

client = chromadb.PersistentClient(path="./med_db")
default_ef = embedding_functions.DefaultEmbeddingFunction()
collection = client.get_or_create_collection(name="medicine_docs", embedding_function=default_ef)

med_data = [
    {"id": "001", "name": "Ibuprofen", "text": "Do not take with Aspirin. Max 1200mg/day. Avoid alcohol."},
    {"id": "002", "name": "Metformin", "text": "Used for Type 2 Diabetes. May cause stomach upset. Take with meals."},
]

for med in med_data:
    collection.add(
        documents=[med["text"]],
        metadatas=[{"name": med["name"]}],
        ids=[med["id"]]
    )

Now, we use the LLaVA model via Ollama. Its job is to look at the image and extract the medicine name. LLaVA is incredible because it understands spatial relationships and can read text even on curved surfaces like pill bottles.

import ollama

def identify_medicine(image_path):
    with open(image_path, 'rb') as f:
        img_data = f.read()

    response = ollama.generate(
        model='llava',
        prompt='Identify the brand name and the active ingredients of the medicine in this image. Output only the names.',
        images=[img_data]
    )
    return response['response'].strip()

This is where the magic happens. We take the visual output from LLaVA, query our vector database, and then pass that context back to the model to generate a safe, conversational answer.

def safety_assistant(image_path):
    identified_med = identify_medicine(image_path)
    print(f"Identified: {identified_med}")

    results = collection.query(
        query_texts=[identified_med],
        n_results=1
    )

    context = results['documents'][0][0] if results['documents'] else "No specific safety data found."

    final_prompt = f"""
    The user is asking about the medicine: {identified_med}.
    Based on the official medical database: {context}.
    Provide a concise safety warning and dosage instructions. 
    If there are no details found, warn the user to consult a doctor.
    """

    final_response = ollama.generate(model='llama3', prompt=final_prompt)
    return final_response['response']

While building a local prototype is great for learning, deploying production-grade AI in highly regulated sectors like healthcare requires more robust patterns.

For advanced architectural patterns, such as Hybrid Search (combining keyword and semantic search) and Agentic RAG workflows, I highly recommend exploring the deep-dive articles at wellally.tech/blog. They provide excellent resources on scaling these LLM implementations for enterprise use cases where reliability is non-negotiable.

Let’s wrap this in a user-friendly interface. Gradio allows us to create a functional UI in just a few lines of code.

import gradio as gr

def process_and_chat(image):
    image.save("temp_input.jpg")
    return safety_assistant("temp_input.jpg")

interface = gr.Interface(
    fn=process_and_chat,
    inputs=gr.Image(type="pil"),
    outputs="text",
    title="AI Medication Safety Assistant 💊",
    description="Upload a photo of your medicine packaging to get safety warnings and dosage info."
)

if __name__ == "__main__":
    interface.launch()

We just built a multimodal system that can potentially save lives! By combining LLaVA for vision and ChromaDB for verified knowledge, we've created a prototype that is both smart and grounded in reality.

What's next?

What do you think? Would you trust an AI assistant to read your meds, or are we still a few years away? Let me know in the comments! 👇

If you enjoyed this tutorial, don't forget to follow for more "Learning in Public" AI guides! 🚀💻