Stop Guessing Calories: Build a Multimodal Food Estimation Pipeline with GPT-4o & SAM

We’ve all been there: staring at a delicious plate of pasta, trying to figure out if it's 400 or 800 calories. Manual tracking is a chore, and standard apps often fail at portion estimation. But what if we could combine Computer Vision, Multimodal LLMs, and Vector Databases to build an automated nutritionist?

In this tutorial, we are building a state-of-the-art Multimodal Food Estimation Pipeline. By leveraging the Segment Anything Model (SAM) for precise boundary detection and GPT-4o Vision for contextual analysis, we can bridge the gap between "looking at a photo" and "calculating nutritional density." Whether you're interested in AI-driven wellness, FastAPI development, or Multimodal RAG, this guide covers the full stack.

The pipeline follows a sophisticated "Identify -> Analyze -> Match" flow. We don't just ask GPT-4o "what is this?"; we use SAM to isolate food items first to ensure the LLM focuses on the right pixels.

graph TD
    A[User Uploads Image] --> B{SAM Model}
    B -->|Segmentation| C[Isolated Food Patches]
    C --> D[GPT-4o Vision API]
    D -->|Item + Volume Est.| E[Embedding Generation]
    E --> F[PostgreSQL + pgvector]
    F -->|RAG Retrieval| G[Verified Nutritional Data]
    G --> H[Final Response: Calories & Macros]

Before we dive in, make sure you have the following ready:

pgvector

extension enabledThe biggest challenge in food AI is overlapping items. Using Meta’s Segment Anything Model (SAM), we can extract the exact mask of a food item, which helps in calculating the relative "area" occupied on the plate.

import torch
from segment_anything import sam_model_registry, SamPredictor
import cv2

sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)

def get_food_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    masks, scores, logits = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks

Once we have the segmented image, we pass the original image and the mask hints to GPT-4o. We ask the model to act as a culinary expert to estimate the volume (in grams/milliliters) and identify the specific ingredients.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class FoodAnalysis(BaseModel):
    item_name: str
    estimated_weight_g: float
    confidence_score: float
    description: str

def analyze_food_with_gpt4o(image_url: str):
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze the food in this image. Estimate the weight in grams for each identified item."},
                    {"type": "image_url", "image_url": {"url": image_url}},
                ],
            }
        ],
        response_format=FoodAnalysis,
    )
    return response.choices[0].message.parsed

LLMs can hallucinate calories. To ensure accuracy, we take the item_name

from GPT-4o, convert it into an embedding, and perform a similarity search against a verified nutritional database stored in PostgreSQL using pgvector

.

Pro Tip:For production-grade implementations and advanced patterns on scaling vector searches for health-tech, I highly recommend exploring the engineering deep-dives at[WellAlly Tech Blog]. They provide excellent resources on fine-tuning RAG pipelines for specialized domains.

-- Search for the closest nutritional match
SELECT food_name, calories_per_100g, protein, carbs, fats
FROM nutritional_db
ORDER BY embedding <=> embedding_vector_from_gpt4o
LIMIT 1;

Now, let's wrap everything into a clean, high-performance API.

from fastapi import FastAPI, UploadFile, File
import uvicorn

app = FastAPI(title="Vision-Calorie-Estimator")

@app.post("/estimate-calories")
async def estimate_calories(file: UploadFile = File(...)):

    analysis = {"item": "Grilled Salmon", "calories": 450, "protein": "40g"}
    return {"status": "success", "data": analysis}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Building a multimodal pipeline is about orchestration. By combining the "eyes" of SAM and GPT-4o with the "memory" of a Vector Database, we’ve created a tool that is significantly more accurate than traditional calorie counters.

What's next?

If you enjoyed this build, don't forget to check out wellally.tech/blog for more advanced tutorials on AI integration and full-stack development.

Happy coding! 🥑💻