Stop Guessing Calories: Build a Multimodal Food Estimation Pipeline with GPT-4o & SAM

A developer has built a multimodal food estimation pipeline that combines Meta's Segment Anything Model (SAM) for precise food boundary detection with GPT-4o Vision for contextual analysis, then uses vector databases to retrieve verified nutritional data. The pipeline follows an 'Identify -> Analyze -> Match' flow, using SAM to isolate food items before querying GPT-4o, and PostgreSQL with pgvector for similarity search against a nutritional database. The approach aims to improve portion estimation accuracy in calorie tracking apps.

We’ve all been there: staring at a delicious plate of pasta, trying to figure out if it's 400 or 800 calories. Manual tracking is a chore, and standard apps often fail at portion estimation. But what if we could combine Computer Vision , Multimodal LLMs , and Vector Databases to build an automated nutritionist? In this tutorial, we are building a state-of-the-art Multimodal Food Estimation Pipeline . By leveraging the Segment Anything Model SAM for precise boundary detection and GPT-4o Vision for contextual analysis, we can bridge the gap between "looking at a photo" and "calculating nutritional density." Whether you're interested in AI-driven wellness , FastAPI development , or Multimodal RAG , this guide covers the full stack. The pipeline follows a sophisticated "Identify - Analyze - Match" flow. We don't just ask GPT-4o "what is this?"; we use SAM to isolate food items first to ensure the LLM focuses on the right pixels. php graph TD A User Uploads Image -- B{SAM Model} B -- |Segmentation| C Isolated Food Patches C -- D GPT-4o Vision API D -- |Item + Volume Est.| E Embedding Generation E -- F PostgreSQL + pgvector F -- |RAG Retrieval| G Verified Nutritional Data G -- H Final Response: Calories & Macros Before we dive in, make sure you have the following ready: pgvector extension enabledThe biggest challenge in food AI is overlapping items. Using Meta’s Segment Anything Model SAM , we can extract the exact mask of a food item, which helps in calculating the relative "area" occupied on the plate. python import torch from segment anything import sam model registry, SamPredictor import cv2 Load SAM model sam checkpoint = "sam vit h 4b8939.pth" model type = "vit h" sam = sam model registry model type checkpoint=sam checkpoint predictor = SamPredictor sam def get food masks image path : image = cv2.imread image path image = cv2.cvtColor image, cv2.COLOR BGR2RGB predictor.set image image In a real scenario, you'd use a prompt or automatic mask generator masks, scores, logits = predictor.predict point coords=None, point labels=None, multimask output=True, return masks Once we have the segmented image, we pass the original image and the mask hints to GPT-4o . We ask the model to act as a culinary expert to estimate the volume in grams/milliliters and identify the specific ingredients. python from openai import OpenAI from pydantic import BaseModel client = OpenAI class FoodAnalysis BaseModel : item name: str estimated weight g: float confidence score: float description: str def analyze food with gpt4o image url: str : response = client.beta.chat.completions.parse model="gpt-4o", messages= { "role": "user", "content": {"type": "text", "text": "Analyze the food in this image. Estimate the weight in grams for each identified item."}, {"type": "image url", "image url": {"url": image url}}, , } , response format=FoodAnalysis, return response.choices 0 .message.parsed LLMs can hallucinate calories. To ensure accuracy, we take the item name from GPT-4o, convert it into an embedding, and perform a similarity search against a verified nutritional database stored in PostgreSQL using pgvector . Pro Tip:For production-grade implementations and advanced patterns on scaling vector searches for health-tech, I highly recommend exploring the engineering deep-dives at WellAlly Tech Blog . They provide excellent resources on fine-tuning RAG pipelines for specialized domains. -- Search for the closest nutritional match SELECT food name, calories per 100g, protein, carbs, fats FROM nutritional db ORDER BY embedding <= embedding vector from gpt4o LIMIT 1; Now, let's wrap everything into a clean, high-performance API. python from fastapi import FastAPI, UploadFile, File import uvicorn app = FastAPI title="Vision-Calorie-Estimator" @app.post "/estimate-calories" async def estimate calories file: UploadFile = File ... : 1. Save uploaded file 2. Run SAM Segmentation 3. Call GPT-4o Vision 4. Query Vector DB for exact macros 5. Math: Weight / 100 Calories per 100g analysis = {"item": "Grilled Salmon", "calories": 450, "protein": "40g"} return {"status": "success", "data": analysis} if name == " main ": uvicorn.run app, host="0.0.0.0", port=8000 Building a multimodal pipeline is about orchestration. By combining the "eyes" of SAM and GPT-4o with the "memory" of a Vector Database, we’ve created a tool that is significantly more accurate than traditional calorie counters. What's next? If you enjoyed this build, don't forget to check out wellally.tech/blog https://www.wellally.tech/blog for more advanced tutorials on AI integration and full-stack development. Happy coding 🥑💻