# Stop Guessing Calories: Build a Multimodal Food Estimation Pipeline with GPT-4o & SAM

> Source: <https://dev.to/beck_moulton/stop-guessing-calories-build-a-multimodal-food-estimation-pipeline-with-gpt-4o-sam-1ohb>
> Published: 2026-08-17 00:07:00+00:00

We’ve all been there: staring at a delicious plate of pasta, trying to figure out if it's 400 or 800 calories. Manual tracking is a chore, and standard apps often fail at portion estimation. But what if we could combine **Computer Vision**, **Multimodal LLMs**, and **Vector Databases** to build an automated nutritionist?

In this tutorial, we are building a state-of-the-art **Multimodal Food Estimation Pipeline**. By leveraging the **Segment Anything Model (SAM)** for precise boundary detection and **GPT-4o Vision** for contextual analysis, we can bridge the gap between "looking at a photo" and "calculating nutritional density." Whether you're interested in *AI-driven wellness*, *FastAPI development*, or *Multimodal RAG*, this guide covers the full stack.

The pipeline follows a sophisticated "Identify -> Analyze -> Match" flow. We don't just ask GPT-4o "what is this?"; we use SAM to isolate food items first to ensure the LLM focuses on the right pixels.

``` php
graph TD
    A[User Uploads Image] --> B{SAM Model}
    B -->|Segmentation| C[Isolated Food Patches]
    C --> D[GPT-4o Vision API]
    D -->|Item + Volume Est.| E[Embedding Generation]
    E --> F[PostgreSQL + pgvector]
    F -->|RAG Retrieval| G[Verified Nutritional Data]
    G --> H[Final Response: Calories & Macros]
```

Before we dive in, make sure you have the following ready:

`pgvector`

extension enabledThe biggest challenge in food AI is overlapping items. Using Meta’s **Segment Anything Model (SAM)**, we can extract the exact mask of a food item, which helps in calculating the relative "area" occupied on the plate.

``` python
import torch
from segment_anything import sam_model_registry, SamPredictor
import cv2

# Load SAM model
sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)

def get_food_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    # In a real scenario, you'd use a prompt or automatic mask generator
    masks, scores, logits = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks
```

Once we have the segmented image, we pass the original image and the mask hints to **GPT-4o**. We ask the model to act as a culinary expert to estimate the volume (in grams/milliliters) and identify the specific ingredients.

``` python
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class FoodAnalysis(BaseModel):
    item_name: str
    estimated_weight_g: float
    confidence_score: float
    description: str

def analyze_food_with_gpt4o(image_url: str):
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze the food in this image. Estimate the weight in grams for each identified item."},
                    {"type": "image_url", "image_url": {"url": image_url}},
                ],
            }
        ],
        response_format=FoodAnalysis,
    )
    return response.choices[0].message.parsed
```

LLMs can hallucinate calories. To ensure accuracy, we take the `item_name`

from GPT-4o, convert it into an embedding, and perform a similarity search against a verified nutritional database stored in **PostgreSQL** using `pgvector`

.

Pro Tip:For production-grade implementations and advanced patterns on scaling vector searches for health-tech, I highly recommend exploring the engineering deep-dives at[WellAlly Tech Blog]. They provide excellent resources on fine-tuning RAG pipelines for specialized domains.

```
-- Search for the closest nutritional match
SELECT food_name, calories_per_100g, protein, carbs, fats
FROM nutritional_db
ORDER BY embedding <=> embedding_vector_from_gpt4o
LIMIT 1;
```

Now, let's wrap everything into a clean, high-performance API.

``` python
from fastapi import FastAPI, UploadFile, File
import uvicorn

app = FastAPI(title="Vision-Calorie-Estimator")

@app.post("/estimate-calories")
async def estimate_calories(file: UploadFile = File(...)):
    # 1. Save uploaded file
    # 2. Run SAM Segmentation
    # 3. Call GPT-4o Vision
    # 4. Query Vector DB for exact macros
    # 5. Math: (Weight / 100) * Calories_per_100g

    analysis = {"item": "Grilled Salmon", "calories": 450, "protein": "40g"}
    return {"status": "success", "data": analysis}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

Building a multimodal pipeline is about orchestration. By combining the "eyes" of SAM and GPT-4o with the "memory" of a Vector Database, we’ve created a tool that is significantly more accurate than traditional calorie counters.

**What's next?**

If you enjoyed this build, don't forget to check out [wellally.tech/blog](https://www.wellally.tech/blog) for more advanced tutorials on AI integration and full-stack development.

Happy coding! 🥑💻