AI-Powered Calorie Counting: Mastering GPT-4o Vision and SAM for Automated Nutrition Tracking

Let’s be honest: manual diet tracking is a chore that almost nobody finishes. We start with good intentions, but typing "150g of grilled chicken" and "half a cup of brown rice" into an app every day is a recipe for burnout. But what if you could just snap a photo and let Multimodal AI do the heavy lifting? 📸

In this tutorial, we are building a production-ready automated nutrition logging system. We will combine the surgical precision of the Segment Anything Model (SAM) with the reasoning power of GPT-4o Vision. By the end of this post, you'll know how to transform raw pixels into a structured JSON of calories, macros, and portion sizes using FastAPI and Pydantic. We'll cover key concepts in Image Segmentation, Computer Vision, and LLM Structured Outputs.

To get accurate results, we can't just toss a messy photo at an LLM and hope for the best. We need a pipeline that identifies individual food items, isolates them, and then performs a multi-step inference.

graph TD
    A[User Uploads Food Image] --> B[FastAPI Backend]
    B --> C[SAM: Segment Anything Model]
    C --> D[Generate Individual Food Masks]
    D --> E[GPT-4o Vision: Multi-crop Analysis]
    E --> F[Pydantic Validation]
    F --> G[Structured Nutrition Report]
    G --> H[User Dashboard]

To follow along, you'll need:

The secret to a reliable AI system is Structured Output. We don't want a "chatty" response; we want data our database can consume. We'll use Pydantic

to define exactly what a "Meal" looks like.

from pydantic import BaseModel, Field
from typing import List

class FoodItem(BaseModel):
    name: str = Field(description="Name of the food item")
    estimated_weight_g: float = Field(description="Weight in grams")
    calories: int = Field(description="Total calories")
    protein_g: float = Field(description="Protein content in grams")
    carbs_g: float = Field(description="Carbohydrate content in grams")
    fat_g: float = Field(description="Fat content in grams")
    confidence_score: float = Field(description="AI's confidence in this estimation (0-1)")

class NutritionReport(BaseModel):
    items: List[FoodItem]
    total_calories: int
    summary: str

Using Meta's Segment Anything Model (SAM) allows us to identify the boundaries of different foods on a plate. This prevents GPT-4o from getting confused by background noise or overlapping items.

import numpy as np
from segment_anything import sam_model_registry, SamPredictor

sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)

def get_food_masks(image_np):
    predictor.set_image(image_np)
    masks, scores, logits = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks

Now for the magic. We send the original image and the segmented metadata to GPT-4o. We use a specific system prompt to force it to act as a professional nutritionist and volume estimator.

Pro Tip: For even better accuracy, check out the advanced prompt engineering patterns at[wellally.tech/blog]. They have fantastic resources on handling high-variance visual data in production environments. 🥑

import openai
from fastapi import FastAPI, UploadFile

app = FastAPI()
client = openai.OpenAI()

async def analyze_nutrition(image_url: str):
    response = client.beta.chat.completions.parse(
        model="gpt-4o-2024-08-06",
        messages=[
            {
                "role": "system", 
                "content": "You are a professional nutritionist. Analyze the image and provide precise weight and nutritional estimations."
            },
            {
                "role": "user", 
                "content": [
                    {"type": "text", "text": "Analyze the food in this image."},
                    {"type": "image_url", "image_url": {"url": image_url}}
                ]
            }
        ],
        response_format=NutritionReport,
    )
    return response.choices[0].message.parsed

Finally, we wrap everything in a FastAPI endpoint. This endpoint takes an image, processes it through our pipeline, and returns the structured nutrition data.

@app.post("/track-meal")
async def track_meal(file: UploadFile):
    content = await file.read()


    image_url = f"data:image/jpeg;base64,{base64_encode(content)}"
    report = await analyze_nutrition(image_url)

    return {
        "status": "success",
        "data": report
    }

Building a demo is easy, but scaling it requires handling edge cases like low lighting, blurry photos, and overlapping food items. For more in-depth guides on deploying these multimodal models and optimizing latency, I highly recommend checking out the ** WellAlly Tech Blog**. They dive deep into production-grade AI patterns that go beyond the basic API calls.

We’ve moved past the era of typing out calories. By combining SAM for spatial awareness and GPT-4o for nutritional reasoning, we can build tools that actually help people live healthier lives without the friction of manual entry.

What's next? You could extend this by adding:

Happy coding! 🚀 If you enjoyed this, drop a comment below and let me know what AI project you're working on!