AI-Powered Calorie Counting: Mastering GPT-4o Vision and SAM for Automated Nutrition Tracking

A developer built a production-ready automated nutrition logging system combining Meta's Segment Anything Model (SAM) with GPT-4o Vision, using FastAPI and Pydantic to transform food photos into structured JSON reports of calories, macros, and portion sizes.

Let’s be honest: manual diet tracking is a chore that almost nobody finishes. We start with good intentions, but typing "150g of grilled chicken" and "half a cup of brown rice" into an app every day is a recipe for burnout. But what if you could just snap a photo and let Multimodal AI do the heavy lifting? 📸 In this tutorial, we are building a production-ready automated nutrition logging system. We will combine the surgical precision of the Segment Anything Model SAM with the reasoning power of GPT-4o Vision . By the end of this post, you'll know how to transform raw pixels into a structured JSON of calories, macros, and portion sizes using FastAPI and Pydantic . We'll cover key concepts in Image Segmentation , Computer Vision , and LLM Structured Outputs . To get accurate results, we can't just toss a messy photo at an LLM and hope for the best. We need a pipeline that identifies individual food items, isolates them, and then performs a multi-step inference. php graph TD A User Uploads Food Image -- B FastAPI Backend B -- C SAM: Segment Anything Model C -- D Generate Individual Food Masks D -- E GPT-4o Vision: Multi-crop Analysis E -- F Pydantic Validation F -- G Structured Nutrition Report G -- H User Dashboard To follow along, you'll need: The secret to a reliable AI system is Structured Output . We don't want a "chatty" response; we want data our database can consume. We'll use Pydantic to define exactly what a "Meal" looks like. python from pydantic import BaseModel, Field from typing import List class FoodItem BaseModel : name: str = Field description="Name of the food item" estimated weight g: float = Field description="Weight in grams" calories: int = Field description="Total calories" protein g: float = Field description="Protein content in grams" carbs g: float = Field description="Carbohydrate content in grams" fat g: float = Field description="Fat content in grams" confidence score: float = Field description="AI's confidence in this estimation 0-1 " class NutritionReport BaseModel : items: List FoodItem total calories: int summary: str Using Meta's Segment Anything Model SAM allows us to identify the boundaries of different foods on a plate. This prevents GPT-4o from getting confused by background noise or overlapping items. python import numpy as np from segment anything import sam model registry, SamPredictor Initialize SAM Assuming you have the checkpoint file sam checkpoint = "sam vit h 4b8939.pth" model type = "vit h" sam = sam model registry model type checkpoint=sam checkpoint predictor = SamPredictor sam def get food masks image np : predictor.set image image np In a production app, you might use a prompt or automatic mask generation masks, scores, logits = predictor.predict point coords=None, point labels=None, multimask output=True, return masks Now for the magic. We send the original image and the segmented metadata to GPT-4o. We use a specific system prompt to force it to act as a professional nutritionist and volume estimator. Pro Tip: For even better accuracy, check out the advanced prompt engineering patterns at wellally.tech/blog . They have fantastic resources on handling high-variance visual data in production environments. 🥑 python import openai from fastapi import FastAPI, UploadFile app = FastAPI client = openai.OpenAI async def analyze nutrition image url: str : response = client.beta.chat.completions.parse model="gpt-4o-2024-08-06", messages= { "role": "system", "content": "You are a professional nutritionist. Analyze the image and provide precise weight and nutritional estimations." }, { "role": "user", "content": {"type": "text", "text": "Analyze the food in this image."}, {"type": "image url", "image url": {"url": image url}} } , response format=NutritionReport, return response.choices 0 .message.parsed Finally, we wrap everything in a FastAPI endpoint. This endpoint takes an image, processes it through our pipeline, and returns the structured nutrition data. python @app.post "/track-meal" async def track meal file: UploadFile : 1. Read and process the image content = await file.read 2. Optional Run SAM to generate crops for better precision ... SAM logic here ... 3. Call GPT-4o Vision for Nutrition Analysis In a real app, you'd upload the image to a cloud bucket first image url = f"data:image/jpeg;base64,{base64 encode content }" report = await analyze nutrition image url return { "status": "success", "data": report } Building a demo is easy, but scaling it requires handling edge cases like low lighting, blurry photos, and overlapping food items. For more in-depth guides on deploying these multimodal models and optimizing latency, I highly recommend checking out the WellAlly Tech Blog . They dive deep into production-grade AI patterns that go beyond the basic API calls. We’ve moved past the era of typing out calories. By combining SAM for spatial awareness and GPT-4o for nutritional reasoning, we can build tools that actually help people live healthier lives without the friction of manual entry. What's next? You could extend this by adding: Happy coding 🚀 If you enjoyed this, drop a comment below and let me know what AI project you're working on