{"slug": "fridge-sloth-snap-your-fridge-get-a-grocery-list-and-healthy-meals-fully-offline", "title": "Fridge Sloth 🦥: snap your fridge, get a grocery list and healthy meals, fully offline on Gemma", "summary": "A developer built Fridge Sloth, an open-source app that runs Gemma 3 (4B) locally through Ollama to turn a single fridge photo into a grocery list and three healthy meal suggestions, with the image never leaving the user's machine. The app uses a zero-dependency Python server and a single HTML page, and enforces JSON-schema structured output on every model call. The developer found that sending the photo alongside a staples checklist caused the vision model to hallucinate items, and fixed it by splitting vision detection from a plain-code matching step.", "body_md": "*This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady](https://dev.to/challenges/hacktoberfest-week3-2026-10-19)*\n\n**Fridge Sloth** is for the moment you stand in front of the open fridge, door wide open, trying to remember what you need to buy and what on earth you could cook tonight.\n\nInstead, you take one photo. Fridge Sloth:\n\n**Who gets to be lazier?** Anyone who does the weekly shop and then stares at a half-full fridge wondering what's for dinner. You skip the mental inventory, the list-writing and the \"what can I even make with this?\" moment.\n\nEverything runs locally on **Gemma 3 (4B)** through **Ollama**, on my 6-year-old M1 MacBook. No account, no API key, and your fridge photo never leaves your machine.\n\nThe video is a real run of the app on my laptop with a public-domain USDA fridge photo ([source](<https://commons.wikimedia.org/wiki/File:Food_Safety_-_Cut_Waste_in_Refrigerators_(20200605-FSIS-LSC-0017).jpg>)). The two waits are sped up and labelled with their real time: **~37 s** for the scan and **~26 s** for the recipes. I recorded it before the speed-up described below; the same scan now takes **~20 s**. The voice-over was generated with **ElevenLabs**.\n\nSnap a photo of your fridge, get a **grocery list** and **three healthy meals** you can cook with what's already there\nRuns fully offline on [Gemma 3](https://ai.google.dev/gemma) through [Ollama](https://ollama.com). Your fridge photo never leaves your machine.\n\nBuilt for the [Hacktoberfest Open-Source AI Challenge, Week 3: Slow and Steady](https://dev.to/challenges/hacktoberfest-week3-2026-10-19).\n\n```\n┌──────────────┐   1. vision (no hints)   ┌──────────────────────┐\n│ fridge photo │ ───────────────────────> │ items actually seen  │\n└──────────────┘        Gemma 3           │ + amount left        │\n                                          └──────────┬───────────┘\n┌──────────────┐   2. match by name                  │\n│ your staples │ ────────────────────────────────────┤\n└──────────────┘     plain code, ~1 ms               │\n                                          ┌──────────▼───────────┐\n                                          │ grocery list         │\n                                          │ (missing / low)      │\n                                          └──────────┬───────────┘\n                     3. healthy recipes              │\n                          Gemma 3         ┌──────────▼───────────┐\n                                          │ 3 meals, ≤ 30 min    │\n                                          └──────────────────────┘\n```\n\n`plenty`, `some`, `low`)\nIt is It's deliberately tiny: a Python server that uses only the standard library (`http.server` + `urllib`), an optional tracing module, and one HTML page with vanilla JS. The only thing you install is Ollama:\n\n```\nbrew install ollama && ollama pull gemma3:4b\npython3 server.py   # open http://localhost:8000\n```\n\nThere are 35 tests, and Gemma is mocked in them, so they run in CI without a model.\n\n**Stack:** Gemma 3 4B (vision) on Ollama, a zero-dependency Python server, and a single HTML page. The page shrinks phone photos to 1024 px in the browser before sending them, which is plenty for Gemma and much faster.\n\n**Structured output everywhere.** Every call to Gemma passes a JSON schema in Ollama's `format` field, so the server always gets back data it can use (`{\"seen\": [{\"item\": \"eggs\", \"amount\": \"plenty\"}]}`), never prose to parse.\n\nMy first version sent the photo *together with* the staples list and asked \"which of these are in the fridge?\". On the USDA photo, Gemma confidently reported butter, tomatoes and apples as \"low\". None of them are in the picture. Handing a small vision model a checklist primes it to find the items on it.\n\nThe fix was to split the work:\n\nThat one change removed the invented items. At first, step 2 was a second, text-only Gemma call. It's now plain code, and the Sentry traces below are why.\n\nThe same rule, **never trust the model's claim, check it**, came back for the recipes. In one run a recipe listed a *banana* \"from your fridge\" when there was none. Now any recipe ingredient that wasn't actually seen is moved to the grocery list.\n\nA scan took about a minute, and I wanted to know *where* that minute went. Fridge Sloth has optional [Sentry Agent Tracing](https://docs.sentry.io/product/agents/getting-started/), off unless you set `SENTRY_DSN`:\n\n`invoke_agent` trace with a `gen_ai.chat` span per Gemma call, carrying the model, token counts and Ollama's own timings (model load, prompt processing, generation).`gen_ai.execute_tool` span: `dedupe_seen_items`, `match_staples_by_name` and `check_recipe_ingredients`. In Sentry's **1. The traces exposed a real bug.** On a messy example photo, two scans failed after about 4½ minutes. The trace showed why: a single vision call ran for **4.46 minutes**, and the next step never started. The 4B model was stuck in a loop, listing the same items until its JSON answer was cut off at about 10,000 characters.\n\nThe fix: the schema now allows at most 30 items, which Ollama enforces while generating; every answer is capped in length; and a cut-off answer fails fast with a clear message. The same photo then scanned in 1.15 minutes, and that trace showed the next problem.\n\n**2. The traces showed where to optimize.** The matching step, Gemma comparing two short lists of words, took **~25 s**, a third of every scan. I first tried asking Gemma only about the staples plain code couldn't place (\"is any of these here under another name?\"). It was faster but **wrong**: the 4B model over-claimed, deciding the spinach was the cheddar cheese. So matching is now plain code, with a strict rule: an item counts as a staple only if it *ends with* it. \"Cheddar cheese\" is cheese and \"free-range eggs\" are eggs, but \"tomato sauce\" is not tomatoes and \"milk chocolate\" is not milk. Missing a synonym (\"OJ\") only means buying something you already have; a false match means *not* buying something you need.\n\nThe result, on the same photos and staples:\n\n| Photo | Before | After | \n|---|---|---|\n| USDA (clear) | 51.6 s | **19.9 s** | \n| Door shelves | 52.9 s | **20.7 s** | \n| Messy | 68.8 s | **43.6 s** | \n\nA scan is now one Gemma call plus two checks that take about a millisecond each:\n\n**3. Generation is what takes the time.** A recipe request is about 800 tokens; nearly all of its ~30 s is Gemma generating the answer, with model loading and prompt reading taking a few seconds. Shorter answers, not a faster prompt, are what would speed it up next.\n\n**4. Checking my own setup.** The looping scans' errors never showed up as Sentry issues, even though their traces did. Sentry's usage stats explained why: the error events had been rejected as **too large**. By default, Sentry's Python SDK attaches each stack frame's local variables to error events, and here those held the prompt, Gemma's answer and the base64 photo, several times over. So errors from real scans were silently dropped, and smaller ones would have sent exactly the data I promised never to send. One setting (`include_local_variables=False`) fixed both, and I checked it end to end: the same error with a real photo now arrives in Sentry at 11 KB, with no prompt, answer or photo in it.\n\n**Privacy is kept,** in traces and error reports alike. Only metadata is sent: timings, token counts and how many items each check looked at. The photo, the prompts, Gemma's answers and even the food names never leave the machine, and [the tests](https://github.com/finallyjay/fridge-sloth/blob/main/tests/test_tracing.py) check exactly that for both.\n\nI also tried the 3× bigger model on the same photos, on the same laptop (before the matching speed-up):\n\n|  | 4B | 12B | \n|---|---|---|\n| Scan, clear photo | ~45 s | ~44 s | \n| Scan, messy photo | ~49 s | ~109 s | \n| Recipes | ~30 s | ~95 s | \n| Messy photo | missed the milk, called beer \"juice\" | saw beer, salsa and bread, but still missed the milk and eggs | \n| Recipes | decent | not better (a *cream bread pudding* ) | \n\n12B reads messy photos a little better, but it's about 3× slower on recipes and the recipes weren't better. **I kept 4B.** Switching is one environment variable (`FRIDGE_MODEL=gemma3:12b`) if your machine is faster than mine.\n\n`eleven_v4`), one line per moment in the video, stitched so it sounds like one take. The script is `scripts/narrate_demo.py`\n*Example photos are from Wikimedia Commons: USDA (public domain) and others under CC BY-SA. Full credits are in the repo.*", "url": "https://wpnews.pro/news/fridge-sloth-snap-your-fridge-get-a-grocery-list-and-healthy-meals-fully-offline", "canonical_source": "https://dev.to/finallyjay/fridge-sloth-snap-your-fridge-get-a-grocery-list-and-healthy-meals-fully-offline-on-gemma-2fkp", "published_at": "2026-10-24 13:38:31+00:00", "updated_at": "2026-10-24 13:42:39.555709+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "computer-vision", "ai-products"], "entities": ["Gemma 3", "Ollama", "Fridge Sloth", "Google", "ElevenLabs", "Hacktoberfest", "Sentry"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fridge-sloth-snap-your-fridge-get-a-grocery-list-and-healthy-meals-fully-offline", "markdown": "https://wpnews.pro/news/fridge-sloth-snap-your-fridge-get-a-grocery-list-and-healthy-meals-fully-offline.md", "text": "https://wpnews.pro/news/fridge-sloth-snap-your-fridge-get-a-grocery-list-and-healthy-meals-fully-offline.txt", "jsonld": "https://wpnews.pro/news/fridge-sloth-snap-your-fridge-get-a-grocery-list-and-healthy-meals-fully-offline.jsonld"}}