Fridge Sloth 🦥: snap your fridge, get a grocery list and healthy meals, fully offline on Gemma

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

Fridge Sloth is for the moment you stand in front of the open fridge, door wide open, trying to remember what you need to buy and what on earth you could cook tonight.

Instead, you take one photo. Fridge Sloth:

Who gets to be lazier? Anyone who does the weekly shop and then stares at a half-full fridge wondering what's for dinner. You skip the mental inventory, the list-writing and the "what can I even make with this?" moment.

Everything runs locally on Gemma 3 (4B) through Ollama, on my 6-year-old M1 MacBook. No account, no API key, and your fridge photo never leaves your machine.

The video is a real run of the app on my laptop with a public-domain USDA fridge photo (source). The two waits are sped up and labelled with their real time: ~37 s for the scan and ~26 s for the recipes. I recorded it before the speed-up described below; the same scan now takes ~20 s. The voice-over was generated with ElevenLabs.

Snap a photo of your fridge, get a grocery list and three healthy meals you can cook with what's already there Runs fully offline on Gemma 3 through Ollama. Your fridge photo never leaves your machine.

Built for the Hacktoberfest Open-Source AI Challenge, Week 3: Slow and Steady.

┌──────────────┐   1. vision (no hints)   ┌──────────────────────┐
│ fridge photo │ ───────────────────────> │ items actually seen  │
└──────────────┘        Gemma 3           │ + amount left        │
                                          └──────────┬───────────┘
┌──────────────┐   2. match by name                  │
│ your staples │ ────────────────────────────────────┤
└──────────────┘     plain code, ~1 ms               │
                                          ┌──────────▼───────────┐
                                          │ grocery list         │
                                          │ (missing / low)      │
                                          └──────────┬───────────┘
                     3. healthy recipes              │
                          Gemma 3         ┌──────────▼───────────┐
                                          │ 3 meals, ≤ 30 min    │
                                          └──────────────────────┘

plenty, some, low) It is It's deliberately tiny: a Python server that uses only the standard library (http.server + urllib), an optional tracing module, and one HTML page with vanilla JS. The only thing you install is Ollama:

brew install ollama && ollama pull gemma3:4b
python3 server.py   # open http://localhost:8000

There are 35 tests, and Gemma is mocked in them, so they run in CI without a model.

Stack: Gemma 3 4B (vision) on Ollama, a zero-dependency Python server, and a single HTML page. The page shrinks phone photos to 1024 px in the browser before sending them, which is plenty for Gemma and much faster.

Structured output everywhere. Every call to Gemma passes a JSON schema in Ollama's format field, so the server always gets back data it can use ({"seen": [{"item": "eggs", "amount": "plenty"}]}), never prose to parse.

My first version sent the photo together with the staples list and asked "which of these are in the fridge?". On the USDA photo, Gemma confidently reported butter, tomatoes and apples as "low". None of them are in the picture. Handing a small vision model a checklist primes it to find the items on it.

The fix was to split the work:

That one change removed the invented items. At first, step 2 was a second, text-only Gemma call. It's now plain code, and the Sentry traces below are why.

The same rule, never trust the model's claim, check it, came back for the recipes. In one run a recipe listed a banana "from your fridge" when there was none. Now any recipe ingredient that wasn't actually seen is moved to the grocery list.

A scan took about a minute, and I wanted to know where that minute went. Fridge Sloth has optional Sentry Agent Tracing, off unless you set SENTRY_DSN:

invoke_agent trace with a gen_ai.chat span per Gemma call, carrying the model, token counts and Ollama's own timings (model load, prompt processing, generation).gen_ai.execute_tool span: dedupe_seen_items, match_staples_by_name and check_recipe_ingredients. In Sentry's 1. The traces exposed a real bug. On a messy example photo, two scans failed after about 4½ minutes. The trace showed why: a single vision call ran for 4.46 minutes, and the next step never started. The 4B model was stuck in a loop, listing the same items until its JSON answer was cut off at about 10,000 characters.

The fix: the schema now allows at most 30 items, which Ollama enforces while generating; every answer is capped in length; and a cut-off answer fails fast with a clear message. The same photo then scanned in 1.15 minutes, and that trace showed the next problem.

2. The traces showed where to optimize. The matching step, Gemma comparing two short lists of words, took ~25 s, a third of every scan. I first tried asking Gemma only about the staples plain code couldn't place ("is any of these here under another name?"). It was faster but wrong: the 4B model over-claimed, deciding the spinach was the cheddar cheese. So matching is now plain code, with a strict rule: an item counts as a staple only if it ends with it. "Cheddar cheese" is cheese and "free-range eggs" are eggs, but "tomato sauce" is not tomatoes and "milk chocolate" is not milk. Missing a synonym ("OJ") only means buying something you already have; a false match means not buying something you need.

The result, on the same photos and staples:

A scan is now one Gemma call plus two checks that take about a millisecond each:

3. Generation is what takes the time. A recipe request is about 800 tokens; nearly all of its ~30 s is Gemma generating the answer, with model and prompt reading taking a few seconds. Shorter answers, not a faster prompt, are what would speed it up next.

4. Checking my own setup. The looping scans' errors never showed up as Sentry issues, even though their traces did. Sentry's usage stats explained why: the error events had been rejected as too large. By default, Sentry's Python SDK attaches each stack frame's local variables to error events, and here those held the prompt, Gemma's answer and the base64 photo, several times over. So errors from real scans were silently dropped, and smaller ones would have sent exactly the data I promised never to send. One setting (include_local_variables=False) fixed both, and I checked it end to end: the same error with a real photo now arrives in Sentry at 11 KB, with no prompt, answer or photo in it.

Privacy is kept, in traces and error reports alike. Only metadata is sent: timings, token counts and how many items each check looked at. The photo, the prompts, Gemma's answers and even the food names never leave the machine, and the tests check exactly that for both.

I also tried the 3× bigger model on the same photos, on the same laptop (before the matching speed-up):

12B reads messy photos a little better, but it's about 3× slower on recipes and the recipes weren't better. I kept 4B. Switching is one environment variable (FRIDGE_MODEL=gemma3:12b) if your machine is faster than mine.

eleven_v4), one line per moment in the video, stitched so it sounds like one take. The script is scripts/narrate_demo.py Example photos are from Wikimedia Commons: USDA (public domain) and others under CC BY-SA. Full credits are in the repo.