Name it, then look up: a field guide where the phone's job ends after one photo

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

Most nature apps have the same quiet problem. You see a bird, you lift your phone, and by the time the app has told you its name, the bird has gone and you're still reading about it.

I wanted the opposite. In Spotted, getting the name is the quick bit. Then the app asks for something harder: put the phone down for one minute and actually look at the thing.

Spotted is a pocket field guide that runs on your phone, in the browser. You take or upload one photo of a bird, lizard, insect, tree or plant. An open model suggests a name from the field guide. Then a button called Look up asks a second open model, Gemma, for one thing to notice. The screen dims to a 60-second timer. When the minute ends there's a soft chime and one question: What did you notice? Your answer is saved into the sighting.

That minute is the whole point. The name, the facts, the feed, the map: all of that is there to get you to the minute and then get out of the way.

A few details I cared about:

It's for anyone who has wondered "what is that?" about a bird on a fence or a lizard on a wall, and wants an answer that ends with them looking up instead of down.

Live app: https://spotted-91b2.onrender.com

It's on Render's free plan, so if nobody has visited for a while the first load takes about a minute to wake up. After that it's quick.

The video is real footage of the live site in a phone-sized browser. The model download, Gemma's thinking time and the 60-second timer are sped up, and labelled as sped up on screen.

Try it in 60 seconds:

See nature. Share it. You photograph a bird, animal, insect, or plant. An open model suggests the name. Then you share the sighting, or you just look up again.

The product name is one constant, APP_NAME in src/config/brand.ts.

The demo trail is fictional and West African: Lekki Conservation Centre, Okomu National Park, Obudu Mountain Resort, and Yankari Game Reserve. The guide also includes species from elsewhere, so a walk outside Nigeria still gets a name. Reference photos are credited in public/seed/credits.json. The people in the feed are not real.

MIT licensed. If you want to look at the interesting parts first:

src/lib/identify/rank.ts     the "not sure" rule
src/lib/identify/look.ts     Gemma in the browser: one thing to notice
src/components/look-up.tsx   the dimmed minute, the chime and the question
src/data/more-species.ts     the West African additions to the guide
src/lib/privacy.ts           opt-in pins, softened cells for sensitive species

Spotted is a Next.js PWA with a small SQLite-backed API for the trail. The two jobs that need AI both run on open models in the browser through Transformers.js.

Naming: CLIP ViT-B/32. I use Xenova/clip-vit-base-patch32 (MIT), an 8-bit ONNX build of OpenAI's CLIP, as a zero-shot classifier. Every species in the guide becomes a text prompt. The photo is compared with all of them, and the scores are a softmax across the whole guide. The guide works as an allow-list: CLIP can only answer with a species that has written facts behind it, so it can't invent an animal.

The scores are spread across 114 options, so the numbers look modest. On my reference pied kingfisher photo, the top match was pied kingfisher at 25%, with the next name at 10%. That lead is what the rule in rank.ts checks for. Below 0.12, or with a gap under 0.04, you get maybes instead of a name.

Look up: Gemma 3 270M. This is the heart of the app. When you press Look up, the page loads onnx-community/gemma-3-270m-it-ONNX and runs it right there in the browser: WebGPU with the q4f16 build where it's available, and WASM with q4 where it isn't. Getting the official weights to run on the WASM runtime took one small change. A gather index in the graph needed casting to int32, and that patched graph ships in public/models/gemma.

Gemma's job here is narrow on purpose. It gets the species name and the guide's behaviour sentence, and has to answer in one sentence that starts with "Watch". For the pied kingfisher, the sentence it gave on my test run was "Watch it hunt this way." Simple, and it points you at the right thing, because pied kingfishers hover over the water before they dive. If Gemma's answer is the wrong shape, or the phone can't run it, the app falls back to the guide's own behaviour line and labels it Field guide. The minute still happens either way.

I kept Gemma away from the facts deliberately. A 270M-parameter model is small enough to run on a phone, and too small to trust with a conservation status. So the facts come from the guide and its sources, and Gemma only gives the nudge.

Some real numbers from testing:

About 20 seconds for one sentence on WASM is slow, so the app shows a calm "Finding one thing" screen while it works. I decided that was an acceptable trade for now: a short wait in exchange for a full minute of looking up, with nothing leaving the phone.

Where it gets things wrong. Here's what I saw testing with Creative Commons photos at my desk:

That's why the "not sure" state exists and why "Looks like" is worded more softly than "This looks like". It's also why the next item below is a better vision model.

Hosting. The app runs on Render's free web service from a render.yaml blueprint, with a cheap /api/health check. The free instance has about 512 MB of RAM, which isn't enough to host CLIP and Gemma on the server, and I like that constraint. It pushes the AI onto the phone, which is where it should be for something you use outdoors. Render serves the app, the trail and the model files. The visitor's phone does the thinking.

For this project, the closed version would have been a worse product, not just a more expensive one.

To be straight about where it stands:

Xenova/clip-vit-base-patch32: MIT onnx-community/gemma-3-270m-it-ONNX: @huggingface/transformers): Apache-2.0 public/seed/credits.json