I built an app that OCRs every screenshot on your phone — here's the architecture

A developer built Screenshot Vault, an app that OCRs every screenshot on a phone and makes them searchable. The app uses on-device Google ML Kit for text recognition to keep images private, and sends only extracted text to Gemini for categorization. The developer chose a staged, resumable processing approach and moved the AI API key to a server-side proxy to avoid exposing it in the client.

We all have that folder of 500+ screenshots we can never find anything in. I got tired of scrolling through mine looking for a receipt from three months ago, so I built Screenshot Vault — an app that reads the text in every screenshot on your phone and lets you search it like Google. Here's how it's built, and a few decisions that ended up mattering more than I expected. The stack Expo React Native + Expo Router — file-based routing, EAS for native builds @react-native-ml-kit/text-recognition — on-device OCR via Google ML Kit expo-sqlite — local persistence expo-media-library — reading the device's Screenshots album Gemini 3.6 Flash — categorization, titles, tags, called through a small Next.js API route Next.js + Vercel — landing page + the backend proxy Decision 1 : OCR stays fully on-device My first instinct was to upload screenshots to a server for processing. Then I actually thought about what's in a typical screenshot folder — bank OTPs, payment confirmations, private chats, addresses. Uploading that to a database I control is a liability for me and a real risk for users if anything ever leaks. Google ML Kit's text recognition runs entirely on-device, for free, with no server round-trip. So OCR happens locally, and the only thing that ever leaves the phone is the already-extracted text — sent to Gemini for a lightweight categorization call, not the image itself. This also happens to be cheaper to run at scale, since I'm not paying for image storage or bandwidth. Decision 2 : Don't block the user on a full scan If someone installs this with 500 existing screenshots, running OCR + AI on all of them before showing anything would mean a 10-20 minute wait on first launch. That's an instant uninstall. Instead, processing is staged and resumable: CREATE TABLE screenshots id TEXT PRIMARY KEY, uri TEXT, text content TEXT, category TEXT, ai title TEXT, tags TEXT, ocr done INTEGER DEFAULT 0, ai done INTEGER DEFAULT 0, created at INTEGER ; On open: If the app gets closed mid-scan , it just picks up from wherever the flags left off next time — no reprocessing, no lost work. This is basically the same idea Google Photos uses for its "preparing your library" background indexing. Decision 3 : Never put the AI API key in the client Early version called Gemini directly from the app with the key in an env var. Realized pretty quickly that anyone who installs the APK can extract that key and rack up usage on my bill — env vars prefixed EXPO PUBLIC get bundled straight into the client, no exceptions. Moved it behind a single Next.js API route on the same Vercel project as the landing page: export async function POST req: NextRequest { const { text } = await req.json ; const apiKey = process.env.GEMINI API KEY; // server-only, never shipped to client const res = await fetch GEMINI ENDPOINT, { method: 'POST', headers: { 'x-goog-api-key': apiKey }, body: JSON.stringify { / prompt asking for category + title + tags in one call / } , } ; // ...parse and return } One call returns category + title + tags together instead of three separate requests — matters a lot on Gemini's free tier, which caps out fast. Still early and Android-only right now, testing with a small group before a wider release. If you want to follow along or try it when it's ready: screenshot-vault-lac.vercel.app Curious if anyone's tackled similar on-device vs. cloud tradeoffs for AI features — happy to talk through any of this in the comments.