I built an app that OCRs every screenshot on your phone — here's the architecture

We all have that folder of 500+ screenshots we can never find anything in. I got tired of scrolling through mine looking for a receipt from three months ago, so I built Screenshot Vault — an app that reads the text in every screenshot on your phone and lets you search it like Google.

Here's how it's built, and a few decisions that ended up mattering more than I expected.

The stack

Expo (React Native) + Expo Router — file-based routing, EAS for native builds

@react-native-ml-kit/text-recognition — on-device OCR via Google ML Kit

expo-sqlite — local persistence

expo-media-library — reading the device's Screenshots album

Gemini 3.6 Flash — categorization, titles, tags, called through a small Next.js API route

Next.js + Vercel — landing page + the backend proxy

Decision #1: OCR stays fully on-device

My first instinct was to upload screenshots to a server for processing. Then I actually thought about what's in a typical screenshot folder — bank OTPs, payment confirmations, private chats, addresses. Up that to a database I control is a liability for me and a real risk for users if anything ever leaks.

Google ML Kit's text recognition runs entirely on-device, for free, with no server round-trip. So OCR happens locally, and the only thing that ever leaves the phone is the already-extracted text — sent to Gemini for a lightweight categorization call, not the image itself. This also happens to be cheaper to run at scale, since I'm not paying for image storage or bandwidth.

Decision #2: Don't block the user on a full scan

If someone installs this with 500 existing screenshots, running OCR + AI on all of them before showing anything would mean a 10-20 minute wait on first launch. That's an instant uninstall.

Instead, processing is staged and resumable:

CREATE TABLE screenshots (
  id TEXT PRIMARY KEY,
  uri TEXT,
  text_content TEXT,
  category TEXT,
  ai_title TEXT,
  tags TEXT,
  ocr_done INTEGER DEFAULT 0,
  ai_done INTEGER DEFAULT 0,
  created_at INTEGER
);

On open:

If the app gets closed mid-scan, it just picks up from wherever the flags left off next time — no reprocessing, no lost work. This is basically the same idea Google Photos uses for its "preparing your library" background indexing.

Decision #3:_ Never put the AI API key in the client_

Early version called Gemini directly from the app with the key in an env var. Realized pretty quickly that anyone who installs the APK can extract that key and rack up usage on my bill — env vars prefixed EXPO_PUBLIC_ get bundled straight into the client, no exceptions.

Moved it behind a single Next.js API route on the same Vercel project as the landing page:

export async function POST(req: NextRequest) {
  const { text } = await req.json();
  const apiKey = process.env.GEMINI_API_KEY; // server-only, never shipped to client

  const res = await fetch(GEMINI_ENDPOINT, {
    method: 'POST',
    headers: { 'x-goog-api-key': apiKey },
    body: JSON.stringify({ /* prompt asking for category + title + tags in one call */ }),
  });
  // ...parse and return
}

One call returns category + title + tags together instead of three separate requests — matters a lot on Gemini's free tier, which caps out fast.

Still early and Android-only right now, testing with a small group before a wider release. If you want to follow along or try it when it's ready: screenshot-vault-lac.vercel.app

Curious if anyone's tackled similar on-device vs. cloud tradeoffs for AI features — happy to talk through any of this in the comments.