{"slug": "stop-using-one-prompt-for-every-photo-local-image-classification-before-calling", "title": "Stop Using One Prompt for Every Photo! Local Image Classification Before Calling Firebase AI Logic GenerativeModel", "summary": "A developer built a client-side image classification step that runs before calling Firebase AI Logic's GenerativeModel, using the @mediapipe/tasks-decision library and the EmbeddingGemma 2 LiteRT model to sort uploads into people, general, or place categories. The classifier caches the model in the browser's CacheStorage, pulls its path from Firebase Remote Config, and routes each category to a matching prompt and model configuration so Gemini generates category-appropriate alt text. The approach replaces a single generic prompt with per-category prompts and schemas.", "body_md": "I upload images of places, wrestlers, and general objects to the application. When uploading images of people, I want the Gemini model to generate alt text about their appearance, posture, and attire. When uploading images of places, I want the Gemini model to generate alt text with their official names, coordinates, and locations. To support this, I installed the `@mediapipe/tasks-decision` library, which uses the EmbeddingGemma 2 model (`embeddinggemma-2-text-vision-440m-litert-lm`) to classify the category of a picture (` people`, `general`, or `place`). The category decides which Firebase AI Logic `GenerativeModel` and prompt generate the text response.\n\n`ImageClassifierService` checks the browser's `CacheStorage` (`decision-model-v1`) for the EmbeddingGemma 2 model, downloads and caches it via `HttpClient` on a cache miss, and uses `DecisionMaker` with the provided choices to determine the image's category (`people`, `general`, or `place`).` AiModelOptions` (optional `systemInstruction`, response `schema`, and `tools`) from `CATEGORY_MODEL_CONFIGS`.` GenerativeModel` Lookup & Content Generation`VisionService` passes `options` to `AiModelFactory.getModel(options)` to reuse or create the matching Firebase AI Logic `generateContent([prompt, imagePart])` with the category's prompt to generate the text response.\n\n``` php\nflowchart TD\n    IMG[\"Selected Image (File)\"] --> PREP[\"preprocessImageForVision() -> imageBytes + base64\"]\n    PREP --> CLS[\"ImageClassifierService.classifyImage(imageBytes)\"]\n    CLS --> CACHE_CHK{\"Local model in CacheStorage?\"}\n    CACHE_CHK -- \"Miss\" --> DL[\"HttpClient GET -> cache.put() -> Blob URL\"]\n    CACHE_CHK -- \"Hit\" --> BLOB[\"cache.match() -> Blob URL\"]\n    DL & BLOB --> DM[\"DecisionMaker.evaluateChoice(imageBytes, IMAGE_CATEGORY_QUESTION)\"]\n    DM --> CAT{\"Best Category\"}\n    CAT -- \"people\" --> CFG_P[\"CATEGORY_MODEL_CONFIGS['people']\"]\n    CAT -- \"general\" --> CFG_G[\"CATEGORY_MODEL_CONFIGS['general']\"]\n    CAT -- \"place\" --> CFG_L[\"CATEGORY_MODEL_CONFIGS['place']\"]\n    CFG_P & CFG_G & CFG_L --> FAC[\"AiModelFactory.getModel(options)\"]\n    FAC --> GEN[\"aiModel.generateContent([prompt, imagePart])\"]\n```\n\n`@mediapipe/tasks-decision`, Serving WASM Assets, and Adding `decisionModelPath` to Remote Config\nI installed the latest `@mediapipe/tasks-decision` package (`1.2.3`), which provides the API to evaluate the category of an image input.\n\n```\nnpm install @mediapipe/tasks-decision\n```\n\n`angular.json`\nThe decision tasks require loading the WASM files at runtime. Instead of pointing to the WASM files on a CDN, I used the files installed in `node_modules/@mediapipe/tasks-decision/wasm`. Since the service cannot load files directly from `node_modules` in the browser, I configured `angular.json` to copy `node_modules/@mediapipe/tasks-decision/wasm` to `/wasm` (`WASM_PATH = '/wasm'`) so the application can access them.\n\n```\n// angular.json (projects.ng-firebase-tts.architect.build.options.assets)\n{\n  \"glob\": \"**/*\",\n  \"input\": \"node_modules/@mediapipe/tasks-decision/wasm\",\n  \"output\": \"/wasm\"\n}\n```\n\nI added the `decisionModelPath` parameter to Firebase Remote Config and `public/remote-config-defaults.json`, defaulting to `https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm` for on-device image classification. I also added the `decisionModelPath` property to `AppRemoteConfig`, which `ConfigService` updates when fetching the model path dynamically from Firebase Remote Config.\n\n```\n// public/remote-config-defaults.json\n{\n  \"decisionModelPath\": \"https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm\",\n  ...\n}\n// src/app/core/firebase/config.service.ts (decisionModelPath in #appConfig & fetchRemoteConfig)\n  #appConfig: AppRemoteConfig = {\n    ...\n    decisionModelPath: remoteConfigDefaults.decisionModelPath,\n  };\n  ...\n  this.#appConfig = {\n    ...\n    decisionModelPath: getValue(rc, 'decisionModelPath').asString(),\n  };\n```\n\n`ImageClassifierService` encapsulates the image-classification logic that `VisionService` uses in step one of the image-analysis pipeline:\n\n`#cachedModelPath``#decisionMakerPromise`` Promise<DecisionMaker>` so the service initializes the local model at most once per active model path.`DestroyRef.onDestroy()`` closeActiveDecisionMaker()` before the application shuts down to avoid a WASM memory leak.\n\n```\n// src/app/core/vision/image-classifier.service.ts (fields & lifecycle)\nexport class ImageClassifierService {\n  #window = inject(WINDOW);\n  #configService = inject(ConfigService);\n  #destroyRef = inject(DestroyRef);\n  #decisionMakerPromise: Promise<DecisionMaker> | null = null;\n  #cachedModelPath: string | null = null;\n  #http = inject(HttpClient);\n\n  constructor() {\n    this.#destroyRef.onDestroy(() => this.closeActiveDecisionMaker());\n  }\n```\n\n`initializeDecisionMaker` & `resolveCachedModelUrl`)\n`initializeDecisionMaker` initializes the MediaPipe WASM runtime, loads the cached model, and prewarms the classification options:\n\n`FilesetResolver` from `FilesetResolver.forDecisionTasks(WASM_PATH)` to load the WASM files from `/wasm`.` this.resolveCachedModelUrl()`)` DECISION_MODEL_CACHE_NAME`) bucket in `modelAssetPath`:\n`Blob`, stores it in the `blob:` URL.`DecisionMaker.createFromOptions()` to create the `delegate` (`'GPU'` or `'CPU'`), `backendType: 3` (the LiteRT-LM backend), and `MAX_NUM_TOKENS`, then calls `prewarmChoice(IMAGE_CATEGORY_QUESTION)` to prewarm the WASM session with the `URL.revokeObjectURL(resolvedModelPath)` revokes the temporary \n\n```\n// src/app/core/vision/image-classifier.service.ts (initializeDecisionMaker & resolveCachedModelUrl)\n  private async initializeDecisionMaker(modelAssetPath: string, delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {\n    const { DecisionMaker, FilesetResolver } = await import('@mediapipe/tasks-decision');\n    const wasmFileset = await FilesetResolver.forDecisionTasks(WASM_PATH);\n    const resolvedModelPath = await this.resolveCachedModelUrl(modelAssetPath);\n\n    try {\n      const decisionMaker = await DecisionMaker.createFromOptions(wasmFileset, {\n        baseOptions: {\n          modelAssetPath: resolvedModelPath,\n          delegate,\n        },\n        backendType: 3,\n        maxNumTokens: MAX_NUM_TOKENS,\n      });\n\n      await decisionMaker.prewarmChoice(IMAGE_CATEGORY_QUESTION);\n\n      return decisionMaker;\n    } finally {\n      if (resolvedModelPath.startsWith('blob:')) {\n        URL.revokeObjectURL(resolvedModelPath);\n      }\n    }\n  }\n\n  private async resolveCachedModelUrl(modelAssetPath: string): Promise<string> {\n    const cacheStorage = this.#window?.caches;\n    if (!cacheStorage) {\n      return modelAssetPath;\n    }\n\n    try {\n      const cache = await cacheStorage.open(DECISION_MODEL_CACHE_NAME);\n      const keys = await cache.keys();\n      await Promise.all(keys.filter((req) => req.url !== modelAssetPath).map((req) => cache.delete(req)));\n\n      let response = await cache.match(modelAssetPath);\n      if (!response) {\n        const httpResponse = await firstValueFrom(\n          this.#http.get(modelAssetPath, { observe: 'response', responseType: 'blob' }),\n        );\n        const downloadedBlob = httpResponse.body;\n        if (!downloadedBlob) {\n          return modelAssetPath;\n        }\n        response = new Response(downloadedBlob, {\n          status: httpResponse.status,\n          headers: {\n            'Content-Type': httpResponse.headers.get('Content-Type') ?? 'application/octet-stream',\n          },\n        });\n        await cache.put(modelAssetPath, response.clone());\n      }\n\n      const blob = await response.blob();\n      return URL.createObjectURL(blob);\n    } catch {\n      return modelAssetPath;\n    }\n  }\n```\n\n*Caption: The downloaded `embeddinggemma-2-text-vision-440m-litert-lm` model stored inside the browser's `decision-model-v1` `CacheStorage`.*\n\n`getValidDecisionMaker` & `closeActiveDecisionMaker`)\n`getValidDecisionMaker()`)` initializeDecisionMaker()` to construct a new `this.#decisionMakerPromise`, calls `decisionMaker.close()` to free the WASM engine and model memory, and resets `null`.\n\n```\n// src/app/core/vision/image-classifier.service.ts (getValidDecisionMaker & closeActiveDecisionMaker)\n  private async getValidDecisionMaker(delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {\n    const modelAssetPath = this.#configService.appConfig.decisionModelPath;\n\n    if (this.#decisionMakerPromise && this.#cachedModelPath !== modelAssetPath) {\n      await this.closeActiveDecisionMaker();\n    }\n\n    if (!this.#decisionMakerPromise) {\n      this.#cachedModelPath = modelAssetPath;\n      this.#decisionMakerPromise = this.initializeDecisionMaker(modelAssetPath, delegate).catch((error) => {\n        this.#decisionMakerPromise = null;\n        this.#cachedModelPath = null;\n        throw error;\n      });\n    }\n\n    return this.#decisionMakerPromise;\n  }\n\n  private async closeActiveDecisionMaker(): Promise<void> {\n    if (!this.#decisionMakerPromise) {\n      return;\n    }\n\n    try {\n      const decisionMaker = await this.#decisionMakerPromise;\n      decisionMaker.close();\n    } catch {\n      // Ignore initialization or WASM teardown errors during cleanup\n    } finally {\n      this.#decisionMakerPromise = null;\n      this.#cachedModelPath = null;\n    }\n  }\n```\n\n`IMAGE_CATEGORY_QUESTION`)\nBefore calling `DecisionMaker`, I define the classification rubric and token budget in `image-classifier.const.ts`:\n\n`instructions`)` normalizePrior: false`)` normalizePrior` to `false` because the input is an image (`{ imageBytes }`) rather than text; null-context prior normalization is intended for text-only choice prompts.`MAX_NUM_TOKENS = 512`)` 512` tokens because EmbeddingGemma 2 maps a single image to `context`, `4096`-token window in browser memory.` temperature: 0.05`)\n\n``` python\n// src/app/core/vision/image-classifier.const.ts\nimport type { ChoiceQuestion } from '@mediapipe/tasks-decision';\n\nexport const WASM_PATH = '/wasm';\nexport const MAX_NUM_TOKENS = 512;\nexport const DECISION_MODEL_CACHE_NAME = 'decision-model-v1';\n\nexport const IMAGE_CATEGORY_QUESTION: ChoiceQuestion = {\n  context: 'Analyze the uploaded image for accessibility alt-text routing.',\n  instructions:\n    'Select the primary subject category of this image. If an outdoor landscape, architecture, monument, landmark, or travel destination is visible behind people, select place. Otherwise, if any person, masked figure, or group of people is visible, select people.',\n  options: [\n    {\n      label: 'place',\n      description: 'An outdoor landscape, natural scenery, architecture, monument, landmark, or travel destination.',\n    },\n    {\n      label: 'people',\n      description: 'A person, portrait, face, masked or costumed figure, performer, or group of people.',\n    },\n    {\n      label: 'general',\n      description: 'A close-up of food, drink, animal, pet, vehicle, or household object.',\n    },\n  ],\n  temperature: 0.05,\n  normalizePrior: false,\n};\n```\n\n`resolveDelegate`)\n`'gpu' in this.#window.navigator`) and returns `.litertlm` Models`undefined` when WebGPU is unavailable and the model path ends with `classifyImage()` to fall back to `'general'`).\n\n```\n// src/app/core/vision/image-classifier.service.ts (resolveDelegate)\n  private resolveDelegate(): 'GPU' | 'CPU' | undefined {\n    const hasWebGpu = Boolean(this.#window?.navigator && 'gpu' in this.#window.navigator);\n    if (hasWebGpu) {\n      return 'GPU';\n    }\n    if (this.#configService.appConfig.decisionModelPath.endsWith('.litertlm')) {\n      return undefined;\n    }\n    return 'CPU';\n  }\n```\n\n`classifyImage`)\n`decisionMaker.evaluateChoice()` to evaluate `imageBytes` is empty, \n\n```\n// src/app/core/vision/image-classifier.service.ts (classifyImage)\n  async classifyImage(imageBytes: Uint8Array): Promise<ImageSubjectCategory> {\n    if (imageBytes.byteLength === 0) {\n      return 'general';\n    }\n\n    const delegate = this.resolveDelegate();\n    if (!delegate) {\n      return 'general';\n    }\n\n    try {\n      const decisionMaker = await this.getValidDecisionMaker(delegate);\n      const { probabilities } = await decisionMaker.evaluateChoice({ imageBytes }, IMAGE_CATEGORY_QUESTION);\n\n      let bestCategory: ImageSubjectCategory = 'general';\n      let highestProbability = -1;\n\n      for (const [key, probability] of Object.entries(probabilities)) {\n        if (probability > highestProbability && (key === 'place' || key === 'people' || key === 'general')) {\n          highestProbability = probability;\n          bestCategory = key;\n        }\n      }\n\n      return bestCategory;\n    } catch (error) {\n      console.warn('[ImageClassifierService] On-device classification failed, falling back to general:', error);\n      return 'general';\n    }\n  }\n```\n\n`prompt` and `peopleSystemInstruction` and `placeSystemInstruction` to define the personas and verification rules of the | Category | `systemInstruction` | `prompt` | `tools` | \n|---|---|---|---|\n| **`general`** | *None* (uses prompt instructions directly) | `altTextPrompt` | `[{ googleSearch: {} }]` | \n| **`people`** | `peopleSystemInstruction` (strict anti-guessing rule for faces/masks) | `peopleAltTextPrompt` | `[{ googleSearch: {} }]` | \n| **`place`** | `placeSystemInstruction` (landmark & coordinate verification, no fabrication) | `placeAltTextPrompt` | `[{ googleSearch: {} }]` | \n\n``` js\n// src/app/core/vision/vision.const.ts\nexport const CATEGORY_MODEL_CONFIGS: Record<ImageSubjectCategory, CategoryModelConfig> = {\n  general: {\n    options: {\n      schema: ImageAnalysisSchema,\n      tools: [{ googleSearch: {} }],\n    },\n    prompt: altTextPrompt,\n  },\n  people: {\n    options: {\n      schema: ImageAnalysisSchema,\n      tools: [{ googleSearch: {} }],\n      systemInstruction: peopleSystemInstruction,\n    },\n    prompt: peopleAltTextPrompt,\n  },\n  place: {\n    options: {\n      schema: ImageAnalysisSchema,\n      tools: [{ googleSearch: {} }],\n      systemInstruction: placeSystemInstruction,\n    },\n    prompt: placeAltTextPrompt,\n  },\n};\n```\n\n*Full prompt and `systemInstruction` strings: [src/app/core/vision/vision.const.ts](https://github.com/railsstudent/ng-firebase-tts/blob/main/src/app/core/vision/vision.const.ts)*\n\n`this.#imageClassifierService`.` this.#imageClassifierService.classifyImage(imageBytes)` to determine the category of the preprocessed `Uint8Array`).` category` to retrieve the `this.#aiModelFactory.getModel(options)` to obtain a `aiModel.generateContent([prompt, imagePart])` to generate the text response.\n\n```\n// src/app/core/vision/vision.service.ts (analyzeImage)\n  readonly #imageClassifierService = inject(ImageClassifierService);\n\n  async analyzeImage(image: File): Promise<ImageAnalysisResponse> {\n    if (!image) {\n      throw Error('image is required to generate texts.');\n    }\n\n    const { data, mimeType, optimizationMetrics, imageBytes } = await preprocessImageForVision(image, this.#window);\n    const category = await this.#imageClassifierService.classifyImage(imageBytes);\n    const { options, prompt } = CATEGORY_MODEL_CONFIGS[category] || CATEGORY_MODEL_CONFIGS['general'];\n\n    const imagePart: InlineDataPart = {\n      type: 'inlineData',\n      inlineData: {\n        data,\n        mimeType,\n      },\n    };\n\n    const aiModel = await this.#aiModelFactory.getModel(options);\n    const result = await aiModel.generateContent([prompt, imagePart]);\n    ...\n```\n\nI uploaded an image of six Japanese wrestlers, and the classifier returned the `people` category. The `people` system instruction tells the model to use the Google Search tool to verify visible text clues and find a real-world fact based on the generated tags. The `people` prompt asks the model to include a person's name in the alt text only when the name is printed in the image or 100% certain and otherwise to describe the person's features, posture, and attire.\n\n*Caption: Output for an image of six Japanese wrestlers when classified as `people`.*\n\nI uploaded an image of a butterfly, and the classifier returned the `general` category. The `general` prompt (`altTextPrompt`) generated the alt text and tags and used the Google Search tool to verify a real-world fact based on the tags.\n\n*Caption: Output for an image of a butterfly when classified as `general`.*\n\nI uploaded an image of Kek Lok Si Temple in Penang, Malaysia, and analyzed the image to generate the alt text and tags. `DecisionMaker` classified the image input as `place` and used the `place` prompt (`placeAltTextPrompt`) to generate the location name, coordinates, and architectural features without fabricating unverified coordinates. The `place` system instruction (`placeSystemInstruction`) directs the model to use the Google Search tool to verify the official place name and coordinates and to find a real-world fact based on the generated tags.\n\n*Caption: Output for Kek Lok Si Temple in Penang, Malaysia when classified as `place`.*\n\n`AiModelFactory` reuse initialized model instances in memory.`main.js`)` await import('@mediapipe/tasks-decision')` inside `URL.revokeObjectURL()` after", "url": "https://wpnews.pro/news/stop-using-one-prompt-for-every-photo-local-image-classification-before-calling", "canonical_source": "https://dev.to/gde/stop-using-one-prompt-for-every-photo-local-image-classification-before-calling-firebase-ai-logic-20gk", "published_at": "2026-11-22 14:11:32+00:00", "updated_at": "2026-11-22 14:12:59.934467+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "computer-vision", "generative-ai"], "entities": ["Firebase AI Logic", "Gemini", "MediaPipe", "EmbeddingGemma 2", "Hugging Face", "Angular"], "also_reported_by": [], "takeaway": "Classify images locally with MediaPipe's DecisionMaker and EmbeddingGemma 2 before calling Gemini, then route each category to its own prompt and schema so alt text matches the subject.", "recommendation": {"advice": "Classify uploads on-device first and route each category to a tailored prompt and schema, so you avoid sending every image through one generic Gemini call.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/stop-using-one-prompt-for-every-photo-local-image-classification-before-calling", "markdown": "https://wpnews.pro/news/stop-using-one-prompt-for-every-photo-local-image-classification-before-calling.md", "text": "https://wpnews.pro/news/stop-using-one-prompt-for-every-photo-local-image-classification-before-calling.txt", "jsonld": "https://wpnews.pro/news/stop-using-one-prompt-for-every-photo-local-image-classification-before-calling.jsonld"}}