cd /news/artificial-intelligence/stop-using-one-prompt-for-every-phot… · home › topics › artificial-intelligence › article
[ARTICLE · art-174209] src=dev.to ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

Stop Using One Prompt for Every Photo! Local Image Classification Before Calling Firebase AI Logic GenerativeModel

A developer built a client-side image classification step that runs before calling Firebase AI Logic's GenerativeModel, using the @mediapipe/tasks-decision library and the EmbeddingGemma 2 LiteRT model to sort uploads into people, general, or place categories. The classifier caches the model in the browser's CacheStorage, pulls its path from Firebase Remote Config, and routes each category to a matching prompt and model configuration so Gemini generates category-appropriate alt text. The approach replaces a single generic prompt with per-category prompts and schemas.

recommendation

Classify uploads on-device first and route each category to a tailored prompt and schema, so you avoid sending every image through one generic Gemini call.

more in playbook: Controlling LLM cost
by read9 min views3 publishedNov 22, 2026

I upload images of places, wrestlers, and general objects to the application. When up images of people, I want the Gemini model to generate alt text about their appearance, posture, and attire. When up images of places, I want the Gemini model to generate alt text with their official names, coordinates, and locations. To support this, I installed the @mediapipe/tasks-decision library, which uses the EmbeddingGemma 2 model (embeddinggemma-2-text-vision-440m-litert-lm) to classify the category of a picture ( people, general, or place). The category decides which Firebase AI Logic GenerativeModel and prompt generate the text response.

ImageClassifierService checks the browser's CacheStorage (decision-model-v1) for the EmbeddingGemma 2 model, downloads and caches it via HttpClient on a cache miss, and uses DecisionMaker with the provided choices to determine the image's category (people, general, or place). AiModelOptions (optional systemInstruction, response schema, and tools) from CATEGORY_MODEL_CONFIGS. GenerativeModel Lookup & Content GenerationVisionService passes options to AiModelFactory.getModel(options) to reuse or create the matching Firebase AI Logic generateContent([prompt, imagePart]) with the category's prompt to generate the text response.

flowchart TD
    IMG["Selected Image (File)"] --> PREP["preprocessImageForVision() -> imageBytes + base64"]
    PREP --> CLS["ImageClassifierService.classifyImage(imageBytes)"]
    CLS --> CACHE_CHK{"Local model in CacheStorage?"}
    CACHE_CHK -- "Miss" --> DL["HttpClient GET -> cache.put() -> Blob URL"]
    CACHE_CHK -- "Hit" --> BLOB["cache.match() -> Blob URL"]
    DL & BLOB --> DM["DecisionMaker.evaluateChoice(imageBytes, IMAGE_CATEGORY_QUESTION)"]
    DM --> CAT{"Best Category"}
    CAT -- "people" --> CFG_P["CATEGORY_MODEL_CONFIGS['people']"]
    CAT -- "general" --> CFG_G["CATEGORY_MODEL_CONFIGS['general']"]
    CAT -- "place" --> CFG_L["CATEGORY_MODEL_CONFIGS['place']"]
    CFG_P & CFG_G & CFG_L --> FAC["AiModelFactory.getModel(options)"]
    FAC --> GEN["aiModel.generateContent([prompt, imagePart])"]

@mediapipe/tasks-decision, Serving WASM Assets, and Adding decisionModelPath to Remote Config I installed the latest @mediapipe/tasks-decision package (1.2.3), which provides the API to evaluate the category of an image input.

npm install @mediapipe/tasks-decision

angular.json The decision tasks require the WASM files at runtime. Instead of pointing to the WASM files on a CDN, I used the files installed in node_modules/@mediapipe/tasks-decision/wasm. Since the service cannot load files directly from node_modules in the browser, I configured angular.json to copy node_modules/@mediapipe/tasks-decision/wasm to /wasm (WASM_PATH = '/wasm') so the application can access them.

// angular.json (projects.ng-firebase-tts.architect.build.options.assets)
{
  "glob": "**/*",
  "input": "node_modules/@mediapipe/tasks-decision/wasm",
  "output": "/wasm"
}

I added the decisionModelPath parameter to Firebase Remote Config and public/remote-config-defaults.json, defaulting to https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm for on-device image classification. I also added the decisionModelPath property to AppRemoteConfig, which ConfigService updates when fetching the model path dynamically from Firebase Remote Config.

// public/remote-config-defaults.json
{
  "decisionModelPath": "https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm",
  ...
}
// src/app/core/firebase/config.service.ts (decisionModelPath in #appConfig & fetchRemoteConfig)
  #appConfig: AppRemoteConfig = {
    ...
    decisionModelPath: remoteConfigDefaults.decisionModelPath,
  };
  ...
  this.#appConfig = {
    ...
    decisionModelPath: getValue(rc, 'decisionModelPath').asString(),
  };

ImageClassifierService encapsulates the image-classification logic that VisionService uses in step one of the image-analysis pipeline:

#cachedModelPath``#decisionMakerPromise`` Promise<DecisionMaker> so the service initializes the local model at most once per active model path.DestroyRef.onDestroy()`` closeActiveDecisionMaker() before the application shuts down to avoid a WASM memory leak.

// src/app/core/vision/image-classifier.service.ts (fields & lifecycle)
export class ImageClassifierService {
  #window = inject(WINDOW);
  #configService = inject(ConfigService);
  #destroyRef = inject(DestroyRef);
  #decisionMakerPromise: Promise<DecisionMaker> | null = null;
  #cachedModelPath: string | null = null;
  #http = inject(HttpClient);

  constructor() {
    this.#destroyRef.onDestroy(() => this.closeActiveDecisionMaker());
  }

initializeDecisionMaker & resolveCachedModelUrl) initializeDecisionMaker initializes the MediaPipe WASM runtime, loads the cached model, and prewarms the classification options:

FilesetResolver from FilesetResolver.forDecisionTasks(WASM_PATH) to load the WASM files from /wasm. this.resolveCachedModelUrl()) DECISION_MODEL_CACHE_NAME) bucket in modelAssetPath: Blob, stores it in the blob: URL.DecisionMaker.createFromOptions() to create the delegate ('GPU' or 'CPU'), backendType: 3 (the LiteRT-LM backend), and MAX_NUM_TOKENS, then calls prewarmChoice(IMAGE_CATEGORY_QUESTION) to prewarm the WASM session with the URL.revokeObjectURL(resolvedModelPath) revokes the temporary

// src/app/core/vision/image-classifier.service.ts (initializeDecisionMaker & resolveCachedModelUrl)
  private async initializeDecisionMaker(modelAssetPath: string, delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
    const { DecisionMaker, FilesetResolver } = await import('@mediapipe/tasks-decision');
    const wasmFileset = await FilesetResolver.forDecisionTasks(WASM_PATH);
    const resolvedModelPath = await this.resolveCachedModelUrl(modelAssetPath);

    try {
      const decisionMaker = await DecisionMaker.createFromOptions(wasmFileset, {
        baseOptions: {
          modelAssetPath: resolvedModelPath,
          delegate,
        },
        backendType: 3,
        maxNumTokens: MAX_NUM_TOKENS,
      });

      await decisionMaker.prewarmChoice(IMAGE_CATEGORY_QUESTION);

      return decisionMaker;
    } finally {
      if (resolvedModelPath.startsWith('blob:')) {
        URL.revokeObjectURL(resolvedModelPath);
      }
    }
  }

  private async resolveCachedModelUrl(modelAssetPath: string): Promise<string> {
    const cacheStorage = this.#window?.caches;
    if (!cacheStorage) {
      return modelAssetPath;
    }

    try {
      const cache = await cacheStorage.open(DECISION_MODEL_CACHE_NAME);
      const keys = await cache.keys();
      await Promise.all(keys.filter((req) => req.url !== modelAssetPath).map((req) => cache.delete(req)));

      let response = await cache.match(modelAssetPath);
      if (!response) {
        const httpResponse = await firstValueFrom(
          this.#http.get(modelAssetPath, { observe: 'response', responseType: 'blob' }),
        );
        const downloadedBlob = httpResponse.body;
        if (!downloadedBlob) {
          return modelAssetPath;
        }
        response = new Response(downloadedBlob, {
          status: httpResponse.status,
          headers: {
            'Content-Type': httpResponse.headers.get('Content-Type') ?? 'application/octet-stream',
          },
        });
        await cache.put(modelAssetPath, response.clone());
      }

      const blob = await response.blob();
      return URL.createObjectURL(blob);
    } catch {
      return modelAssetPath;
    }
  }

Caption: The downloaded embeddinggemma-2-text-vision-440m-litert-lm model stored inside the browser's decision-model-v1 CacheStorage.

getValidDecisionMaker & closeActiveDecisionMaker) getValidDecisionMaker()) initializeDecisionMaker() to construct a new this.#decisionMakerPromise, calls decisionMaker.close() to free the WASM engine and model memory, and resets null.

// src/app/core/vision/image-classifier.service.ts (getValidDecisionMaker & closeActiveDecisionMaker)
  private async getValidDecisionMaker(delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
    const modelAssetPath = this.#configService.appConfig.decisionModelPath;

    if (this.#decisionMakerPromise && this.#cachedModelPath !== modelAssetPath) {
      await this.closeActiveDecisionMaker();
    }

    if (!this.#decisionMakerPromise) {
      this.#cachedModelPath = modelAssetPath;
      this.#decisionMakerPromise = this.initializeDecisionMaker(modelAssetPath, delegate).catch((error) => {
        this.#decisionMakerPromise = null;
        this.#cachedModelPath = null;
        throw error;
      });
    }

    return this.#decisionMakerPromise;
  }

  private async closeActiveDecisionMaker(): Promise<void> {
    if (!this.#decisionMakerPromise) {
      return;
    }

    try {
      const decisionMaker = await this.#decisionMakerPromise;
      decisionMaker.close();
    } catch {
      // Ignore initialization or WASM teardown errors during cleanup
    } finally {
      this.#decisionMakerPromise = null;
      this.#cachedModelPath = null;
    }
  }

IMAGE_CATEGORY_QUESTION) Before calling DecisionMaker, I define the classification rubric and token budget in image-classifier.const.ts:

instructions) normalizePrior: false) normalizePrior to false because the input is an image ({ imageBytes }) rather than text; null-context prior normalization is intended for text-only choice prompts.MAX_NUM_TOKENS = 512) 512 tokens because EmbeddingGemma 2 maps a single image to context, 4096-token window in browser memory. temperature: 0.05)

// src/app/core/vision/image-classifier.const.ts
import type { ChoiceQuestion } from '@mediapipe/tasks-decision';

export const WASM_PATH = '/wasm';
export const MAX_NUM_TOKENS = 512;
export const DECISION_MODEL_CACHE_NAME = 'decision-model-v1';

export const IMAGE_CATEGORY_QUESTION: ChoiceQuestion = {
  context: 'Analyze the uploaded image for accessibility alt-text routing.',
  instructions:
    'Select the primary subject category of this image. If an outdoor landscape, architecture, monument, landmark, or travel destination is visible behind people, select place. Otherwise, if any person, masked figure, or group of people is visible, select people.',
  options: [
    {
      label: 'place',
      description: 'An outdoor landscape, natural scenery, architecture, monument, landmark, or travel destination.',
    },
    {
      label: 'people',
      description: 'A person, portrait, face, masked or costumed figure, performer, or group of people.',
    },
    {
      label: 'general',
      description: 'A close-up of food, drink, animal, pet, vehicle, or household object.',
    },
  ],
  temperature: 0.05,
  normalizePrior: false,
};

resolveDelegate) 'gpu' in this.#window.navigator) and returns .litertlm Modelsundefined when WebGPU is unavailable and the model path ends with classifyImage() to fall back to 'general').

// src/app/core/vision/image-classifier.service.ts (resolveDelegate)
  private resolveDelegate(): 'GPU' | 'CPU' | undefined {
    const hasWebGpu = Boolean(this.#window?.navigator && 'gpu' in this.#window.navigator);
    if (hasWebGpu) {
      return 'GPU';
    }
    if (this.#configService.appConfig.decisionModelPath.endsWith('.litertlm')) {
      return undefined;
    }
    return 'CPU';
  }

classifyImage) decisionMaker.evaluateChoice() to evaluate imageBytes is empty,

// src/app/core/vision/image-classifier.service.ts (classifyImage)
  async classifyImage(imageBytes: Uint8Array): Promise<ImageSubjectCategory> {
    if (imageBytes.byteLength === 0) {
      return 'general';
    }

    const delegate = this.resolveDelegate();
    if (!delegate) {
      return 'general';
    }

    try {
      const decisionMaker = await this.getValidDecisionMaker(delegate);
      const { probabilities } = await decisionMaker.evaluateChoice({ imageBytes }, IMAGE_CATEGORY_QUESTION);

      let bestCategory: ImageSubjectCategory = 'general';
      let highestProbability = -1;

      for (const [key, probability] of Object.entries(probabilities)) {
        if (probability > highestProbability && (key === 'place' || key === 'people' || key === 'general')) {
          highestProbability = probability;
          bestCategory = key;
        }
      }

      return bestCategory;
    } catch (error) {
      console.warn('[ImageClassifierService] On-device classification failed, falling back to general:', error);
      return 'general';
    }
  }

prompt and peopleSystemInstruction and placeSystemInstruction to define the personas and verification rules of the | Category | systemInstruction | prompt | tools | |---|---|---|---| | general | None (uses prompt instructions directly) | altTextPrompt | [{ googleSearch: {} }] | | people | peopleSystemInstruction (strict anti-guessing rule for faces/masks) | peopleAltTextPrompt | [{ googleSearch: {} }] | | place | placeSystemInstruction (landmark & coordinate verification, no fabrication) | placeAltTextPrompt | [{ googleSearch: {} }] |

// src/app/core/vision/vision.const.ts
export const CATEGORY_MODEL_CONFIGS: Record<ImageSubjectCategory, CategoryModelConfig> = {
  general: {
    options: {
      schema: ImageAnalysisSchema,
      tools: [{ googleSearch: {} }],
    },
    prompt: altTextPrompt,
  },
  people: {
    options: {
      schema: ImageAnalysisSchema,
      tools: [{ googleSearch: {} }],
      systemInstruction: peopleSystemInstruction,
    },
    prompt: peopleAltTextPrompt,
  },
  place: {
    options: {
      schema: ImageAnalysisSchema,
      tools: [{ googleSearch: {} }],
      systemInstruction: placeSystemInstruction,
    },
    prompt: placeAltTextPrompt,
  },
};

Full prompt and systemInstruction strings: src/app/core/vision/vision.const.ts

this.#imageClassifierService. this.#imageClassifierService.classifyImage(imageBytes) to determine the category of the preprocessed Uint8Array). category to retrieve the this.#aiModelFactory.getModel(options) to obtain a aiModel.generateContent([prompt, imagePart]) to generate the text response.

// src/app/core/vision/vision.service.ts (analyzeImage)
  readonly #imageClassifierService = inject(ImageClassifierService);

  async analyzeImage(image: File): Promise<ImageAnalysisResponse> {
    if (!image) {
      throw Error('image is required to generate texts.');
    }

    const { data, mimeType, optimizationMetrics, imageBytes } = await preprocessImageForVision(image, this.#window);
    const category = await this.#imageClassifierService.classifyImage(imageBytes);
    const { options, prompt } = CATEGORY_MODEL_CONFIGS[category] || CATEGORY_MODEL_CONFIGS['general'];

    const imagePart: InlineDataPart = {
      type: 'inlineData',
      inlineData: {
        data,
        mimeType,
      },
    };

    const aiModel = await this.#aiModelFactory.getModel(options);
    const result = await aiModel.generateContent([prompt, imagePart]);
    ...

I uploaded an image of six Japanese wrestlers, and the classifier returned the people category. The people system instruction tells the model to use the Google Search tool to verify visible text clues and find a real-world fact based on the generated tags. The people prompt asks the model to include a person's name in the alt text only when the name is printed in the image or 100% certain and otherwise to describe the person's features, posture, and attire.

Caption: Output for an image of six Japanese wrestlers when classified as people.

I uploaded an image of a butterfly, and the classifier returned the general category. The general prompt (altTextPrompt) generated the alt text and tags and used the Google Search tool to verify a real-world fact based on the tags.

Caption: Output for an image of a butterfly when classified as general.

I uploaded an image of Kek Lok Si Temple in Penang, Malaysia, and analyzed the image to generate the alt text and tags. DecisionMaker classified the image input as place and used the place prompt (placeAltTextPrompt) to generate the location name, coordinates, and architectural features without fabricating unverified coordinates. The place system instruction (placeSystemInstruction) directs the model to use the Google Search tool to verify the official place name and coordinates and to find a real-world fact based on the generated tags.

Caption: Output for Kek Lok Si Temple in Penang, Malaysia when classified as place.

AiModelFactory reuse initialized model instances in memory.main.js) await import('@mediapipe/tasks-decision') inside URL.revokeObjectURL() after

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @firebase ai logic 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/stop-using-one-promp…] indexed:0 read:9min 2026-11-22 · —