I upload images of places, wrestlers, and general objects to the application. When up images of people, I want the Gemini model to generate alt text about their appearance, posture, and attire. When up images of places, I want the Gemini model to generate alt text with their official names, coordinates, and locations. To support this, I installed the @mediapipe/tasks-decision library, which uses the EmbeddingGemma 2 model (embeddinggemma-2-text-vision-440m-litert-lm) to classify the category of a picture ( people, general, or place). The category decides which Firebase AI Logic GenerativeModel and prompt generate the text response.
ImageClassifierService checks the browser's CacheStorage (decision-model-v1) for the EmbeddingGemma 2 model, downloads and caches it via HttpClient on a cache miss, and uses DecisionMaker with the provided choices to determine the image's category (people, general, or place). AiModelOptions (optional systemInstruction, response schema, and tools) from CATEGORY_MODEL_CONFIGS. GenerativeModel Lookup & Content GenerationVisionService passes options to AiModelFactory.getModel(options) to reuse or create the matching Firebase AI Logic generateContent([prompt, imagePart]) with the category's prompt to generate the text response.
flowchart TD
IMG["Selected Image (File)"] --> PREP["preprocessImageForVision() -> imageBytes + base64"]
PREP --> CLS["ImageClassifierService.classifyImage(imageBytes)"]
CLS --> CACHE_CHK{"Local model in CacheStorage?"}
CACHE_CHK -- "Miss" --> DL["HttpClient GET -> cache.put() -> Blob URL"]
CACHE_CHK -- "Hit" --> BLOB["cache.match() -> Blob URL"]
DL & BLOB --> DM["DecisionMaker.evaluateChoice(imageBytes, IMAGE_CATEGORY_QUESTION)"]
DM --> CAT{"Best Category"}
CAT -- "people" --> CFG_P["CATEGORY_MODEL_CONFIGS['people']"]
CAT -- "general" --> CFG_G["CATEGORY_MODEL_CONFIGS['general']"]
CAT -- "place" --> CFG_L["CATEGORY_MODEL_CONFIGS['place']"]
CFG_P & CFG_G & CFG_L --> FAC["AiModelFactory.getModel(options)"]
FAC --> GEN["aiModel.generateContent([prompt, imagePart])"]
@mediapipe/tasks-decision, Serving WASM Assets, and Adding decisionModelPath to Remote Config
I installed the latest @mediapipe/tasks-decision package (1.2.3), which provides the API to evaluate the category of an image input.
npm install @mediapipe/tasks-decision
angular.json
The decision tasks require the WASM files at runtime. Instead of pointing to the WASM files on a CDN, I used the files installed in node_modules/@mediapipe/tasks-decision/wasm. Since the service cannot load files directly from node_modules in the browser, I configured angular.json to copy node_modules/@mediapipe/tasks-decision/wasm to /wasm (WASM_PATH = '/wasm') so the application can access them.
// angular.json (projects.ng-firebase-tts.architect.build.options.assets)
{
"glob": "**/*",
"input": "node_modules/@mediapipe/tasks-decision/wasm",
"output": "/wasm"
}
I added the decisionModelPath parameter to Firebase Remote Config and public/remote-config-defaults.json, defaulting to https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm for on-device image classification. I also added the decisionModelPath property to AppRemoteConfig, which ConfigService updates when fetching the model path dynamically from Firebase Remote Config.
// public/remote-config-defaults.json
{
"decisionModelPath": "https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm",
...
}
// src/app/core/firebase/config.service.ts (decisionModelPath in #appConfig & fetchRemoteConfig)
#appConfig: AppRemoteConfig = {
...
decisionModelPath: remoteConfigDefaults.decisionModelPath,
};
...
this.#appConfig = {
...
decisionModelPath: getValue(rc, 'decisionModelPath').asString(),
};
ImageClassifierService encapsulates the image-classification logic that VisionService uses in step one of the image-analysis pipeline:
#cachedModelPath``#decisionMakerPromise`` Promise<DecisionMaker> so the service initializes the local model at most once per active model path.DestroyRef.onDestroy()`` closeActiveDecisionMaker() before the application shuts down to avoid a WASM memory leak.
// src/app/core/vision/image-classifier.service.ts (fields & lifecycle)
export class ImageClassifierService {
#window = inject(WINDOW);
#configService = inject(ConfigService);
#destroyRef = inject(DestroyRef);
#decisionMakerPromise: Promise<DecisionMaker> | null = null;
#cachedModelPath: string | null = null;
#http = inject(HttpClient);
constructor() {
this.#destroyRef.onDestroy(() => this.closeActiveDecisionMaker());
}
initializeDecisionMaker & resolveCachedModelUrl)
initializeDecisionMaker initializes the MediaPipe WASM runtime, loads the cached model, and prewarms the classification options:
FilesetResolver from FilesetResolver.forDecisionTasks(WASM_PATH) to load the WASM files from /wasm. this.resolveCachedModelUrl()) DECISION_MODEL_CACHE_NAME) bucket in modelAssetPath:
Blob, stores it in the blob: URL.DecisionMaker.createFromOptions() to create the delegate ('GPU' or 'CPU'), backendType: 3 (the LiteRT-LM backend), and MAX_NUM_TOKENS, then calls prewarmChoice(IMAGE_CATEGORY_QUESTION) to prewarm the WASM session with the URL.revokeObjectURL(resolvedModelPath) revokes the temporary
// src/app/core/vision/image-classifier.service.ts (initializeDecisionMaker & resolveCachedModelUrl)
private async initializeDecisionMaker(modelAssetPath: string, delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
const { DecisionMaker, FilesetResolver } = await import('@mediapipe/tasks-decision');
const wasmFileset = await FilesetResolver.forDecisionTasks(WASM_PATH);
const resolvedModelPath = await this.resolveCachedModelUrl(modelAssetPath);
try {
const decisionMaker = await DecisionMaker.createFromOptions(wasmFileset, {
baseOptions: {
modelAssetPath: resolvedModelPath,
delegate,
},
backendType: 3,
maxNumTokens: MAX_NUM_TOKENS,
});
await decisionMaker.prewarmChoice(IMAGE_CATEGORY_QUESTION);
return decisionMaker;
} finally {
if (resolvedModelPath.startsWith('blob:')) {
URL.revokeObjectURL(resolvedModelPath);
}
}
}
private async resolveCachedModelUrl(modelAssetPath: string): Promise<string> {
const cacheStorage = this.#window?.caches;
if (!cacheStorage) {
return modelAssetPath;
}
try {
const cache = await cacheStorage.open(DECISION_MODEL_CACHE_NAME);
const keys = await cache.keys();
await Promise.all(keys.filter((req) => req.url !== modelAssetPath).map((req) => cache.delete(req)));
let response = await cache.match(modelAssetPath);
if (!response) {
const httpResponse = await firstValueFrom(
this.#http.get(modelAssetPath, { observe: 'response', responseType: 'blob' }),
);
const downloadedBlob = httpResponse.body;
if (!downloadedBlob) {
return modelAssetPath;
}
response = new Response(downloadedBlob, {
status: httpResponse.status,
headers: {
'Content-Type': httpResponse.headers.get('Content-Type') ?? 'application/octet-stream',
},
});
await cache.put(modelAssetPath, response.clone());
}
const blob = await response.blob();
return URL.createObjectURL(blob);
} catch {
return modelAssetPath;
}
}
Caption: The downloaded embeddinggemma-2-text-vision-440m-litert-lm model stored inside the browser's decision-model-v1 CacheStorage.
getValidDecisionMaker & closeActiveDecisionMaker)
getValidDecisionMaker()) initializeDecisionMaker() to construct a new this.#decisionMakerPromise, calls decisionMaker.close() to free the WASM engine and model memory, and resets null.
// src/app/core/vision/image-classifier.service.ts (getValidDecisionMaker & closeActiveDecisionMaker)
private async getValidDecisionMaker(delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
const modelAssetPath = this.#configService.appConfig.decisionModelPath;
if (this.#decisionMakerPromise && this.#cachedModelPath !== modelAssetPath) {
await this.closeActiveDecisionMaker();
}
if (!this.#decisionMakerPromise) {
this.#cachedModelPath = modelAssetPath;
this.#decisionMakerPromise = this.initializeDecisionMaker(modelAssetPath, delegate).catch((error) => {
this.#decisionMakerPromise = null;
this.#cachedModelPath = null;
throw error;
});
}
return this.#decisionMakerPromise;
}
private async closeActiveDecisionMaker(): Promise<void> {
if (!this.#decisionMakerPromise) {
return;
}
try {
const decisionMaker = await this.#decisionMakerPromise;
decisionMaker.close();
} catch {
// Ignore initialization or WASM teardown errors during cleanup
} finally {
this.#decisionMakerPromise = null;
this.#cachedModelPath = null;
}
}
IMAGE_CATEGORY_QUESTION)
Before calling DecisionMaker, I define the classification rubric and token budget in image-classifier.const.ts:
instructions) normalizePrior: false) normalizePrior to false because the input is an image ({ imageBytes }) rather than text; null-context prior normalization is intended for text-only choice prompts.MAX_NUM_TOKENS = 512) 512 tokens because EmbeddingGemma 2 maps a single image to context, 4096-token window in browser memory. temperature: 0.05)
// src/app/core/vision/image-classifier.const.ts
import type { ChoiceQuestion } from '@mediapipe/tasks-decision';
export const WASM_PATH = '/wasm';
export const MAX_NUM_TOKENS = 512;
export const DECISION_MODEL_CACHE_NAME = 'decision-model-v1';
export const IMAGE_CATEGORY_QUESTION: ChoiceQuestion = {
context: 'Analyze the uploaded image for accessibility alt-text routing.',
instructions:
'Select the primary subject category of this image. If an outdoor landscape, architecture, monument, landmark, or travel destination is visible behind people, select place. Otherwise, if any person, masked figure, or group of people is visible, select people.',
options: [
{
label: 'place',
description: 'An outdoor landscape, natural scenery, architecture, monument, landmark, or travel destination.',
},
{
label: 'people',
description: 'A person, portrait, face, masked or costumed figure, performer, or group of people.',
},
{
label: 'general',
description: 'A close-up of food, drink, animal, pet, vehicle, or household object.',
},
],
temperature: 0.05,
normalizePrior: false,
};
resolveDelegate)
'gpu' in this.#window.navigator) and returns .litertlm Modelsundefined when WebGPU is unavailable and the model path ends with classifyImage() to fall back to 'general').
// src/app/core/vision/image-classifier.service.ts (resolveDelegate)
private resolveDelegate(): 'GPU' | 'CPU' | undefined {
const hasWebGpu = Boolean(this.#window?.navigator && 'gpu' in this.#window.navigator);
if (hasWebGpu) {
return 'GPU';
}
if (this.#configService.appConfig.decisionModelPath.endsWith('.litertlm')) {
return undefined;
}
return 'CPU';
}
classifyImage)
decisionMaker.evaluateChoice() to evaluate imageBytes is empty,
// src/app/core/vision/image-classifier.service.ts (classifyImage)
async classifyImage(imageBytes: Uint8Array): Promise<ImageSubjectCategory> {
if (imageBytes.byteLength === 0) {
return 'general';
}
const delegate = this.resolveDelegate();
if (!delegate) {
return 'general';
}
try {
const decisionMaker = await this.getValidDecisionMaker(delegate);
const { probabilities } = await decisionMaker.evaluateChoice({ imageBytes }, IMAGE_CATEGORY_QUESTION);
let bestCategory: ImageSubjectCategory = 'general';
let highestProbability = -1;
for (const [key, probability] of Object.entries(probabilities)) {
if (probability > highestProbability && (key === 'place' || key === 'people' || key === 'general')) {
highestProbability = probability;
bestCategory = key;
}
}
return bestCategory;
} catch (error) {
console.warn('[ImageClassifierService] On-device classification failed, falling back to general:', error);
return 'general';
}
}
prompt and peopleSystemInstruction and placeSystemInstruction to define the personas and verification rules of the | Category | systemInstruction | prompt | tools |
|---|---|---|---|
| general | None (uses prompt instructions directly) | altTextPrompt | [{ googleSearch: {} }] |
| people | peopleSystemInstruction (strict anti-guessing rule for faces/masks) | peopleAltTextPrompt | [{ googleSearch: {} }] |
| place | placeSystemInstruction (landmark & coordinate verification, no fabrication) | placeAltTextPrompt | [{ googleSearch: {} }] |
// src/app/core/vision/vision.const.ts
export const CATEGORY_MODEL_CONFIGS: Record<ImageSubjectCategory, CategoryModelConfig> = {
general: {
options: {
schema: ImageAnalysisSchema,
tools: [{ googleSearch: {} }],
},
prompt: altTextPrompt,
},
people: {
options: {
schema: ImageAnalysisSchema,
tools: [{ googleSearch: {} }],
systemInstruction: peopleSystemInstruction,
},
prompt: peopleAltTextPrompt,
},
place: {
options: {
schema: ImageAnalysisSchema,
tools: [{ googleSearch: {} }],
systemInstruction: placeSystemInstruction,
},
prompt: placeAltTextPrompt,
},
};
Full prompt and systemInstruction strings: src/app/core/vision/vision.const.ts
this.#imageClassifierService. this.#imageClassifierService.classifyImage(imageBytes) to determine the category of the preprocessed Uint8Array). category to retrieve the this.#aiModelFactory.getModel(options) to obtain a aiModel.generateContent([prompt, imagePart]) to generate the text response.
// src/app/core/vision/vision.service.ts (analyzeImage)
readonly #imageClassifierService = inject(ImageClassifierService);
async analyzeImage(image: File): Promise<ImageAnalysisResponse> {
if (!image) {
throw Error('image is required to generate texts.');
}
const { data, mimeType, optimizationMetrics, imageBytes } = await preprocessImageForVision(image, this.#window);
const category = await this.#imageClassifierService.classifyImage(imageBytes);
const { options, prompt } = CATEGORY_MODEL_CONFIGS[category] || CATEGORY_MODEL_CONFIGS['general'];
const imagePart: InlineDataPart = {
type: 'inlineData',
inlineData: {
data,
mimeType,
},
};
const aiModel = await this.#aiModelFactory.getModel(options);
const result = await aiModel.generateContent([prompt, imagePart]);
...
I uploaded an image of six Japanese wrestlers, and the classifier returned the people category. The people system instruction tells the model to use the Google Search tool to verify visible text clues and find a real-world fact based on the generated tags. The people prompt asks the model to include a person's name in the alt text only when the name is printed in the image or 100% certain and otherwise to describe the person's features, posture, and attire.
Caption: Output for an image of six Japanese wrestlers when classified as people.
I uploaded an image of a butterfly, and the classifier returned the general category. The general prompt (altTextPrompt) generated the alt text and tags and used the Google Search tool to verify a real-world fact based on the tags.
Caption: Output for an image of a butterfly when classified as general.
I uploaded an image of Kek Lok Si Temple in Penang, Malaysia, and analyzed the image to generate the alt text and tags. DecisionMaker classified the image input as place and used the place prompt (placeAltTextPrompt) to generate the location name, coordinates, and architectural features without fabricating unverified coordinates. The place system instruction (placeSystemInstruction) directs the model to use the Google Search tool to verify the official place name and coordinates and to find a real-world fact based on the generated tags.
Caption: Output for Kek Lok Si Temple in Penang, Malaysia when classified as place.
AiModelFactory reuse initialized model instances in memory.main.js) await import('@mediapipe/tasks-decision') inside URL.revokeObjectURL() after