Semantic Image Optimizer

An e-commerce image optimization method using Google's AI models and vector embeddings iteratively generated and scored product alt text, improving semantic similarity from 59% to 83% over ten attempts. The best caption, 'Four colorful fabric wristbands: red, blue, gray, and light green, are arranged diagonally on a light background,' was achieved for a Google Fitbit Air product image. The approach is part of the AI Influence app Auxy.

An overview of an e-commerce image optimization method using Google's AI models and vector embeddings to iteratively generate and score product alt text. I saw a really cool post written by Gianluca Fiorelli shared on LinkedIn today https://www.linkedin.com/feed/update/urn:li:activity:7498709392712790016/ and I'd love to share how we approach image optimization for e-commerce and multi-modal AI search. The above image is located on the official Google Fitbit Air https://store.google.com/au/product/google fitbit air page. We upload the image alongside the source URL and the original alt text. <img data-component-name="media area image" jscontroller="Tbys9b" jsaction="rcuQ6b:npT2md;k0bule:jAmaef" class="WeYPde" src="https://lh3.googleusercontent.com/2B ulFLMDz6jJXNnoktx QgeEMEDAt-UfvbmEmTitaG4Y1eUt6qIGblB7CiHJHnKw0lraLHMu1rC1RyOi-aF-EBr31WtlNY52A=w1156-h924-rw-nu" alt="" width="1156" height="924" loading="eager" decoding="auto" fetchpriority="auto" Our optimizer uses Google's latest model to generate vector embeddings of multimedia inputs to provide our optimization algorithm with semantic similarity scoring. Here is the full list of attempts: | Step | Match | Δ | Caption | |---|---|---|---| | 0 | 59% | . | | | 1 | 78% | +19% | Four colorful wristbands are displayed diagonally on a light background. | | 2 | 82% | +4% | Four colored fabric wristbands, red, blue, gray, and light green, are arranged diagonally on a white background. | | 3 | 81% | -2% | Four woven fabric wristbands, red, blue, gray, and light green, are arranged diagonally on a light white background. | | 4 | 79% | -2% | Four woven fabric wristbands: red, blue, gray, and light green, diagonally on white. | | 5 | 80% | +1% | Four colorful woven fabric wristbands, red, blue, gray, and light green, are arranged diagonally on a white background. | | 6 | 81% | +1% | Four colored fabric wristbands in red, blue, gray, and light green, shown diagonally on white. | | 7 | 83% | +2% | Four colorful fabric wristbands, red, blue, gray, and light green, are arranged diagonally on a light background. | | 8 | 82% | -1% | Four colorful fabric wristbands, red, blue, gray, and light green, are arranged diagonally on a bright white background. | | 9 | 83% | +1% | Four colorful fabric wristbands: red, blue, gray, and light green, are arranged diagonally on a light background. | | 10 | 83% | 0% | Four fabric wristbands: red, blue, gray, and light green, arranged diagonally on a light background. | After ten iterations the best matching alt text is: Four colorful fabric wristbands: red, blue, gray, and light green, are arranged diagonally on a light background. Image optimizer is small part of our AI Influence app, Auxy https://app.auxy.com/ .