Improving Contextual Image Retrieval from a Local Dataset Using CLIP or SigLIP 2

A developer building a local image-retrieval system with CLIP ViT-B/32 and SigLIP 2 reported that image-to-image queries return visually similar images but miss contextually related ones — a car query returns another car but not a bus or road — across a test dataset of roughly 39 images. The suggested fix is to index the images twice, once by the SigLIP image vector and once by a caption generated offline with a small local captioner, then fuse both rankings for an image query using the query image's own caption, avoiding per-query VLM cost. The developer said Gemini produced promising contextual rankings when given candidate images but preferred a lighter solution than a large generative VLM, and found LocalLens's text search worked well while its image-to-image results were inconsistent.

Hi everyone, I am working on a project that retrieves visually similar and contextually related images from a local image folder using either a text query or a query image. I have experimented with CLIP ViT-B/32 and SigLIP 2. They retrieve relevant results for some queries, but image-to-image retrieval sometimes returns only visually similar images instead of contextually related ones. For example, a car-image query may return another car but not a bus or road from the dataset. I also tested Gemini by providing candidate images, and it produced promising contextual rankings. However, I would prefer a lighter solution rather than depending on a large generative VLM. Additionally, I came across LocalLens , an existing local image-search application on GitHub, and decided to try it with my image folder using SigLIP 2. Its text search worked well for some queries, but image-to-image results were inconsistent for others. My current test dataset contains approximately 39 images. My questions: 1. Is this expected behaviour for embedding-based retrieval, or could it indicate an implementation or threshold issue? 2. What approaches are suitable for retrieving both visually similar and contextually related images? 3. Would query expansion, image captions, metadata, or a lightweight reranking model help? 4. Are there existing implementations or research papers you would recommend for this use case? I would appreciate suggestions for a controlled evaluation and a practical next step. Thank you That is expected from a single embedding: cosine closeness mostly tracks what an image looks like, so a car query returns cars — a bus or a road is related by scene and function, which that vector barely separates. Index your 39 images twice, once by the SigLIP image vector and once by a caption written once offline with a small local captioner, and fuse both rankings for an image query, using its own caption for the second — bus and road can win in caption space, with no per-query VLM cost. Check it by hand-labelling a few queries with two separate gold lists, visually similar and contextually related, and scoring each method against both: a change counts only if it lifts the related list without collapsing the similar one.