Hi everyone! I’m working on a non-profit project that helps shelter dogs find a new home. People can upload a photo of a dog they like, and we show them the dogs from animal shelters that look most similar to it. The goal isn’t to find the exact same dog again, but dogs that a person would say “look alike” (similar colour, coat, build, face, and so on).
What already works well
For purebred dogs, this works really well already. We compared several pretrained image models without any fine-tuning: we compute an embedding for each photo and rank the other dogs by cosine similarity. The models we tested were DINOv2 small, DINOv2 base (facebook/dinov2-base), SigLIP ( google/siglip-base-patch16-224), SigLIP 2 ( google/siglip2-base-patch16-224), CLIP ( openai/clip-vit-base-patch32), MegaDescriptor ( BVRA/MegaDescriptor-T-224), and a DINOv2 small model fine-tuned for animal identification (AvitoTech/DINO-v2-small-for-animal-identification) on the PetFace dataset. The last two are licensed for non-commercial use only, so we only used them for comparison.
DINOv2 small came out on top. It predicts the breed almost perfectly and is also very good at recognising the same dog again in a different photo. So for purebred dogs, the most similar matches already make a lot of sense.
Where we’re stuck: mixed breeds
Most dogs in shelters are mixed breeds, and that’s where it falls apart. When we checked the results by hand, the model’s top match was often not the dog a human would have picked as most similar. It does better than chance, but only agrees with the human in roughly a third of the cases.
We tried a few things to fix this, but none of them really helped:
Our questions
We’re happy to use a GPU if it’s needed. We’re just looking for the most cost-effective option.
Thanks a lot for any ideas or pointers!