Augmentation with Reinforcement Transformation for Low-textured Scientific Sketch Image Retrieval

A new arXiv paper (2611.05408v1) proposes Augmentation with Reinforcement Transformation (ART), a method that treats image transformations as arms in a multi-armed bandit problem to adaptively build reward-guided view banks for self-supervised fine-tuning of pretrained models on low-textured scientific sketches. On Florida manatee sketches provided by Mote Marine Laboratory, ART reached 71.40% average Top-1 accuracy for transformed-scar retrieval versus 68.83% for random augmentation selection, and 55.84% for location-aware sketch retrieval versus 18.18% for frozen CLIP. The approach targets retrieval tasks in wildlife conservation, archaeological illustration and forensic composites, where sketches lack the surface textures modern deep learning models rely on.

arXiv:2611.05408v1 Announce Type: new Abstract: Low-textured sketch images are widely used in scientific applications, including wildlife conservation, archaeological illustration, and forensic composites. Learning from and retrieving such images is challenging because they lack detailed surface textures and visual variations commonly exploited by modern deep learning models. This paper proposes Augmentation with Reinforcement Transformation ART to fine-tune pretrained models to learn effective representations for accurate sketch image retrieval. Our key observation is that sketches are particularly sensitive to image transformations, such as rotation, flipping, zooming, and stretching. While some transformations preserve their structural integrity, others may introduce distortions that hinder representation learning. Instead of randomly choosing transformations to construct fine-tuning views, ART formulates augmentation selection as a multi-armed bandit MAB problem, where transformations are treated as arms and their effectiveness is continuously evaluated through rewards. This enables ART to adaptively construct reward-guided view banks before self-supervised fine-tuning; the subsequent representation loss does not feed back to ART. On Florida manatee sketches provided by Mote Marine Laboratory, ART achieves 71.40% average Top-1 accuracy for transformed-scar retrieval, compared with 68.83% for random augmentation selection, and 55.84% for location-aware sketch retrieval, compared with 18.18% for frozen CLIP.