Sub-Semantic Image Segmentation

Researchers propose sub-semantic image segmentation, a new category that partitions images into stable appearance patterns described by language rather than naming whole objects. They introduce DETECTURE, a system that overcomes failure modes in coupling vision-language models with segmentation backbones, and present TextureADE, a new dataset derived from ADE20K. DETECTURE achieves state-of-the-art performance on multiple datasets.

arXiv:2606.14754v1 Announce Type: new Abstract: Images can be segmented based on visual cues i.e., texture segmentation or into objects i.e., semantic segmentation . We propose a new category of sub-semantic image segmentation that blurs the line between the two. In sub-semantic image segmentation, language is not used to name whole objects. Instead, it is used to partition an image into stable appearance patterns that can be described by language. To do that, we couple a general-purpose vision-language model to SAM 3, a promptable segmentation backbone whose native text pathway can ground rich descriptions into masks. Simple coupling fails for a number of reasons that we identify in the paper, and we overcome them by introducing DETECTURE that resolves three concrete failure modes -- language leakage between texture regions, prompt competition inside the segmentation backbone, and semantic distortion at the language-to-mask interface. Since there is no dataset of sub-semantic image segmentation, we introduce one, termed TextureADE. The new dataset is derived from the ADE20K dataset using a system we designed. We compare DETECTURE to a number of baselines and find that it achieves the strongest performance on several datasets using different metrics. Code is available at https://github.com/Scientific-Computing-Lab/TextureDetecture.