How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?

A study of six self-supervised geospatial foundation models found that model rankings change across tasks and adaptation settings, with task-relevant information more accessible in intermediate transformer blocks than final-layer embeddings. The research also showed that downstream adaptation settings like decoder design can be as impactful as model choice, and fine-tuning effects are localized to specific layers.

arXiv:2606.13896v1 Announce Type: new Abstract: Self-supervised geospatial foundation models GeoFMs learn transferable representations from remote sensing data, but their downstream behavior is difficult to characterize. We study six representative GeoFMs spanning joint-embedding, reconstruction, and multimodal pretraining families, and evaluate transfer across classification, regression, and segmentation benchmarks under different label availability and downstream pipelines. We find that model rankings change across tasks and adaptation settings. Layerwise probing shows that, in most cases, task-relevant information is more accessible in intermediate transformer blocks compared to final-layer embeddings, and that GeoFMs exhibit distinct depthwise profiles. In segmentation case studies on PASTIS and Sen1Floods11, downstream adaptation settings such as decoder design and fine-tuning can be as impactful as the choice of GeoFM, and standard dense-prediction heads may be poorly aligned with how GeoFMs organize information over depth. Finally, CKA analysis on case studies shows that fine-tuning does not rewrite GeoFMs uniformly across depth, and the strongest changes are localized to the first linear layer of the MLP in ViT blocks. These results help explain why GeoFM rankings shift across benchmarks and motivate more representation-aware evaluation and adaptation strategies.