Similar but Different: A Benchmark for Measuring Whether Models Actually Use Multispectral Bands

The broad availability of multispectral data is one thing that separates satellite-based machine learning from mainstream computer vision. Sentinel-2 L2A data has 12 surface-reflectance bands, which include visible bands (RGB), red edge, near-infrared (NIR), and shortwave infrared (SWIR). The non-visible bands carry information that distinguishes vegetation types, water, and soil classes that may look similar in RGB — but do pre-trained geospatial foundation models actually take advantage of these bands? We made this benchmark dataset, Similar-but-Different, to help isolate this question.

Many existing satellite imagery benchmark datasets cannot answer our question. For example, on EuroSAT, an RGB-only model matches a full multispectral model,1 because visible color and texture largely separate classes such as “residential,” “sea/lake,” and “forest.” When RGB already solves the task, an ablation for NIR or SWIR tells you little about reliance on those bands.

This is the same motivation we had for creating Seeing the Roads Through the Trees. In that dataset we used roads hidden under tree canopy to test whether segmentation models use long-range spatial context. Here we apply the same recipe to spectral channels: build a dataset where the visible bands carry little class information, so accuracy above the RGB ceiling has to come from the rest of the spectrum.

Our Similar-but-Different dataset contains 30,927 Sentinel-2 L2A patches from 2,709 source scenes across ten ESA WorldCover classes. The selection keeps patches whose true-color statistics sit close to patches of

Spoiler:on a benchmark built so that the answer lives in the non-visible bands, most pre-trained geospatial foundation models barely reach for them. Handed all 12 bands instead of RGB, several gain fewer than 8 points of accuracy, and a plainly fine-tuned ResNet-18 beats every frozen foundation model but one.

We build the dataset in four steps, using the Microsoft Planetary Computer:

uint8

digital numbers) and the distance between their per-channel RGB standard-deviation vectors is ≤ 3 (a texture check). Among those candidates, we keep pairs whose mean NIR (B08) differs by ≥ 1,000 (uint16

surface-reflectance units). Every surviving patch has at least one RGB doppelganger with a The selection produces the cross-class pairs shown in the figure near the top of this post. Within each row the two true-color chips pass the 2-DN RGB-mean candidate threshold, but their B08 NIR means differ by at least 1,000 surface-reflectance units and they carry different WorldCover labels.

The pattern holds in aggregate too. In mean RGB values, the classes overlap along a common brightness axis, while in NIR/SWIR space they pull apart.

The gallery below shows some example patches per class. Few distinct objects stand out because these chips cover only 320 m on a side at 10 m resolution. The RGB view here mostly shows color and texture.

The final dataset has ten classes, unevenly distributed as land cover is in the wild:

We stratify the fixed split (24,741 train, 3,093 validation, and 3,093 test patches) so that no Sentinel-2 scene contributes patches to more than one split. We publish the dataset as 12×32×32 uint16

GeoTIFFs with geotransforms, WorldCover statistics, and the fixed split file on Hugging Face.

We evaluate three groups of pre-trained models: RGB-only backbones (a supervised ImageNet ResNet-50, DINOv3, DINOv3-SAT, and SSL4EO’s RGB checkpoint), fixed-input multispectral backbones (SSL4EO’s 13-band checkpoint and CROMA), and band-flexible models that accept arbitrary channel subsets (DOFA, Panopticon, and OlmoEarth). We run each model on the band sets it supports: RGB (B04/B03/B02), RGB+NIR (+ B08), and all 12 bands, so reading across a row of the tables is a band ablation. Each wrapper resizes inputs to the model’s native resolution and applies its default normalization.2

For every foundation-model row we use the torchgeo-bench frozen-probe recipe: we freeze the backbone, extract patch embeddings, and train an L-BFGS logistic-regression probe.3

The first two rows are image-statistics baselines: a linear model and a k-NN classifier (k=5) that we train on per-band means and standard deviations (six numbers for RGB, 24 for the full stack), the closest view to the selection filter. The ResNet-18 row is a supervised reference — we fine-tune it end-to-end from ImageNet weights on each band set, sweeping learning rate and class weighting and selecting on the validation split. For scale, always guessing the majority class scores 0.19. First, the standard per-patch test accuracy:

n/a marks a band set unavailable for that checkpoint. We fine-tune ResNet-18 end-to-end; every other deep row is a frozen linear probe.

The dataset construction starts from visually similar patches, so we also evaluate on the hard clusters that ship with the dataset. The eval/test_hard_clusters.parquet file contains 53 test-set clusters (121 patches, sizes 2–4). Each cluster is a connected component of the RGB mean/std similarity graph and spans at least two WorldCover labels. The table below shows what we call the

This cluster table makes the trends we observe in the first table sharper, because the hard clusters are exactly the cases the dataset targets. We find that the image-statistics baselines bottom out where they should — the linear model on RGB statistics correctly classifies only 1 of the 53 clusters (0.02). The deep learning models with RGB-only inputs sit well above that floor because the filter constrains only color statistics; within-patch texture still carries class signal, and pixel-level models can use it. DINOv3-SAT has the strongest RGB-only probe in both views (0.735 per-patch, 0.53 cluster), but all-band multispectral models perform better as expected. CROMA and Panopticon resolve 37 of 53 hard clusters, and OlmoEarth resolves 42 of 53.

Every band-flexible model achieves better performance in both metrics as we add bands: from RGB to the full stack, DOFA-Base gains 6.9 points of per-patch accuracy, DOFA-Large 5.7, Panopticon 7.8, and OlmoEarth 24.3, and the cluster metric stretches each gap further (OlmoEarth goes from 0.30 to 0.79).

OlmoEarth also illustrates a missing-band tradeoff. It has the best performance when using all bands in both tables, but has below-average RGB-only results. Its pre-training masks Sentinel-2 at the band-set level, and the finest band set bundles RGB with NIR (B02/B03/B04/B08), so it never sees an RGB-only input during pre-training. RGB+NIR, on the other hand, is exactly its 10 m band set, which helps explain the jump from 0.624 to 0.812 per-patch. Panopticon’s pre-training instead builds views from randomly subsampled channels (as few as one per view), so sparse band subsets stay in distribution and it drops less. Whether a model exploits extra bands and whether it tolerates missing ones are different capabilities, and the same ablation protocol measures both.

Use Similar-but-Different as a small, targeted probe. If you’re building or evaluating a multispectral model, a frozen-probe run will take minutes and tell you whether your model’s spectral capability shows up when the visible bands stop carrying the answer.

Also note that the labels we use here are WorldCover per-pixel modes, and therefore inherit WorldCover’s errors (we keep only patches at least 70% one class, but residual noise remains). Treat accuracy on this dataset as a measurement of band reliance rather than operational land-cover accuracy.

Finally, we think more people should build small, capability-targeted datasets like this! The recipe generalizes beyond spectra: generate a large candidate pool, then keep the examples where a chosen restricted view of the input disagrees with the label. Restrict color statistics and you probe spectral reliance (this post); restrict local appearance and you probe spatial context (Chesapeake RSC); restrict to a single date and you could probe temporal reasoning the same way.

We ran the same fine-tuned ResNet-18 recipe we use below on EuroSAT and got 0.986 test accuracy from RGB alone and 0.986 from all 13 bands.↩︎

For OlmoEarth-v1-Base that means upsampling our 32×32 chips to 64×64 and using patch size 8.↩︎

We sweep the inverse-regularization strength C

over 40 values on the validation split, refit on train+val, and score on test.↩︎

@online{robinson2026,
  author = {Robinson, Caleb and Corley, Isaac},
  title = {Similar but {Different:} {A} {Benchmark} for {Measuring}
    {Whether} {Models} {Actually} {Use} {Multispectral} {Bands}},
  date = {2026-07-07},
  url = {https://geospatialml.com/posts/similar-but-different/},
  langid = {en}
}