A Tutorial on GeoAI: Designing Footprint Extraction from NAIP Imagery Using U-Net, Grounding DINO, SAM, and Mask R-CNN

A new tutorial from the open-source GeoAI project demonstrates a complete workflow for extracting building footprints from NAIP aerial imagery using U-Net with a ResNet-34 encoder, Grounding DINO, SAM, and Mask R-CNN. The tutorial, published on GitHub, covers environment setup, data preparation, model training, inference, and evaluation, including zero-shot segmentation and comparison with a pretrained Mask R-CNN model. It also shows how to extend the pipeline to real-world areas using NAIP imagery from Microsoft Planetary Computer and building labels from Overture Maps.

In this tutorial, we design a complete GeoAI https://github.com/opengeos/geoai workflow for extracting building footprints from high-resolution NAIP aerial imagery. We begin by configuring the geospatial deep learning environment, downloading raster imagery and vector labels, and inspecting their spatial properties before generating georeferenced image chips and segmentation masks. We then train a U-Net model with a ResNet-34 encoder, evaluate its learning behavior, and apply sliding-window inference to an unseen scene. Beyond semantic segmentation, we convert predicted masks into cleaned and regularized building polygons, calculate IoU and F1 metrics, explore zero-shot segmentation with Grounding DINO and SAM, and compare the results with a pretrained Mask R-CNN instance segmentation model. We also demonstrate how the same pipeline extends to real-world areas using NAIP imagery from Microsoft Planetary Computer and building labels from Overture Maps. python import os import subprocess import sys import time import warnings warnings.filterwarnings "ignore" IN COLAB = "google.colab" in sys.modules def pip install packages, quiet=True : """Install packages with pip from inside the notebook process.""" cmd = sys.executable, "-m", "pip", "install", "--upgrade" if quiet: cmd.append "-q" subprocess.run cmd + list packages , check=False try: import geoai except ImportError: print " Installing geoai-py and friends takes ~2-4 minutes on Colab ..." pip install "geoai-py", "segmentation-models-pytorch", "buildingregulariser", try: import geoai except Exception as e: raise SystemExit f"Import failed after install {e} .\n" "= Runtime Restart session, then re-run this cell. " "The install is cached, so it will be fast the second time." import geopandas as gpd import matplotlib.pyplot as plt import numpy as np import rasterio import torch from rasterio.plot import plotting extent from IPython.display import display print f"geoai : {geoai. version }" print f"torch : {torch. version }" print f"CUDA available: {torch.cuda.is available }" if torch.cuda.is available : print f"GPU : {torch.cuda.get device name 0 }" else: print " No GPU detected. Training will still run but be much slower." print " Colab: Runtime Change runtime type Hardware accelerator T4 GPU" DEVICE = geoai.get device print f"geoai device : {DEVICE}" CFG = { "tile size": 512, "stride": 256, "buffer radius": 0, "architecture": "unet", "encoder": "resnet34", "encoder weights": "imagenet", "num channels": 3, "num classes": 2, "batch size": 8, "num epochs": 12, "learning rate": 1e-3, "val split": 0.2, "window size": 512, "overlap": 256, "run zero shot": True, "run pretrained": True, "run real aoi": False, } WORK = "/content/geoai tutorial" if IN COLAB else os.path.abspath "geoai tutorial" os.makedirs WORK, exist ok=True os.chdir WORK print f"working dir : {WORK}" def banner text : print "\n" + "=" 92 + f"\n {text}\n" + "=" 92 def timed fn, label : """Run fn , report wall time, never let one step kill the notebook.""" banner label t0 = time.time try: out = fn print f"\n OK {label} — {time.time - t0:.1f}s" return out except Exception as exc: import traceback print f"\n SKIPPED {label}\n{type exc . name }: {exc}" traceback.print exc limit=3 return None HF = "https://huggingface.co/datasets/giswqs/geospatial/resolve/main" train raster url = f"{HF}/naip rgb train.tif" train vector url = f"{HF}/naip train buildings.geojson" test raster url = f"{HF}/naip test.tif" def step1 : train raster = geoai.download file train raster url train vector = geoai.download file train vector url test raster = geoai.download file test raster url for p in train raster, train vector, test raster : print f" {os.path.getsize p / 1e6:8.2f} MB {p}" return train raster, train vector, test raster paths = timed step1, "STEP 1 — Downloading sample NAIP imagery and building labels" TRAIN RASTER, TRAIN VECTOR, TEST RASTER = paths We configure the environment, install the required GeoAI and deep learning libraries, and verify GPU availability. We define the central configuration parameters for dataset creation, model training, inference, and optional processing stages. We then create the working directory, define reusable execution utilities, and download the NAIP imagery and building footprint labels. python def step2 : info = geoai.get raster info TRAIN RASTER for k, v in info.items : print f" {k:<16}: {v}" print "\n--- per-band statistics ---" print geoai.get raster stats TRAIN RASTER print "\n--- vector info ---" vinfo = geoai.get vector info TRAIN VECTOR for k, v in vinfo.items : print f" {k:<16}: {v}" gdf = gpd.read file TRAIN VECTOR print f"\n {len gdf } training buildings | CRS {gdf.crs}" print gdf.head 3 geoai.view vector gdf, raster path=TRAIN RASTER, outline only=True, edge color="yellow", outline linewidth=0.8, figsize= 11, 11 , title="NAIP training scene + building footprints", try: display geoai.view vector interactive TRAIN VECTOR, layer name="Buildings" except Exception as e: print f" interactive map unavailable here: {e} " return gdf LABELS GDF = timed step2, "STEP 2 — Inspecting raster + vector data" TILES DIR = os.path.join WORK, "tiles" def step3 : stats = geoai.export geotiff tiles in raster=TRAIN RASTER, out folder=TILES DIR, in class data=TRAIN VECTOR, tile size=CFG "tile size" , stride=CFG "stride" , buffer radius=CFG "buffer radius" , all touched=True, skip empty tiles=False, quiet=False, n img = len os.listdir f"{TILES DIR}/images" n lbl = len os.listdir f"{TILES DIR}/labels" print f"\n chips: {n img} images / {n lbl} masks" if isinstance stats, dict : tot = max stats.get "total tiles", n img , 1 print f" tiles containing buildings: {stats.get 'tiles with features' } " f" {100 stats.get 'tiles with features', 0 / tot:.1f}% " print f" foreground pixels: {stats.get 'feature pixels' :,}" geoai.display training tiles TILES DIR, num tiles=6, figsize= 18, 6 return stats TILE STATS = timed step3, "STEP 3 — Exporting image chips and label masks" We inspect the raster and vector datasets to understand their coordinate systems, dimensions, statistics, and feature structures. We visualize the building labels over the aerial imagery and generate an interactive map for spatial exploration. We then divide the source imagery into overlapping georeferenced chips and create matching raster masks for model training. MODEL DIR = os.path.join WORK, "models unet" BEST MODEL = os.path.join MODEL DIR, "best model.pth" def step4 : geoai.train segmentation model images dir=f"{TILES DIR}/images", labels dir=f"{TILES DIR}/labels", output dir=MODEL DIR, architecture=CFG "architecture" , encoder name=CFG "encoder" , encoder weights=CFG "encoder weights" , num channels=CFG "num channels" , num classes=CFG "num classes" , batch size=CFG "batch size" , num epochs=CFG "num epochs" , learning rate=CFG "learning rate" , val split=CFG "val split" , save best only=True, early stopping patience=5, verbose=True, print f"\n best checkpoint: {BEST MODEL}" print f" size: {os.path.getsize BEST MODEL / 1e6:.1f} MB" return BEST MODEL timed step4, f"STEP 4 — Training {CFG 'architecture' }/{CFG 'encoder' } " f"for {CFG 'num epochs' } epochs" def step5 : hist path = os.path.join MODEL DIR, "training history.pth" geoai.plot performance metrics history path=hist path, figsize= 15, 5 , verbose=True, save path=os.path.join WORK, "training curves.png" , h = torch.load hist path, weights only=False best ep = int np.argmax h "val iou" + 1 print f"\n best val IoU {max h 'val iou' :.4f} at epoch {best ep}" print " Reading the curves: val loss rising while train loss falls = overfitting;" print " both flat and high = underfitting more epochs, bigger encoder, or more chips ." timed step5, "STEP 5 — Training diagnostics" We train a U-Net semantic segmentation model with a ResNet-34 encoder using the prepared image and mask tiles. We configure the training process with validation splitting, early stopping, checkpoint saving, and performance monitoring. We then load the training history, plot the learning curves, and identify the epoch that produces the highest validation IoU. PRED MASK = os.path.join WORK, "test prediction.tif" PRED PROB = os.path.join WORK, "test probability.tif" def step6 : geoai.semantic segmentation input path=TEST RASTER, output path=PRED MASK, model path=BEST MODEL, architecture=CFG "architecture" , encoder name=CFG "encoder" , num channels=CFG "num channels" , num classes=CFG "num classes" , window size=CFG "window size" , overlap=CFG "overlap" , batch size=4, probability path=PRED PROB, geoai.print raster info PRED MASK, show preview=False geoai.plot prediction comparison original image=TEST RASTER, prediction image=PRED MASK, titles= "NAIP test scene", "Predicted building mask" , figsize= 16, 8 , prediction colormap="viridis", save path=os.path.join WORK, "prediction comparison.png" , with rasterio.open PRED MASK as src: m = src.read 1 px = float abs src.transform.a abs src.transform.e print f" predicted building pixels: {int m 0 .sum :,} " f" {100 m 0 .mean :.2f}% of scene, ~{ m 0 .sum px:,.0f} m2 " timed step6, "STEP 6 — Sliding-window inference on the test scene" VEC RAW = os.path.join WORK, "buildings raw.geojson" VEC ORTHO = os.path.join WORK, "buildings orthogonal.geojson" VEC FINAL = os.path.join WORK, "buildings final.geojson" def step7 : grouped = geoai.region groups PRED MASK, connectivity=2, min size=50, out image=os.path.join WORK, "test prediction cleaned.tif" , clean mask = os.path.join WORK, "test prediction cleaned.tif" raw = geoai.raster to vector clean mask, output path=VEC RAW, threshold=0, min area=15, simplify tolerance=0.5, print f" raw polygons : {len raw }" ortho = geoai.orthogonalize input path=clean mask, output path=VEC ORTHO, epsilon=1.5, min area=15, print f" orthogonalized : {len ortho }" final = geoai.regularization ortho, angle tolerance=12, simplify tolerance=0.4 final = geoai.add geometric properties final, properties= "area", "perimeter", "solidity", "elongation", "orientation" final.to file VEC FINAL, driver="GeoJSON" print f" final footprints : {len final }" print final.head if "area" in final.columns: print "\n footprint area stats m2 :" print final "area" .describe .round 1 .to string fig, axes = plt.subplots 1, 2, figsize= 16, 8 with rasterio.open TEST RASTER as src: rgb = src.read 1, 2, 3 .transpose 1, 2, 0 rgb = np.clip rgb / np.percentile rgb, 99 , 0, 1 ext = plotting extent src for ax, g, t in zip axes, raw, final , "Raw polygonization", "Orthogonalized + regularized" : ax.imshow rgb, extent=ext g.plot ax=ax, facecolor="none", edgecolor="red", linewidth=1.1 ax.set title t ax.set axis off plt.tight layout plt.show try: display geoai.view vector interactive VEC FINAL, layer name="Predicted buildings" except Exception: pass return final FINAL GDF = timed step7, "STEP 7 — Vectorizing and regularizing the predicted footprints" We apply sliding-window inference to an unseen NAIP scene and generate both prediction and probability rasters. We remove small noisy regions, convert the predicted mask into vector polygons, and regularize the footprint geometries to produce cleaner building boundaries. We also calculate geometric properties and compare the raw polygonized results with the orthogonalized and regularized outputs. python def step8 : gt raster = os.path.join WORK, "train gt mask.tif" geoai.vector to raster vector path=TRAIN VECTOR, output path=gt raster, reference raster=TRAIN RASTER, fill value=0, all touched=True, dtype=np.uint8, train pred = os.path.join WORK, "train prediction.tif" geoai.semantic segmentation input path=TRAIN RASTER, output path=train pred, model path=BEST MODEL, architecture=CFG "architecture" , encoder name=CFG "encoder" , num channels=CFG "num channels" , num classes=CFG "num classes" , window size=CFG "window size" , overlap=CFG "overlap" , quiet=True, metrics = geoai.calc segmentation metrics ground truth=gt raster, prediction=train pred, num classes=2, metrics= "iou", "f1" , print "\n --- pixel-wise metrics class 0 = background, class 1 = building ---" for k, v in metrics.items : print f" {k:<10}: {np.round v, 4 }" print "\n Building-class IoU is the number that matters; background IoU is inflated" print " by the huge negative class and always looks great." geoai.plot prediction comparison original image=TRAIN RASTER, prediction image=train pred, ground truth image=gt raster, titles= "Imagery", "Prediction", "Ground truth" , figsize= 18, 6 , save path=os.path.join WORK, "accuracy comparison.png" , return metrics METRICS = timed step8, "STEP 8 — Quantitative accuracy assessment" def step9 : if not CFG "run zero shot" : print " disabled in CFG" ; return chip = os.path.join WORK, "test chip.tif" with rasterio.open TEST RASTER as src: b = src.bounds cx, cy = b.left + b.right / 2, b.bottom + b.top / 2 half = min b.right - b.left , b.top - b.bottom / 6 bbox = cx - half, cy - half, cx + half, cy + half geoai.clip raster by bbox TEST RASTER, chip, bbox=bbox, bbox type="geo" print f" clipped chip: {chip}" sam = geoai.GroundedSAM detector id="IDEA-Research/grounding-dino-tiny", segmenter id="facebook/sam-vit-base", tile size=1024, overlap=128, threshold=0.3, out mask = os.path.join WORK, "zeroshot mask.tif" gdf = sam.segment image input path=chip, output path=out mask, text prompts= "building", "house", "rooftop" , polygon refinement=True, export polygons=True, min polygon area=30, simplify tolerance=1.5, geoai.plot prediction comparison original image=chip, prediction image=out mask, titles= "Chip", "Zero-shot: 'building / house / rooftop'" , figsize= 14, 7 , prediction colormap="viridis", print f" zero-shot objects found: {len gdf if gdf is not None else 0}" geoai.empty cache timed step9, "STEP 9 — Zero-shot text-prompted segmentation Grounding DINO + SAM " We evaluate the segmentation model by comparing its predictions with rasterized ground-truth building labels. We calculate pixel-level IoU and F1 metrics and visualize the imagery, predictions, and reference masks together. We then apply Grounding DINO and SAM to perform zero-shot building segmentation using text prompts without additional model training. python def step10 : if not CFG "run pretrained" : print " disabled in CFG" ; return extractor = geoai.BuildingFootprintExtractor model path="building footprints usa.pth" gdf = extractor.process raster TEST RASTER, output path=os.path.join WORK, "buildings maskrcnn.geojson" , batch size=4, confidence threshold=0.5, overlap=0.25, mask threshold=0.5, min object area=100, filter edges=True, if gdf is None or len gdf == 0: print " no instances returned" ; return print f" building instances: {len gdf }" reg = extractor.regularize buildings gdf, min area=20, angle threshold=15 reg.to file os.path.join WORK, "buildings maskrcnn regularized.geojson" , driver="GeoJSON" extractor.visualize results TEST RASTER, gdf=reg, figsize= 12, 12 if FINAL GDF is not None: print f"\n your U-Net : {len FINAL GDF } polygons" print f" pretrained R-CNN: {len reg } polygons" print " Different counts are expected: U-Net merges adjacent roofs, Mask R-CNN splits" print " them into instances. Pick the paradigm that matches your downstream question." geoai.empty cache timed step10, "STEP 10 — Pretrained Mask R-CNN instance segmentation" def step11 : if not CFG "run real aoi" : print " disabled set CFG 'run real aoi' = True to run; needs open internet " return bbox = -83.9400, 35.9500, -83.9250, 35.9600 items = geoai.pc stac search collection="naip", bbox=list bbox , time range="2021-01-01/2023-12-31", max items=3, print f" STAC items found: {len items }" aoi dir = os.path.join WORK, "aoi" tif = geoai.download naip bbox=bbox, output dir=aoi dir, max items=1, preview=False print f" NAIP: {tif}" ovt = os.path.join aoi dir, "overture buildings.geojson" geoai.download overture buildings bbox=bbox, output=ovt, overture type="building" print f" Overture buildings: {ovt}" print geoai.extract building stats ovt raster = tif 0 if isinstance tif, list, tuple else tif geoai.export geotiff tiles in raster=raster, out folder=os.path.join aoi dir, "tiles" , in class data=ovt, tile size=512, stride=256, print " AOI dataset ready — feed it to train segmentation model exactly as in STEP 4." timed step11, "STEP 11 — optional Real AOI: Planetary Computer NAIP + Overture Maps labels" def step12 : outputs = f for f in sorted os.listdir WORK if f.endswith ".tif", ".geojson", ".png", ".pth" print " artifacts produced:" for f in outputs: print f" {os.path.getsize os.path.join WORK, f / 1e6:8.2f} MB {f}" zip path = os.path.join WORK, "geoai results.zip" subprocess.run "zip", "-qr", zip path, ".", "-i", " .geojson", " .png", " .tif", "-x", " tiles " , cwd=WORK, check=False, print f"\n bundle: {zip path}" if IN COLAB: print " Download it with: from google.colab import files; " "files.download '%s' " % zip path timed step12, "STEP 12 — Results summary" banner "DONE" print """ Where to go next ---------------- Swap the head, keep the code: architecture="deeplabv3plus", encoder name="efficientnet-b3" or any timm encoder in train segmentation model . 4-band NAIP RGB+NIR : num channels=4 everywhere; the first conv is auto-adapted. Multi-class land cover: geoai.train segmentation landcover + geoai.export landcover tiles , with DiceLoss / FocalLoss / TverskyLoss from geoai.landcover train for class imbalance. Instance segmentation you train yourself: geoai.train MaskRCNN model then geoai.instance segmentation ..., vectorize=True . Object detection with georeferenced boxes: geoai.train.object detection / geoai.object detection text="cars" for open-vocabulary Grounding DINO. Foundation models: geoai.prithvi inference NASA/IBM Prithvi , geoai.universat inference, geoai.DINOv3GeoProcessor for embeddings and similarity maps. Change detection: geoai.change detection torchange backends . Deploy: geoai.export to onnx + geoai.onnx semantic segmentation , or the QGIS plugin. Docs and notebooks: https://opengeoai.org | Book: https://book.opengeoai.org """ We use a pretrained Mask R-CNN model to extract individual building instances and compare them with the U-Net results. We optionally create a real-world dataset by downloading NAIP imagery from Microsoft Planetary Computer and matching building labels from Overture Maps. We finally collect the generated rasters, vectors, plots, and model outputs into a compressed results package for further analysis or download. In conclusion, we completed an end-to-end geospatial deep learning pipeline that transforms raw aerial imagery into structured and analysis-ready building footprint data. We prepared training samples, trained and evaluated a semantic segmentation model, generated seamless predictions, and refined raster outputs into orthogonalized vector geometries with useful spatial attributes. We also examined alternative extraction approaches through zero-shot foundation models and pretrained instance segmentation, which helps us understand the trade-offs between custom training, prompt-based detection, and ready-to-use models. By packaging the generated masks, probability rasters, evaluation plots, trained weights, and GeoJSON outputs, we created a reusable foundation that we can adapt for land-cover mapping, infrastructure detection, change analysis, and large-scale GeoAI applications. Check out the Full Codes here. Also, feel free to follow us on and don’t forget to join our Twitter https://x.com/intent/follow?screen name=marktechpost and Subscribe to 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ . Wait are you on telegram? our Newsletter https://www.aidevsignals.com/ now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/wbash1wF6efRj8G58 Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.