# Object detection and vision transformers > Source: > Published: 2026-10-30 11:50:43+00:00 [Stochastic Blog](https://stochastic.blog/tag/stochastic-blog/) # Object detection and vision transformers Post 29 taught us to look at a picture and say what is in it. A convolutional classifier takes an image and returns one label, and with transfer learning it does that well. But a photo of a street is not a single answer. It is a car here, a person there, a bicycle at the edge, each with a location. This post builds the machinery that returns boxes and labels together, and it reaches a concrete result: a tiny Vision Transformer with an anchor head lands at **0.815** mean IoU on validation, well above a random box baseline of **0.309**. The through-line we will carry the whole way is the idea of *proposals and their pruning*. A detector is a machine that throws out candidate boxes and then decides which ones to keep, and every architecture in this post is a different answer to how you generate those candidates and how you throw the bad ones away. We work on PASCAL VOC 2007, the classic 20-class detection benchmark, and we build everything from scratch in PyTorch. The dataset card says about 9,900 images and 900 MB for both splits, but the notebook builds a deterministic VOC-shaped cache of 500 images so the run fits a laptop budget. Every later cell sees the same schema and real JPEG files without waiting on the network. ## The data before the model Before any resize or normalization, we look at the raw images and their boxes. The first thing we check is whether the schema is clean. The schema check passes: no scalar field is missing, no box or label is empty, and every box has exactly one label, so we trust all five hundred cached rows. The five raw examples in the first figure show what a box actually is on disk: a rectangle drawn in pixel coordinates, with a class name attached. The class balance is the finding that shapes the rest of the post. Every class appears in exactly 25 images, so the base rate for the most common class, aeroplane, is **0.05**. That is a flat distribution by construction, which means we cannot lean on a majority class to fake accuracy. We have to report box IoU beside class accuracy, because a model that guesses the label right but puts the box in the wrong place has learned nothing useful. Image sizes vary, from 200 to 400 pixels on each side, with a mean width of **300.7** and mean height of **302.1**. Every image is RGB, so the channel count is a clean 3. The scatter of width against height in the next figure is a filled square, which tells us the cache was generated with independent random sizes and no aspect-ratio structure to exploit. The pixel statistics matter for normalization. Across 100 sampled images the grayscale mean is **128.0** with a standard deviation of **40.6**, so the images are neither dark nor blown out on average. The per-class gallery confirms the vocabulary: 20 classes, one labeled example each, with the target class boxed in red and the others in yellow. The intensity histogram is broad and roughly centered, which is what we want before we scale pixels to the unit interval. Finally we hunt for the hard cases: the smallest image, the most crowded, and the darkest. The smallest needs a resize, the crowded one needs non-maximum suppression to avoid duplicate boxes, and the dark one needs normalization. Those three needs are exactly the three mechanisms the rest of the post builds. The EDA hands us a plan. We resize to 96 by 96, normalize by scaling pixel values to the unit interval, keep the largest box per image to stay inside the CPU budget, and use 300 train and 100 validation images with seed 42. The same tiny Vision Transformer (ViT) backbone will feed both an anchor head and a Detection Transformer (DETR) head, so the comparison isolates the head design. ## Measuring overlap The first primitive every detector needs is a way to say how much two boxes agree. **Intersection over Union** is the area of the intersection divided by the area of the union, a number between 0 and 1 that is 1 only for identical boxes. **Non-maximum suppression** uses IoU to clean up: sort boxes by score, keep the best, and drop every box that overlaps it too much. The snippet below runs both on toy boxes so the numbers are exact and checkable. ``` ious = box_iou(boxes_a, boxes_b) keep = nms(boxes_a, scores, 0.5) # keep top box, drop near duplicates print('NMS keep indices', keep.tolist()) ``` The IoU matrix comes back with a 1.0 on the diagonal where a box meets itself, and lower values off it, such as **0.471** between the first and second boxes. NMS keeps indices `[0, 1, 2, 3]` here because the toy boxes are spread out enough that none crosses the 0.5 threshold against a higher-scoring neighbor. The mechanism is the point: the top-scoring box survives and its near duplicates do not.