Pixels, Not Boxes: How a Detection Transformer Paints a Street

RF-DETR 1.11.2, the real-time detection transformer maintained by Roboflow, performs instance segmentation by pairing each real object in a training image with exactly one of 100 learned queries in the Segmentation Small model, using the Hungarian algorithm to minimize total fit cost across objects. The model resizes input to 384 × 384 pixels, splits it into 1,024 patches of 12 × 12 pixels, and passes tokens through a DINOv2-initialized vision transformer backbone into a 32 × 32 memory grid that four decoder layers read from. The walkthrough's heat and channel maps are illustrative, built from the drawing's true outlines rather than a trained model, with accuracy and latency figures sourced from RF-DETR's benchmarks page.

Look at a photo of a street. A cyclist rides across a zebra crossing. One car waits before the crossing, another drives away, and a dog sits on the pavement. Ask a computer where the objects are, and most systems answer with boxes. Now look closely at those boxes. In our drawing, only a fifth of the cyclist’s box is cyclist. The rest is road, crossing stripes, and the empty space inside the bicycle wheels. Each car fills about half of its box, and so does the dog. A box tells you where to look. It does not tell you which pixels belong to the object. Answering that question, one yes-or-no for every pixel and a separate answer for every object, is called instance segmentation https://en.wikipedia.org/wiki/Image segmentation . You need it to cut out a product photo, measure the area of a lake in a satellite image, or figure out how much of a field is covered by weeds. This post explains how one real-time model, RF-DETR, does it. Itis presented in eight steps, and each step has its own animation. If you only scroll through the animations, you should still get the main idea; the text adds the why and the numbers. No prior knowledge of transformers is needed. Scope. RF-DETR 1.11.2 from PyPI https://pypi.org/project/rfdetr/ , the RFDETRSeg instance-segmentation models. Mechanism details come from the RF-DETR source code and the RF-DETR paper https://arxiv.org/abs/2511.09554 ; accuracy and latency numbers come from the RF-DETR benchmarks page https://rfdetr.roboflow.com/latest/learn/benchmarks/ . Heat maps and channel maps in the animations are illustrative, built from the true outlines of our drawing, not taken from a trained model. Disclosure: I maintain RF-DETR at Roboflow. A neural network does not see a street. It sees numbers. RF-DETR-Seg Small, the size we use in this post, first resizes the photo to 384 × 384 pixels and cuts it into small 12 × 12-pixel squares called patches. That creates a 32 × 32 grid of 1,024 patches. Each patch becomes a list of numbers called a token. The tokens then pass through the backbone, a vision transformer https://en.wikipedia.org/wiki/Vision transformer initialized from DINOv2 https://arxiv.org/abs/2304.07193 . Meta trained DINOv2 on a curated set of 142 million images without any labels, so before RF-DETR has seen a single annotated street, its backbone already tells fur from metal and wheels from asphalt. Inside the backbone, tokens can compare themselves, so a patch on the car’s bonnet can take the rest of the car into account. At the end, a small module called the projector packs everything into one grid of 32 × 32 descriptions, 256 numbers per patch. RF-DETR calls this grid the memory. Every later step reads from it. A model never sees the street, only a grid of descriptions… Now the model has to find the objects. Many detectors guess a box at every position in the image, then delete duplicates afterward. RF-DETR asks questions instead. It maintains a fixed set of learned queries: 100 in Segmentation Small. Think of each query as a question: “Is there an object I should be responsible for? If so, what is it, and where?” Every query answers exactly once, either with a class and a box, or with “nothing here”, written ∅. The queries work through a short stack of decoder layers, four in Seg Small. In each layer, every query reads evidence from the memory, compares notes with the other queries, and updates its answer, including a fresh guess of its box from the same starting point. The last layer’s answer counts. Pause and ponder. Several queries can see the dog. What stops two of them from both answering “dog”? Training does. For every training image, each real object is paired with exactly one query. The pairs are chosen by the Hungarian algorithm https://en.wikipedia.org/wiki/Hungarian algorithm so that the total fit over all objects is as good as possible. Training actually runs 13 groups of 100 queries side by side and pairs each group on its own, which gives the model more practice; only one group is kept for use. Every other query is taught that the right answer was “nothing here”. A second query that also says “dog” is penalized, so the model learns not to repeat itself. That is why RF-DETR needs no NMS non-maximum suppression , the clean-up step that many detectors use to delete overlapping duplicate boxes. So far, this is exactly how RF-DETR detects objects. If you want more depth on that side, I covered it in an earlier post https://medium.com/@jborovec/300-questions-not-8-400-guesses-how-a-detection-transformer-sees-a-street-685472f6eca8 . Segmentation adds one thing on top: every query will also paint a mask. Queries trained not to repeat each other leave nothing to clean up. To paint a mask, the model needs something at pixel level. The memory is too coarse for that: each cell covers 12 × 12 pixels of the input. So the segmentation branch first enlarges the memory with bilinear interpolation https://en.wikipedia.org/wiki/Bilinear interpolation , the same smooth resizing a photo editor uses. Seg Small goes from 32 × 32 to 96 × 96. The paper states the rule: “segmentation features are always upsampled to 1/4 of the input image resolution” https://arxiv.org/abs/2511.09554 . The enlarged map then passes through a few light convolution blocks. Each block is a 3 × 3 depthwise convolution, LayerNorm, a linear layer, and a GELU activation, with a shortcut around them: a slimmed-down ConvNeXt https://arxiv.org/abs/2201.03545 block. Each decoder layer has one block, so Segmentation Small has four. A final 1 × 1 convolution and a linear layer applied at every pixel then produce the pixel map: 256 numbers for each of the 96 × 96 = 9,216 pixels. It computes this once and shares it across all queries. The design is borrowed from Mask DINO https://arxiv.org/abs/2206.02777 , which, in its authors’ words, “makes use of the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks” https://arxiv.org/abs/2206.02777 . RF-DETR keeps it lighter: “Unlike MaskDINO Li et al., 2023 , we do not incorporate multi-scale backbone features in our segmentation head to minimize latency.” https://arxiv.org/abs/2511.09554 Remember that choice; it comes back in step 8. On the query side, a small MLP multi-layer perceptron and a linear layer turn each query's decoder output into 256 numbers. Now every query and every pixel are described by lists of the same length. The mask comes from a dot product https://en.wikipedia.org/wiki/Dot product : multiply the two lists element-wise and sum them, plus one learned offset. That gives one score per pixel. A positive score means “this pixel is mine”, a negative one means “not mine”. For Seg Small, that is 9,216 scores per query. Pause and ponder. The pixel map is computed once, before any query. It does not know which query will ask. How can the same map give the dog’s query a dog mask and a car’s query a car mask? The answer is easiest to see if you read the dot product the other way round. Think of the 256 channels of the pixel map as 256 images stacked on top of each other. The dot product is then a weighted sum of those images, and the query’s 256 numbers are the weights. In this illustration, one channel responds to anything vehicle-like, one to animals, one to people, and one simply grows from left to right. The dog’s query gives the animal channel a large positive weight and the others negative weights, so the sum is positive only on the dog. The two cars show why this works for individual objects, not just for categories. Both car queries weight the vehicle channel up. Car 1’s query also weights the left↔right channel down, while car 2’s query weights it up, so each sum stays positive only on its own car. Real learned channels are not this tidy, but the arithmetic is exactly this. The paper puts it the same way: “Interestingly, we can interpret these pixel embeddings as segmentation prototypes” https://arxiv.org/abs/2511.09554 , citing YOLACT https://arxiv.org/abs/1904.02689 , which “produce\ s\ instance masks by linearly combining the prototypes with the mask coefficients” https://arxiv.org/abs/1904.02689 . The Ultralytics YOLO segmentation models use that recipe with 32 prototypes https://docs.ultralytics.com/reference/nn/modules/head/ and 32 weights per detected object. RF-DETR uses 256 channels, and its weights come from the queries, one per object. A mask is not drawn; it is a weighted sum each query chooses. The model does not wait for its last decoder layer to paint. In the paper’s words, it dots “all projected query token embeddings at the output of each decoder layer transformed by a FFN ” https://arxiv.org/abs/2511.09554 with the pixel map. Decoder layer 1 uses the pixel map after conv block 1, layer 2 the map after block 2, and so on. Each mask is trained against the true outline, so later layers can refine what earlier ones sketched, and the last layer’s mask is the one kept. A useful side effect: if you need more speed, later layers can be dropped at inference, and the earlier ones still give usable masks. Dropping a decoder layer also drops a conv block, so, as the paper notes, “truncating the decoder also shrinks the size of the segmentation branch” https://arxiv.org/abs/2511.09554 . Training extends the pairing from step 3. When the model decides which query belongs to which object, it now compares masks too, not only classes and boxes. It uses two scores for that: a binary cross-entropy https://en.wikipedia.org/wiki/Cross-entropy , which checks each pixel’s yes-or-no, and a Dice https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice coefficient score, which checks how well the two shapes overlap as a whole. After pairing, the same two scores become the mask losses, each weighted 5.0 by default. Pause and ponder. Segmentation Small paints 9,216 mask pixels per query, and during training it compares 13 groups of 100 queries against every object in every image. Do we really need every pixel to tell a good mask from a bad one? RF-DETR does not compare every pixel. It compares masks at sampled points, an idea from PointRend https://arxiv.org/abs/1912.08193 . The number of points is a sixteenth of the mask: 9,216 ÷ 16 = 576 points for Seg Small. The two use sampling differently. Pairing draws one set of uniform random points and scores every query against every object at those same points, which keeps the comparison fair. The loss is computed only for the paired queries, so it can be smarter. It draws three times as many random candidates and keeps the 75% of its points where the predicted score is closest to zero, the pixels the model is least sure about. The remaining 25% are random. Points where the model is already confident are mostly skipped; points near its predicted outline get most of the attention. Before it ever sees COCO, the standard benchmark dataset, RF-DETR-Seg is also pretrained on Objects365 https://www.objects365.org/ , with instance masks generated automatically by SAM 2 https://arxiv.org/abs/2408.00714 , so the detection and segmentation parts learn from the same data. Most pixels are easy; training focuses on the hard ones. The quarter-resolution pixel map sets a limit. In Seg Small, one mask cell covers 4 × 4 input pixels. Pause and ponder. Our dog spans about 8 × 7 mask cells. Its legs and ears are about one cell wide. What happens to them? To see the limit, I built a simple ideal mask on the 96 × 96 grid: each cell’s score comes from how much of it the dog covers, then the map is enlarged and cut at zero the same way the model does it. Its IoU https://en.wikipedia.org/wiki/Jaccard index intersection over union, the overlap divided by the combined area with the drawn dog is 0.87, not 1.0. The body is fine; it loses on the thin legs and the ear tip. Tweaking how the cells are filled moves the score by about a point; it does not get near 1.0. Mask DINO also paints on a quarter-resolution grid, so both share that limit. What RF-DETR gives up is the high-resolution backbone detail that helps fill that grid on fine structures; in exchange, it stays fast. The remedy is a larger model: RF-DETR-Seg 2XL works at 768 pixels with a 192 × 192 mask grid. A mask cannot be sharper than the grid it is painted on. At the end, RF-DETR keeps the highest-scoring query-class pairs. Each kept query brings its own mask from the last decoder layer. The mask scores are enlarged to the size of the original photo, and every pixel with a score above zero and a sigmoid https://en.wikipedia.org/wiki/Sigmoid function probability above 50% belongs to the mask. Box, class, and mask all come from the same query, so there is no clean-up step. One query can still appear twice if it is confident about two classes, with the same mask both times. python from rfdetr import RFDETRSegSmallmodel = RFDETRSegSmall detections = model.predict "https://media.roboflow.com/dog.jpg", threshold=0.5, masks = detections.mask one boolean H × W mask per object The result is a supervision https://supervision.roboflow.com/latest/detection/core/ Detections object with boxes, classes, confidences, and a mask array. An exported model returns per-query boxes, class scores, and mask scores at the quarter-resolution grid; picking the top answers, enlarging, and thresholding run after the model. RF-DETR-Seg comes in six sizes, all built on the same DINOv2-S backbone. They differ mainly in input resolution and decoder depth. Compared with the detection-only models, the segmentation models have deeper decoders and fewer queries: Seg Nano has 4 decoder layers where detection Nano has 2, and Seg Small has 4 where detection Small has 3. The paper suggests this “likely reflects a minimum viable segmentation head depth” https://arxiv.org/abs/2511.09554 , and explains the trade: “to offset its latency, RF-DETR-Seg reduces its total number of queries, yielding a thin, deep decoder” https://arxiv.org/abs/2511.09554 . The parameter count barely moves across sizes, from 33.6 million for N to 38.6 million for 2XL, because every size shares the same backbone. Here are the reported instance-segmentation numbers next to YOLO26-Seg, a recent Ultralytics generation, size for size. AP50:95 is COCO’s standard accuracy score: average precision, averaged over overlap thresholds from 0.50 to 0.95. Latency is measured on an NVIDIA T4 GPU with TensorRT at half precision FP16 , one image at a time. RF-DETR-Seg is more accurate at every size, but the lead varies: 5.6 AP50:95 points at N, 1.3 at M, 2.0 at XL against X. YOLO26-Seg is faster at N, S, L, and X; RF-DETR-Seg is faster at M, 5.9 ms against 6.32 ms. YOLO26-Seg has fewer parameters up to L, while at the top RF-DETR-Seg XL has 38.1 million against 62.8 million for YOLO26-X-Seg. Against the previous generation the gap is wider. The paper reports that “RF-DETR-Seg nano outperforms YOLOv11-Seg x-large on COCO while running 4 × as fast” https://arxiv.org/abs/2511.09554 : 40.3 against 40.1 AP50:95, at 3.4 ms against 13.7 ms. Three caveats. First, these numbers were measured by the team that builds RF-DETR; treat them as a starting point for your own test. Second, the YOLO26-Seg rows come from the RF-DETR benchmarks page, not the paper. Third, the paper notes that “our latencies for YOLOs also include the conversion of protos into masks” https://arxiv.org/abs/2511.09554 , a step some published YOLO timings leave out, so compare latencies only when you know whether that step is included. Accuracy per millisecond is the claim; your masks on your data are the test. Strip away the details, and RF-DETR-Seg does two things at once. It finds each object with one question, and it paints that object’s pixels by mixing a shared pixel map with the question’s own weights. Here is the path from photo to mask, one step per animation: If your objects have complicated shapes, like bicycles, people, or animals, a box wastes most of its area on them, and this is the cheapest way to get the pixels right. If they are tiny, pick a larger size for a finer grid. If every millisecond counts, YOLO26-Seg is faster at most sizes; RF-DETR-Seg trades a little time for accuracy. And whatever you pick, test it on your own data. The next post takes the same street and the same queries and adds keypoints: the cyclist’s joints and the dog’s pose. Co-authored by Claude. The model sizes and how to run them are on the RF-DETR segmentation page https://rfdetr.roboflow.com/latest/learn/run/segmentation/ . Which object in your own images would lose the most to a box? Tell me in the comments, and if you want to try RF-DETR-Seg, the code is at github.com/roboflow/rf-detr https://github.com/roboflow/rf-detr . Pixels, Not Boxes: How a Detection Transformer Paints a Street https://pub.towardsai.net/pixels-not-boxes-how-a-detection-transformer-paints-a-street-0e1cdb938925 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.