300 Questions, Not 8,400 Guesses: How a Detection Transformer Sees a Street

RF-DETR 1.11, a detection transformer maintained at Roboflow, replaces the 8,400 candidate boxes and non-maximum suppression cleanup used by convolutional detectors such as YOLO11 with 300 learned queries that each answer once, the author writes. At 640 × 640 pixels, YOLO11's head predicts on three grids with strides of 8, 16 and 32 pixels, producing 80 × 80 + 40 × 40 + 20 × 20 = 8,400 candidate boxes per image, which NMS then deduplicates using a hand-tuned IoU threshold. The author, who maintains RF-DETR, cites accuracy and latency figures from the RF-DETR paper (arXiv 2511.09554) and the RF-DETR benchmarks page, measured on an NVIDIA T4 with TensorRT FP16 at batch size 1, and notes the original 2020 DETR required around 500 training epochs on COCO and struggled with small objects.

Look at a photo of a street. It runs away from you toward the top right, with a zebra crossing in front. A cyclist rides across the crossing from left to right. One car approaches and stops before the crossing; another has already passed and drives away. A dog sits on the right pavement, waiting to cross. You did not consider ten thousand possible boxes and then throw most of them away. You just saw four things. Now try to build a network that answers the same way. How would you approach it? How would you design it? There are two common designs. A convolutional detector such as YOLO guesses everywhere and cleans up afterward. A detection transformer such as RF-DETR asks a fixed number of questions and is trained not to repeat itself. This post builds the second design from scratch, using this street in every picture. No prior transformer knowledge is assumed. Scope. RF-DETR 1.11 from PyPI https://pypi.org/project/rfdetr/ , object detection only. Segmentation and keypoints get their own post. Accuracy and latency numbers come from the RF-DETR paper https://arxiv.org/abs/2511.09554 and the RF-DETR benchmarks page https://rfdetr.roboflow.com/latest/learn/benchmarks/ , measured on an NVIDIA T4 with TensorRT https://developer.nvidia.com/tensorrt FP16 at batch size 1; I did not re-run them for this post. Disclosure: I maintain RF-DETR at Roboflow. A convolutional https://en.wikipedia.org/wiki/Convolutional neural network detector such as YOLO11 YOLO stands for “You Only Look Once” answers by guessing everywhere. At 640 × 640 pixels, its head predicts on three grids with strides of 8, 16, and 32 pixels. That is 80 × 80 + 40 × 40 + 20 × 20 = 8,400 candidate boxes for every image. Each grid cell says: if an object is centered near me, here is its box and class. Most cells on our street see road or buildings. A handful see the dog, and several neighboring cells see the same dog. So the photo produces a cluster of overlapping boxes around every object. Non-maximum suppression https://docs.pytorch.org/vision/main/generated/torchvision.ops.nms.html NMS then cleans up. It keeps the highest-scoring box and deletes every box that overlaps it by more than a threshold of intersection over union https://en.wikipedia.org/wiki/Jaccard index IoU, the overlap area divided by the combined area . Then it repeats. It works, and on a GPU it is fast. It also has three properties worth noticing. The IoU threshold is a hand-tuned knob, and its right value changes between datasets. Set it too low and two objects that genuinely overlap, such as two people walking side by side, can collapse into one box. Its runtime depends on how many candidates survive, so a crowded street takes longer than an empty one. Most importantly, it lives outside the network. The model is never trained to avoid duplicates; it is trained to produce them and rely on the cleanup. A detector that depends on NMS has learned to make a mess and trusts someone else to tidy it. In 2020, DETR https://arxiv.org/abs/2005.12872 DEtection TRansformer reframed detection as set prediction. The model holds a fixed set of learned queries. Each query asks: is there an object I am responsible for, and if so, where and what is it? Every query answers once. A query with nothing to report answers “no object”, written ∅. RF-DETR uses 300 queries. For our street, 4 of them should name the cyclist, the two cars, and the dog. The other 296 should say ∅. Pause and ponder. Several queries can see the dog. If only one of them is allowed to say “dog”, how does the second one know to stay quiet? It needs two things: a way to see what the other queries are doing, and a reason to back off. Keep this puzzle in mind. The decoder supplies the first half, and training supplies the second. The original DETR had two known problems. It needed around 500 training epochs on COCO https://cocodataset.org/ Common Objects in Context , and it struggled with small objects. Deformable DETR https://arxiv.org/abs/2010.04159 fixed both and reported 10 times fewer training epochs. RF-DETR inherits that fix and adds several more to reach real-time speed. Training time has kept falling since: between releases 1.9 and 1.11, one COCO epoch of RF-DETR Large went from 59 to 13 minutes on a single GPU, as I describe in Closing the training gap between DETR and YOLO https://levelup.gitconnected.com/closing-the-training-gap-between-detr-and-yolo-12-faster-from-engineering-not-a-new-architecture-1e91b84bf70e . NMS-free detection is no longer unique to transformers. YOLOv10 https://arxiv.org/abs/2405.14458 introduced a one-to-one training head that removes NMS, and Ultralytics YOLO26 https://docs.ultralytics.com/models/yolo26/ ships end-to-end without it. The idea that came from DETR has spread. What still separates the two families is what sits in front of the head, and that is where the rest of this post spends its time. Here is the whole path for RF-DETR Small. The six cards at the top are the six sections below; the animation brings each stage forward in turn. RF-DETR Small resizes the street to 512 × 512 pixels and cuts it into 16 × 16 pixel patches. That makes a 32 × 32 grid of 1,024 patches. Each patch is flattened into a vector called a token. From here on, the network never sees pixels again, only these 1,024 tokens and their positions. The four released sizes differ mainly in resolution and decoder depth. More tokens mean finer detail, which helps the small dog, at the cost of more compute. The tokens pass through a vision transformer https://en.wikipedia.org/wiki/Vision transformer ViT initialized from DINOv2 https://arxiv.org/abs/2304.07193 , a model Meta pretrained on a curated set of 142 million images with self-supervised learning https://en.wikipedia.org/wiki/Self-supervised learning , that is, without labels. Before RF-DETR has seen a single bounding box, its backbone already separates fur from metal and wheels from asphalt. The RF-DETR paper https://arxiv.org/abs/2511.09554 gives this as a reason RF-DETR adapts well to small or unusual datasets: it reports that initialising the backbone from DINOv2 “significantly improves detection accuracy on small datasets” https://arxiv.org/abs/2511.09554 . A transformer block uses attention <https://en.wikipedia.org/wiki/Attention machine learning : every token compares itself with every other token. A patch on the dog’s tail can consult a patch on its ears directly, in one step. A 3 × 3 convolution sees only its immediate neighbours and reaches the whole image only after many layers, as its receptive field https://en.wikipedia.org/wiki/Receptive field grows. The catch is cost: comparisons grow with the square of the token count. For Small, one global block means 1,024 × 1,024, about one million token pairs. RF-DETR keeps this affordable with windowed attention. Most blocks split the 32 × 32 grid into 2 × 2 windows of 256 tokens each, and a token only looks inside its own window. That is 4 × 256 × 256, about 262 thousand pairs, a quarter of the global cost. Pause and ponder. Look where the window border falls on our street. Car 1, the one waiting before the crossing, sits across it: its front is in the top-left window and its rear in the top-right one. If every block only looked inside its own window, would the two halves of the same car ever compare notes? They would not. That is why three of the twelve blocks stay global: they are the bridges that carry information across window borders. Features are taken from four backbone depths. Early blocks carry fine local detail, and late blocks carry more abstract meaning. A small projector fuses them into one feature map at stride 16, using C2f blocks, the convolutional block based on CSPNet https://arxiv.org/abs/1911.11929 that YOLOv8 popularised. This fused map is called the memory, and the decoder reads it. Two details are easy to miss. First, there is no separate transformer encoder. DETR and Deformable DETR stack encoder layers after the backbone; RF-DETR, like LW-DETR https://arxiv.org/abs/2406.03459 , lets the pretrained ViT and the projector do that job. Second, the projector uses LayerNorm https://arxiv.org/abs/1607.06450 instead of BatchNorm https://en.wikipedia.org/wiki/Batch normalization . BatchNorm normalizes over the images in each step, so with the small per-step batches on a consumer GPU, its statistics get noisy, and gradient accumulation doesn't help because each step still sees only those few images. LayerNorm normalizes each token independently. The authors say they "facilitate training on consumer-grade GPUs via gradient accumulation by using layer norm instead of batch norm in the multi-scale projector" https://arxiv.org/abs/2511.09554 . The fastest encoder is the one a better backbone made unnecessary. The queries need a starting point. RF-DETR uses a two-stage scheme. Every memory token makes a quick class score and box proposal, and the 300 highest-scoring proposals become the queries’ initial reference boxes. So the query that will end up naming the dog does not start by asking whether anything is out there. It starts with: something dog-shaped is near here, let me check. The decoder is short: three layers for Small. In each layer, every query does three things. It first runs self-attention against the other queries. This is the first half of the puzzle from earlier: two queries that both latched onto the dog can now see each other. Seeing is not yet backing off; the reason to back off comes from training, two sections below. It then runs cross-attention into the memory to read image evidence. Finally, it predicts a small correction to its box. Cross-attention is the second efficiency trick. In the original DETR, the dog’s query reads every location in the feature map: all 1,024 for a Small-sized input. RF-DETR uses deformable attention instead. Each query predicts a few sampling offsets around its current box and reads only those points. In Small, that is 16 attention heads × 2 points = 32 samples per query, no matter how large the image is. The last decoder layer gives each of the 300 queries a class score and a box. RF-DETR keeps the highest-scoring query-class pairs, and you apply a confidence threshold. That is the entire post-processing. No IoU threshold to tune. python from rfdetr import RFDETRSmallmodel = RFDETRSmall detections = model.predict "https://media.roboflow.com/dog.jpg", threshold=0.5, When the post-processing is a top-k and a threshold, there is nothing left to tune and nothing to port. Training is where the one-answer-per-object behavior comes from. For our street there are 300 predictions and 4 ground-truth objects. Which prediction should be compared with which object? Pause and ponder. Try a greedy https://en.wikipedia.org/wiki/Greedy algorithm rule first: let every object take its cheapest query. What goes wrong when one query is the cheapest choice for two objects? RF-DETR scores every query, object pair with three cost terms: class error, L1 distance https://en.wikipedia.org/wiki/Taxicab geometry between the boxes, and generalized IoU https://arxiv.org/abs/1902.09630 GIoU, an IoU variant that still gives a useful signal when boxes do not overlap . This is an assignment problem https://en.wikipedia.org/wiki/Assignment problem , and the Hungarian algorithm https://en.wikipedia.org/wiki/Hungarian algorithm solves it exactly: it finds the cheapest assignment in which every object gets exactly one query. The 4 matched queries are trained toward their objects. The other 296 are trained toward ∅. Look at the animation. Greedy fails exactly where the ponder box predicted: query 5 is the cheapest choice for both car 2 and the dog. The Hungarian algorithm resolves the conflict by giving the dog to query 6, at a slightly higher cost. Now look at query 3. It is a decent guess for car 1, but query 2 is better. So query 3 is told “no object”. This is the second half of the puzzle: the reason to back off. Self-attention lets query 3 see that query 2 has car 1; the loss makes speaking up anyway expensive. Applied across many images, that single rule removes duplicates. The network is penalized for them during training, so at inference it stops producing them. A single positive query per object is a thin learning signal, and it is the main reason the original DETR trained so slowly. RF-DETR adds three things to speed convergence. During training, it uses 13 independent groups of 300 queries, a technique from Group DETR https://arxiv.org/abs/2207.13085 , each matched to the ground truth independently. Our dog now gets 13 positive queries per image instead of one. At inference, we keep only one group, so deployment cost does not change. The classification target for a matched query is not a plain 1 either. It blends the predicted score with the IoU of the predicted box, so the model learns to be confident in proportion to how well its box fits. And every decoder layer gets its own loss, not just the last one. That last detail matters for the next section. Duplicates are not filtered out of RF-DETR; they are trained out of it. Most detector families train each size separately. The RF-DETR authors instead designed the training as weight-sharing neural architecture search https://en.wikipedia.org/wiki/Neural architecture search NAS . The idea, popularised by Once-for-All https://arxiv.org/abs/1908.09791 , is to train one set of weights so that many smaller sub-networks inside it work without retraining. The search varies five knobs, but not all at once. During training, it samples input resolution 320 to 960 pixels , patch size 8 to 32 pixels , and the number of attention windows 1, 2, or 4 . In the paper’s https://arxiv.org/abs/2511.09554 words, “at every training iteration, we uniformly sample a random model configuration and perform a gradient update” https://arxiv.org/abs/2511.09554 . The other two knobs, decoder depth and query count, stay fixed at 6 layers and 300 queries during training. Appendix A of the paper is explicit: “Importantly, we only drop decoder layers and queries during inference.” https://arxiv.org/abs/2511.09554 That works because every decoder layer was supervised during training, so dropping later layers still leaves usable boxes. After training, we evaluate each configuration on a validation set without fine-tuning. The best ones form an accuracy-latency curve, and the released Nano, Small, Medium, and Large checkpoints are points on that curve. Model size here is a point you pick on a curve, not a model you retrain. Here are the reported numbers next to YOLO26, the newest YOLO generation, on a size-for-size basis. RF100-VL https://rf100-vl.org/ averages 100 diverse real-world datasets, which is closer to what most of us fine-tune on than COCO. AP50:95 is COCO’s standard accuracy score, averaged over IoU thresholds from 0.50 to 0.95. The pattern is clean. At every size, RF-DETR wins on accuracy, by 2.2 to 8.1 AP50:95 points on COCO and 2.5 to 5.7 on RF100-VL. YOLO26 is faster at N, S, and L, ties at M 4.4 ms , and always has fewer parameters. One cross-size point the table hides: YOLO26-X https://rfdetr.roboflow.com/latest/learn/benchmarks/ reaches 56.9 on COCO, ahead of RF-DETR-L’s 56.5, but needs 9.6 ms against 6.8 ms. Three notes on reading this table. First, the transfer gain on RF100-VL is not all backbone. LW-DETR https://arxiv.org/abs/2406.03459 -L, built on a different backbone, reaches 61.5 on the same benchmark, against 62.2 for RF-DETR-L. The paper credits DINOv2 for part of the gain on small datasets; this table does not isolate it. Second, parameter count is not latency. RF-DETR-N has 11.7 times the parameters of YOLO26-N but is only 0.6 ms slower on a T4. A ViT block is built from large dense matrix multiplications, the operation GPU Tensor Cores are designed for, and NVIDIA’s Transformer Engine https://github.com/NVIDIA/TransformerEngine adds FP8 support for transformer layers on Hopper, Ada, and Blackwell GPUs. The parameters still cost memory and file size, so if those are your binding constraints, measure on the target device. Roboflow also lists smaller Atto, Femto, and Pico models 7.4 to 8.4 million parameters on the develop benchmarks page https://rfdetr.roboflow.com/develop/learn/benchmarks/ ; they use a different backbone, ship through rfdetr plus under a separate license, and are outside this post. Third, these numbers come from the team that builds RF-DETR. Treat them as a starting point for your own benchmark, not a verdict. The paper itself makes the point: it notes that RT-DETR beats D-FINE on RF100-VL AP50, which suggests D-FINE’s hyperparameters are “potentially overfit to COCO” https://arxiv.org/abs/2511.09554 . COCO alone does not tell you how a detector will behave on your data. A leaderboard tells you which model fits COCO; only your own validation set tells you which model fits you. If you are choosing a detector, these are the questions the architecture actually answers: The next post takes the same street and the same 300 queries, and adds heads for instance masks and keypoints. The pipeline above stays almost unchanged, which is the interesting part. Co-authored by Claude. The full walkthrough is also part of the RF-DETR documentation: see How It Works https://rfdetr.roboflow.com/develop/learn/how-it-works/ on the develop docs. Which of these trade-offs decides it for your use case? Tell me in the comments, and if you want to try it, the code is at github.com/roboflow/rf-detr https://github.com/roboflow/rf-detr . 300 Questions, Not 8,400 Guesses: How a Detection Transformer Sees a Street https://pub.towardsai.net/300-questions-not-8-400-guesses-how-a-detection-transformer-sees-a-street-cd1123b2fcea was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.