Where the Knees Are: How a Detection Transformer Reads Poses on a Street

RF-DETR 1.11.2's RFDETRKeypointPreview model, an early-access keypoint detector from Roboflow that runs at 576 × 576 pixels and predicts the 17 COCO person keypoints, is detailed in a technical walkthrough by an author who maintains RF-DETR at Roboflow. The model resizes images into 2,304 patches of 12 × 12 pixels, tokenizes them through a DINOv2-initialized vision transformer backbone, and builds a 256-number-per-patch memory grid that a second projector duplicates so the box-finding and joint-finding heads read separate copies. The walkthrough notes the RF-DETR paper covers detection and segmentation rather than keypoints, so the mechanism is drawn from the 1.11.2 source code, and that fine-tuning on custom keypoint datasets is the model's primary intended use case.

Look at a photo of a street. A cyclist rides across a zebra crossing. One car waits before the crossing, another drives away, and a dog sits on the pavement. A detector draws a box around each of them. That tells you where they are. It does not tell you whether the cyclist is pedalling or reaching for the brake, or whether the dog is sitting or about to run. For that you need keypoints. Keypoints are named points on an object: a nose, a left knee, a tail tip. Join them with lines, and you get a skeleton. Finding them for every person in a photo is called pose estimation https://en.wikipedia.org/wiki/Articulated body pose estimation ; the same idea works for animals, hands, or the corners of a parcel. It is how sports apps count reps, how ergonomics tools spot a bad lifting posture, and how biologists track how animals move. The hard part is not finding knees. It is knowing whose knee it is. Pause and ponder. Two people stand close together, and their legs overlap. A model has found four knees. Which two belong to the person on the left? Pose models have given different answers to that question over the years, and RF-DETR’s answer is the thread of this post. It is told in nine steps, and each step has its own animation. If you only scroll through the animations, you should still get the main idea; the text adds the why and the numbers. No prior knowledge of transformers is needed. Scope. RF-DETR 1.11.2 from PyPI https://pypi.org/project/rfdetr/ , the RFDETRKeypointPreview model. The docs call it "an early-access release. Fine-tuning on custom keypoint datasets is the primary intended use case." https://rfdetr.roboflow.com/latest/learn/run/keypoints/ It comes in one size, works at 576 × 576 pixels, and its pretrained checkpoint predicts the 17 COCO person keypoints. The RF-DETR paper https://arxiv.org/abs/2511.09554 covers detection and segmentation, not keypoints, so the mechanism here comes from the 1.11.2 source code. Accuracy and latency numbers come from the RF-DETR benchmarks page https://rfdetr.roboflow.com/latest/learn/benchmarks/ . Our street uses three classes with three skeletons cyclist 17 joints, dog 8, car none : that is the kind of model you would get by fine-tuning, not the pretrained checkpoint. All street visuals are drawn, not model output; uncertainty ellipses and guesses are illustrative. Disclosure: I maintain RF-DETR at Roboflow. Before the details, here is the full path from photo to skeletons in one animation. Each stage takes the previous stage’s output as its input, does one job, and hands its result on. The rest of the post walks through these stages one at a time, with one animation per step. A neural network does not see a street. It sees numbers. RF-DETR Keypoint first resizes the photo to 576 × 576 pixels and cuts it into small 12 × 12 pixel squares, called patches. That makes a grid of 48 × 48 = 2,304 patches. Each patch becomes a list of numbers called a token. The tokens pass through the backbone, a vision transformer https://en.wikipedia.org/wiki/Vision transformer initialised from DINOv2 https://arxiv.org/abs/2304.07193 . Meta trained DINOv2 on 142 million curated images without any labels, so before RF-DETR has seen a single annotated skeleton, its backbone already produces general-purpose visual features. Inside the backbone, tokens compare themselves with each other, so a patch on the rider’s knee can account for the rest of the leg. At the end, a small module called the projector packs everything into a 48 × 48 grid of descriptions, 256 numbers per patch. RF-DETR calls this grid the memory. The keypoint model has a second projector that builds a second memory of the same size from the same backbone output. The box-finding part reads the first; the joint-finding part reads the second. We will meet it again in step 6. A model never sees the street, only grids of descriptions… Many detectors guess a box at every position in the image and then delete the duplicates afterward. RF-DETR asks questions instead. It holds a fixed set of 100 learned queries. Think of each query as a question: “Is there an object I should be responsible for? If so, what is it, and where?” Every query answers with one box and a score for each class, or with “nothing here”, written ∅. Each query starts from a reference box, chosen from the most promising regions of the memory, and works through four decoder layers. In each layer, it reads evidence from the memory, compares notes with the other queries, and updates its answer. It always predicts its box as a shift and resize of that same reference box. Remember the reference box: the joints will be measured from it too. During training, each real object is paired with exactly one query once in each of 13 parallel query groups, which gives the model more practice , chosen by the Hungarian algorithm https://en.wikipedia.org/wiki/Hungarian algorithm so the total fit across all objects is as good as possible. Every other query is taught to answer “nothing here”. A second query that also claims the dog is penalized, so the model learns not to repeat itself, and no NMS non-maximum suppression, the clean-up step that deletes overlapping duplicates is needed. So far, this is exactly how RF-DETR detects objects, with 100 queries here instead of the detector’s 300; I covered that side in an earlier post https://medium.com/@jborovec/300-questions-not-8-400-guesses-how-a-detection-transformer-sees-a-street-685472f6eca8 if you want more depth. Keypoints add one idea on top, and it answers the “whose knee” question before it can even come up: every query brings its own joints. A question that owns an object can also own its knees. Next to each query, the model keeps one extra token per joint in the schema. In our street, the schema is cyclist 17, car 0, dog 8, so every one of the 100 queries carries 25 joint tokens. Each token is a slot that will become one joint: “left knee of whatever this query finds”. A slot starts from a learned token for its joint type, one for “nose”, one for “left knee”, and so on. The token is then scaled and shifted by numbers computed from its own query, a trick called adaptive layer normalization AdaLN . So the cyclist’s query and a car’s query both start their “left knee” slot from the same learned knee, each adapted to its own query. What the object in front of it actually looks like comes in later, in the decoder layers. The slots are not read for every class. A query that ends up as “car” has its joints ignored; one that ends up as “dog” is read from the 8 dog slots. A zero in the schema means “this class is box only”. The pretrained preview checkpoint has just one class with one skeleton, the 17 COCO person keypoints; the street’s three classes come from fine-tuning on your own labels. The question of whose knee it is never arises: each knee slot is born inside one query. In every decoder layer, each query and its joint tokens run a round of self-attention <https://en.wikipedia.org/wiki/Attention machine learning : every token looks at the others in its group and takes in what it finds useful. A knee can use where the hip and the ankle are. The query also participates, so it learns from its own joints, and the joints learn from it what the object is. Two walls keep this tidy. Slots of different skeletons inside one query are masked from each other, so a query's person slots don't mix with its dog slots. And joints from different objects don't attend to each other at all: the code has an option for that, inter instance kp attn, and the preview model ships with it off. This idea comes from GroupPose https://arxiv.org/abs/2308.07313 , which uses an instance query plus keypoint queries for each pose and two kinds of group self-attention: one within each person, and one across people for the same joint type. RF-DETR’s preview keeps the first. Of the second, it keeps only the object-to-object part, the ordinary detection self-attention; joint-to-joint attention across objects is switched off. Pause and ponder. Joints never look at other objects. So why don’t two overlapping cyclists end up with each other’s knees? Because each object query is still a detection query: in the detection part of every layer, the queries compare notes with each other, which keeps them on separate objects. The joints inherit that separation through their own query. The boxes do the bookkeeping; the joints do the anatomy. Joints learn from each other only inside their own object. Talking to its neighbors tells a knee roughly where it should be. To know where it actually is, it has to look at the image. After the talking, every joint token reads the second memory with deformable attention https://arxiv.org/abs/2010.04159 : instead of looking at all 2,304 cells, it learns to sample a few points around its query’s reference box and weighs what it finds there. It is the same reading mechanism the object queries use on the first memory. In the preview model, each joint reads 32 sampled points 16 attention heads, 2 points each instead of all 2,304 cells. The second projector lets the joints get descriptions tuned for fine anatomy, while the box-finding part keeps its own. Then each joint is placed. The model predicts an offset, and the joint lands at the reference box’s center plus that offset times the box’s width and height. An offset of −0.2, 0.3 means “a fifth of a box-width left of center, three tenths of a box-height down”, and that holds for a cyclist who fills the frame and for one far down the street. A joint is not a pixel; it is a position inside its object’s box. A pose model always has to answer for every joint, even one hidden behind the body. RF-DETR answers with more than a position. For each joint, the model predicts: The spread is trained with a Gaussian negative log-likelihood https://en.wikipedia.org/wiki/Likelihood function Log-likelihood : the model is penalized both for missing a joint and for claiming to be sure when it is not. That trains it to be honest about how sure it is. Finally, the spreads also adjust the object’s score. In RF-DETR 1.11.2, the score is combined with the average size of its joints’ ellipses weighted by how findable each joint is , so a confident box with wobbly joints ends up with a lower score than an equally confident box with tight joints. The fused score is then squeezed back into the range 0 to 1. Ultralytics’ YOLO26-pose also models uncertainty, with Residual Log-Likelihood Estimation https://arxiv.org/abs/2107.11291 : “Integrates Residual Log-Likelihood Estimation RLE for more accurate keypoint localization” https://docs.ultralytics.com/models/yolo26/ . A position without confidence is half an answer. Training extends the pairing from step 3. When the model decides which query belongs to which real object, it now compares joints too, not only classes and boxes. The pairing cost and the training losses use the same ingredients. The distance between each predicted and true joint is measured with an L1 distance https://en.wikipedia.org/wiki/Taxicab geometry and divided by the square root of the true box’s area, so a far, small cyclist counts as much as a near, big one. A binary cross-entropy https://en.wikipedia.org/wiki/Cross-entropy for findable and one for visible. And the negative log-likelihood of the spread. By default, each keypoint loss has weight 1.0. Pause and ponder. Why divide by the box size? A 10-pixel miss is a 10-pixel miss. Not to a person looking at the result. Ten pixels is a small miss on a cyclist who fills the frame and a whole leg on one far down the street. Dividing by the box size measures the miss in “object sizes”, the same idea the benchmark uses in the next step. Positions are compared only for labeled joints; a joint with no label has no position to compare against. One-to-one pairing keeps duplicates from being rewarded; adding the joints makes it pick the query with the best pose. How do you score a predicted pose? For boxes, there is IoU https://en.wikipedia.org/wiki/Jaccard index , the overlap of two boxes. For keypoints, COCO defines Object Keypoint Similarity https://cocodataset.org/ keypoints-eval , OKS, which, in COCO’s words, “plays the same role as the IoU” https://cocodataset.org/ keypoints-eval . Each true joint gets a score between 0 and 1 that falls off like a bell curve with the distance d to the guess: exp −d² / 2 s² κ² Here s² is the object’s area and κ = 2σ, where σ is a constant for each joint type. COCO measured those constants from how much human annotators disagreed on 5,000 redundantly annotated images: eyes 0.025, nose 0.026, shoulders 0.079, hips 0.107. People agree on where an eye is far more than on where a hip is, so a hip can be further off for the same score. OKS averages over the labeled joints. Average precision then counts a pose as correct if its OKS passes a threshold, and averages over thresholds from 0.50 to 0.95. That is the AP50:95 in the comparison below. That brings us to the most practical tip in this post. When you fine-tune on your own skeleton, COCO’s σ values do not apply, and RF-DETR needs to know yours to report a meaningful AP. If you pass none and your skeleton does not have 17 joints, version 1.11.2 falls back to a uniform σ of 0.05 for every joint and warns you. A custom skeleton that happens to have 17 joints silently gets COCO’s person values. The schema helpers in the docs pass a flat 0.1 instead. Neither is wrong, but they give different AP numbers for the same model, and the docs say plainly https://rfdetr.roboflow.com/latest/learn/train/training-parameters/ keypoint-preview-parameters that “\ f\ lat custom sigmas are not directly comparable to official COCO benchmark numbers” https://rfdetr.roboflow.com/latest/learn/train/training-parameters/ keypoint-preview-parameters . Pick σ values on purpose, write them down, and compare models only with the same ones. An accuracy number for keypoints is only as meaningful as the tolerances behind it. At the end, RF-DETR keeps the highest-scoring query-class pairs. Each kept query is read out according to its class: the 17 person slots for a person, nothing beyond the box for a car. Scores are fused with the joints’ spread step 7 . No NMS and no grouping step: the box, class, and skeleton all come from the same query. python import cv2import supervision as svfrom rfdetr import RFDETRKeypointPreviewmodel = RFDETRKeypointPreview image rgb = cv2.cvtColor cv2.imread "street.jpg" , cv2.COLOR BGR2RGB key points = model.predict image rgb, threshold=0.5 hide joints the model thinks an annotator would not have placedkey points.visible = key points.keypoint confidence 0.5annotated = sv.VertexAnnotator .annotate image rgb.copy , key points The result is a supervision https://supervision.roboflow.com/latest/keypoint/core/ KeyPoints object: xy holds the joints in pixels, keypoint confidence the confidence score of each joint, detection confidence the fused object score that the threshold filters on, and data "xyxy" the boxes. Two practical notes. Because the object score is fused with the joints' spread, a threshold of 0.5 doesn't mean the same as in a detection-only model, so tune it on your own images. And the per-joint confidence is yours to apply: the docs suggest thresholding keypoint confidence and setting visible, as above, to hide unlikely joints. One trap is fine-tuning a mixed schema like our street. In 1.11.2, the score fusion also runs for box-only classes a 0 in the schema . They have no joints, so their score s becomes s / 1 + s , which is always below 0.5. With threshold=0.5, every car would disappear. Until that changes, predict with a low threshold and filter box-only classes with their own cut-off. To fine-tune on your own skeleton, pass num keypoints per class to the model, one entry per class, with 0 for box-only classes, plus the left-right keypoint flip pairs so flipped training images swap the right joints. Both COCO keypoint JSON and Ultralytics YOLO pose labels are accepted; the custom dataset guide https://rfdetr.roboflow.com/latest/learn/train/ keypoint-preview-custom-datasets has a schema helper that reads them from your files. Two training defaults are worth knowing: the keypoint training config smooths the validation keypoint AP before picking the best checkpoint, and skips the first 10 epochs when picking it, because that metric is noisy early in fine-tuning. Here are the reported COCO person-keypoint numbers next to Ultralytics’ pose models at a comparable tier. There is only one RF-DETR Keypoint size, so the table puts it next to the YOLO models closest in latency YOLO26-pose X and YOLO11-pose X and the next size down YOLO26-pose L . AP50:95 is the OKS-based average precision from step 9. We measure latency on an NVIDIA T4 GPU with TensorRT at half precision FP16 , one image at a time. In the RF-DETR benchmarks, RF-DETR Keypoint scores 71.8 AP50:95 at 9.7 ms, against 71.0 at 9.8 ms for YOLO26-pose X, with 40.7 million parameters against 57.6 million. Compared with RF-DETR, YOLO26-pose L gives up 2.6 points for about 40% less latency. Four caveats, and the second one matters. First, the team that built RF-DETR measured these numbers. Second, Ultralytics reports higher accuracy and higher latency for its own models: 71.6 AP50:95 for YOLO26x-pose https://docs.ultralytics.com/tasks/pose/ , at 12.2 ms on a T4 with TensorRT10, using the NMS-free head. The RF-DETR benchmarks re-run every model in one setup and note that “numbers may therefore differ from vendor-reported figures” https://rfdetr.roboflow.com/latest/learn/benchmarks/ . If you take Ultralytics’ figure for YOLO26x-pose instead, the AP gap shrinks from 0.8 to 0.2 points, and the latency comparison depends on whose setup you trust. Third, RF-DETR runs at 576 pixels, while the YOLO models run at 640. Fourth, this is a preview: one size, one pretrained skeleton, and an API that may change. Ultralytics has also announced YOLO27 https://docs.ultralytics.com/models/yolo27/ as an unreleased preview, with only preliminary pose results at the time of writing. A 0.2-to-0.8-point lead is a reason to test, not a reason to switch. Back to the question from the start: whose knee is it? Here is how the main families of multi-person pose models answer it. Top-down models such as HRNet https://arxiv.org/abs/1902.09212 first run a person detector, crop each person, and find the joints in the crop. Ownership is free, but the pose network runs once per person, so the cost grows with the crowd. Bottom-up models find every joint in the image first and group them afterward. OpenPose https://arxiv.org/abs/1812.08008 uses “Part Affinity Fields PAFs , to learn to associate body parts with individuals” https://arxiv.org/abs/1812.08008 ; Associative Embedding https://arxiv.org/abs/1611.05424 gives each joint a tag and groups joints with similar tags. One-stage models such as YOLO-Pose https://arxiv.org/abs/2204.06806 predict joints from the same grid cell as the box, so “each bounding box has an associated pose” https://arxiv.org/abs/2204.06806 ; duplicates are then removed with NMS, though YOLO26 also offers an optional head that does without it. Query-based models such as PETR https://openaccess.thecvf.com/content/CVPR2022/html/Shi End-to-End Multi-Person Pose Estimation With Transformers CVPR 2022 paper.html , ED-Pose https://arxiv.org/abs/2302.01593 , and GroupPose let each query predict one whole pose and need no clean-up step. RF-DETR Keypoint belongs to the last family and builds on GroupPose. What it does differently: Strip away the details and RF-DETR Keypoint does one thing differently from most pose models: it never has to work out whose knee it is, because every knee is born inside one object’s question. Here is the path from photo to skeleton, one step per animation: If you need poses and boxes from one fast model, or different skeletons for different classes, this design fits well. If you need several model sizes or a stable API today, remember that this is a preview, and YOLO26-pose offers a range of sizes. If you fine-tune, choose your OKS σ values intentionally before you compare anything. Co-authored by Claude. The model and how to run it are on the RF-DETR keypoint page https://rfdetr.roboflow.com/latest/learn/run/keypoints/ . The same street, the same queries, and masks instead of joints are in the segmentation post https://claude.ai/chat/LINK-TO-SEGMENTATION-POST if you want to see how the other heads work. What would you track with your own skeleton: a hand, a horse, a forklift? Tell me in the comments, and if you want to try it, the code is at github.com/roboflow/rf-detr https://github.com/roboflow/rf-detr . Where the Knees Are: How a Detection Transformer Reads Poses on a Street https://pub.towardsai.net/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street-de2baccd3a96 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.