{"slug": "where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street", "title": "Where the Knees Are: How a Detection Transformer Reads Poses on a Street", "summary": "RF-DETR 1.11.2's RFDETRKeypointPreview model, an early-access keypoint detector from Roboflow that runs at 576 × 576 pixels and predicts the 17 COCO person keypoints, is detailed in a technical walkthrough by an author who maintains RF-DETR at Roboflow. The model resizes images into 2,304 patches of 12 × 12 pixels, tokenizes them through a DINOv2-initialized vision transformer backbone, and builds a 256-number-per-patch memory grid that a second projector duplicates so the box-finding and joint-finding heads read separate copies. The walkthrough notes the RF-DETR paper covers detection and segmentation rather than keypoints, so the mechanism is drawn from the 1.11.2 source code, and that fine-tuning on custom keypoint datasets is the model's primary intended use case.", "body_md": "Look at a photo of a street. A cyclist rides across a zebra crossing. One car waits before the crossing, another drives away, and a dog sits on the pavement. A detector draws a box around each of them. That tells you where they are. It does not tell you whether the cyclist is pedalling or reaching for the brake, or whether the dog is sitting or about to run.\n\nFor that you need keypoints.\n\nKeypoints are named points on an object: a nose, a left knee, a tail tip. Join them with lines, and you get a skeleton. Finding them for every person in a photo is called [pose estimation](https://en.wikipedia.org/wiki/Articulated_body_pose_estimation); the same idea works for animals, hands, or the corners of a parcel. It is how sports apps count reps, how ergonomics tools spot a bad lifting posture, and how biologists track how animals move.\n\nThe hard part is not finding knees. It is knowing whose knee it is.\n\n**Pause and ponder.** Two people stand close together, and their legs overlap. A model has found four knees. Which two belong to the person on the left?\n\nPose models have given different answers to that question over the years, and RF-DETR’s answer is the thread of this post. It is told in nine steps, and each step has its own animation. If you only scroll through the animations, you should still get the main idea; the text adds the why and the numbers. No prior knowledge of transformers is needed.\n\n**Scope.** RF-DETR 1.11.2 from [PyPI](https://pypi.org/project/rfdetr/), the RFDETRKeypointPreview model. The docs call it [\"an early-access release. Fine-tuning on custom keypoint datasets is the primary intended use case.\"](https://rfdetr.roboflow.com/latest/learn/run/keypoints/) It comes in one size, works at 576 × 576 pixels, and its pretrained checkpoint predicts the 17 COCO person keypoints. The [RF-DETR paper](https://arxiv.org/abs/2511.09554) covers detection and segmentation, not keypoints, so the mechanism here comes from the 1.11.2 source code. Accuracy and latency numbers come from the [RF-DETR benchmarks page](https://rfdetr.roboflow.com/latest/learn/benchmarks/). Our street uses three classes with three skeletons (cyclist 17 joints, dog 8, car none): that is the kind of model you would get by fine-tuning, not the pretrained checkpoint. All street visuals are drawn, not model output; uncertainty ellipses and guesses are illustrative. Disclosure: I maintain RF-DETR at Roboflow.\n\nBefore the details, here is the full path from photo to skeletons in one animation. Each stage takes the previous stage’s output as its input, does one job, and hands its result on.\n\nThe rest of the post walks through these stages one at a time, with one animation per step.\n\nA neural network does not see a street. It sees numbers. RF-DETR Keypoint first resizes the photo to 576 × 576 pixels and cuts it into small 12 × 12 pixel squares, called patches. That makes a grid of 48 × 48 = 2,304 patches.\n\nEach patch becomes a list of numbers called a token. The tokens pass through the backbone, a [vision transformer](https://en.wikipedia.org/wiki/Vision_transformer)initialised from [DINOv2](https://arxiv.org/abs/2304.07193). Meta trained DINOv2 on 142 million curated images without any labels, so before RF-DETR has seen a single annotated skeleton, its backbone already produces general-purpose visual features. Inside the backbone, tokens compare themselves with each other, so a patch on the rider’s knee can account for the rest of the leg.\n\nAt the end, a small module called the projector packs everything into a 48 × 48 grid of descriptions, 256 numbers per patch. RF-DETR calls this grid the memory. The keypoint model has a second projector that builds a second memory of the same size from the same backbone output. The box-finding part reads the first; the joint-finding part reads the second. We will meet it again in step 6.\n\n**A model never sees the street, only grids of descriptions…**\n\nMany detectors guess a box at every position in the image and then delete the duplicates afterward. RF-DETR asks questions instead.\n\nIt holds a fixed set of 100 learned queries. Think of each query as a question: “Is there an object I should be responsible for? If so, what is it, and where?” Every query answers with one box and a score for each class, or with “nothing here”, written ∅.\n\nEach query starts from a reference box, chosen from the most promising regions of the memory, and works through four decoder layers. In each layer, it reads evidence from the memory, compares notes with the other queries, and updates its answer. It always predicts its box as a shift and resize of that same reference box. Remember the reference box: the joints will be measured from it too.\n\nDuring training, each real object is paired with exactly one query (once in each of 13 parallel query groups, which gives the model more practice), chosen by the [Hungarian algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) so the total fit across all objects is as good as possible. Every other query is taught to answer “nothing here”. A second query that also claims the dog is penalized, so the model learns not to repeat itself, and no NMS (non-maximum suppression, the clean-up step that deletes overlapping duplicates) is needed.\n\nSo far, this is exactly how RF-DETR detects objects, with 100 queries here instead of the detector’s 300; I covered that side in [an earlier post](https://medium.com/@jborovec/300-questions-not-8-400-guesses-how-a-detection-transformer-sees-a-street-685472f6eca8) if you want more depth. Keypoints add one idea on top, and it answers the “whose knee” question before it can even come up: **every query brings its own joints.**\n\n**A question that owns an object can also own its knees.**\n\nNext to each query, the model keeps one extra token per joint in the schema. In our street, the schema is cyclist 17, car 0, dog 8, so every one of the 100 queries carries 25 joint tokens. Each token is a slot that will become one joint: “left knee of whatever this query finds”.\n\nA slot starts from a learned token for its joint type, one for “nose”, one for “left knee”, and so on. The token is then scaled and shifted by numbers computed from its own query, a trick called adaptive layer normalization (AdaLN). So the cyclist’s query and a car’s query both start their “left knee” slot from the same learned knee, each adapted to its own query. What the object in front of it actually looks like comes in later, in the decoder layers.\n\nThe slots are not read for every class. A query that ends up as “car” has its joints ignored; one that ends up as “dog” is read from the 8 dog slots. A zero in the schema means “this class is box only”. The pretrained preview checkpoint has just one class with one skeleton, the 17 COCO person keypoints; the street’s three classes come from fine-tuning on your own labels.\n\n**The question of whose knee it is never arises: each knee slot is born inside one query.**\n\nIn every decoder layer, each query and its joint tokens run a round of [self-attention](<https://en.wikipedia.org/wiki/Attention_(machine_learning)>): every token looks at the others in its group and takes in what it finds useful. A knee can use where the hip and the ankle are. The query also participates, so it learns from its own joints, and the joints learn from it what the object is.\n\nTwo walls keep this tidy. Slots of different skeletons inside one query are masked from each other, so a query's person slots don't mix with its dog slots. And joints from different objects don't attend to each other at all: the code has an option for that, inter_instance_kp_attn, and the preview model ships with it off.\n\nThis idea comes from [GroupPose](https://arxiv.org/abs/2308.07313), which uses an instance query plus keypoint queries for each pose and two kinds of group self-attention: one within each person, and one across people for the same joint type. RF-DETR’s preview keeps the first. Of the second, it keeps only the object-to-object part, the ordinary detection self-attention; joint-to-joint attention across objects is switched off.\n\n**Pause and ponder.** Joints never look at other objects. So why don’t two overlapping cyclists end up with each other’s knees?\n\nBecause each object query is still a detection query: in the detection part of every layer, the queries compare notes with each other, which keeps them on separate objects. The joints inherit that separation through their own query. The boxes do the bookkeeping; the joints do the anatomy.\n\n**Joints learn from each other only inside their own object.**\n\nTalking to its neighbors tells a knee roughly where it should be. To know where it actually is, it has to look at the image.\n\nAfter the talking, every joint token reads the second memory with [deformable attention](https://arxiv.org/abs/2010.04159): instead of looking at all 2,304 cells, it learns to sample a few points around its query’s reference box and weighs what it finds there. It is the same reading mechanism the object queries use on the first memory. In the preview model, each joint reads 32 sampled points (16 attention heads, 2 points each) instead of all 2,304 cells. The second projector lets the joints get descriptions tuned for fine anatomy, while the box-finding part keeps its own.\n\nThen each joint is placed. The model predicts an offset, and the joint lands at the reference box’s center plus that offset times the box’s width and height. An offset of (−0.2, 0.3) means “a fifth of a box-width left of center, three tenths of a box-height down”, and that holds for a cyclist who fills the frame and for one far down the street.\n\n**A joint is not a pixel; it is a position inside its object’s box.**\n\nA pose model always has to answer for every joint, even one hidden behind the body. RF-DETR answers with more than a position.\n\n**For each joint, the model predicts:**\n\nThe spread is trained with a Gaussian [negative log-likelihood](https://en.wikipedia.org/wiki/Likelihood_function#Log-likelihood): the model is penalized both for missing a joint and for claiming to be sure when it is not. That trains it to be honest about how sure it is.\n\nFinally, the spreads also adjust the object’s score. In RF-DETR 1.11.2, the score is combined with the average size of its joints’ ellipses (weighted by how findable each joint is), so a confident box with wobbly joints ends up with a lower score than an equally confident box with tight joints. The fused score is then squeezed back into the range 0 to 1. Ultralytics’ YOLO26-pose also models uncertainty, with [Residual Log-Likelihood Estimation](https://arxiv.org/abs/2107.11291): [“Integrates Residual Log-Likelihood Estimation (RLE) for more accurate keypoint localization”](https://docs.ultralytics.com/models/yolo26/).\n\n**A position without confidence is half an answer.**\n\nTraining extends the pairing from step 3. When the model decides which query belongs to which real object, it now compares joints too, not only classes and boxes.\n\nThe pairing cost and the training losses use the same ingredients. The distance between each predicted and true joint is measured with an [L1 distance](https://en.wikipedia.org/wiki/Taxicab_geometry) and divided by the square root of the true box’s area, so a far, small cyclist counts as much as a near, big one. A [binary cross-entropy](https://en.wikipedia.org/wiki/Cross-entropy) for findable and one for visible. And the negative log-likelihood of the spread. By default, each keypoint loss has weight 1.0.\n\n**Pause and ponder.** Why divide by the box size? \n\nA 10-pixel miss is a 10-pixel miss.\n\nNot to a person looking at the result. Ten pixels is a small miss on a cyclist who fills the frame and a whole leg on one far down the street. Dividing by the box size measures the miss in “object sizes”, the same idea the benchmark uses in the next step. Positions are compared only for labeled joints; a joint with no label has no position to compare against.\n\n**One-to-one pairing keeps duplicates from being rewarded; adding the joints makes it pick the query with the best pose.**\n\nHow do you score a predicted pose? For boxes, there is [IoU](https://en.wikipedia.org/wiki/Jaccard_index), the overlap of two boxes. For keypoints, COCO defines [Object Keypoint Similarity](https://cocodataset.org/#keypoints-eval), OKS, which, in COCO’s words, [“plays the same role as the IoU”](https://cocodataset.org/#keypoints-eval).\n\nEach true joint gets a score between 0 and 1 that falls off like a bell curve with the distance d to the guess:\n\nexp(−d² / (2 s² κ²))\n\nHere s² is the object’s area and κ = 2σ, where σ is a constant for each joint type. COCO measured those constants from how much human annotators disagreed on 5,000 redundantly annotated images: eyes 0.025, nose 0.026, shoulders 0.079, hips 0.107. People agree on where an eye is far more than on where a hip is, so a hip can be further off for the same score. OKS averages over the labeled joints. Average precision then counts a pose as correct if its OKS passes a threshold, and averages over thresholds from 0.50 to 0.95. That is the AP50:95 in the comparison below.\n\nThat brings us to the most practical tip in this post. When you fine-tune on your own skeleton, COCO’s σ values do not apply, and RF-DETR needs to know yours to report a meaningful AP. If you pass none and your skeleton does not have 17 joints, version 1.11.2 falls back to a uniform σ of 0.05 for every joint and warns you. A custom skeleton that happens to have 17 joints silently gets COCO’s person values. The schema helpers in the docs pass a flat 0.1 instead. Neither is wrong, but they give different AP numbers for the same model, and [the docs say plainly](https://rfdetr.roboflow.com/latest/learn/train/training-parameters/#keypoint-preview-parameters) that [“\\[f\\]lat custom sigmas are not directly comparable to official COCO benchmark numbers”](https://rfdetr.roboflow.com/latest/learn/train/training-parameters/#keypoint-preview-parameters). Pick σ values on purpose, write them down, and compare models only with the same ones.\n\n**An accuracy number for keypoints is only as meaningful as the tolerances behind it.**\n\nAt the end, RF-DETR keeps the highest-scoring query-class pairs. Each kept query is read out according to its class: the 17 person slots for a person, nothing beyond the box for a car. Scores are fused with the joints’ spread (step 7). No NMS and no grouping step: the box, class, and skeleton all come from the same query.\n\n``` python\nimport cv2import supervision as svfrom rfdetr import RFDETRKeypointPreviewmodel = RFDETRKeypointPreview()image_rgb = cv2.cvtColor(cv2.imread(\"street.jpg\"), cv2.COLOR_BGR2RGB)key_points = model.predict(image_rgb, threshold=0.5)# hide joints the model thinks an annotator would not have placedkey_points.visible = key_points.keypoint_confidence > 0.5annotated = sv.VertexAnnotator().annotate(image_rgb.copy(), key_points)\n```\n\nThe result is a [supervision](https://supervision.roboflow.com/latest/keypoint/core/) KeyPoints object: xy holds the joints in pixels, keypoint_confidence the confidence score of each joint, detection_confidence the fused object score that the threshold filters on, and data[\"xyxy\"] the boxes. Two practical notes. Because the object score is fused with the joints' spread, a threshold of 0.5 doesn't mean the same as in a detection-only model, so tune it on your own images. And the per-joint confidence is yours to apply: the docs suggest thresholding keypoint_confidence and setting visible, as above, to hide unlikely joints.\n\nOne trap is fine-tuning a mixed schema like our street. In 1.11.2, the score fusion also runs for box-only classes (a 0 in the schema). They have no joints, so their score s becomes s / (1 + s), which is always below 0.5. With threshold=0.5, every car would disappear. Until that changes, predict with a low threshold and filter box-only classes with their own cut-off.\n\nTo fine-tune on your own skeleton, pass num_keypoints_per_class to the model, one entry per class, with 0 for box-only classes, plus the left-right keypoint_flip_pairs so flipped training images swap the right joints. Both COCO keypoint JSON and Ultralytics YOLO pose labels are accepted; the [custom dataset guide](https://rfdetr.roboflow.com/latest/learn/train/#keypoint-preview-custom-datasets) has a schema helper that reads them from your files. Two training defaults are worth knowing: the keypoint training config smooths the validation keypoint AP before picking the best checkpoint, and skips the first 10 epochs when picking it, because that metric is noisy early in fine-tuning.\n\nHere are the reported COCO person-keypoint numbers next to Ultralytics’ pose models at a comparable tier. There is only one RF-DETR Keypoint size, so the table puts it next to the YOLO models closest in latency (YOLO26-pose X and YOLO11-pose X) and the next size down (YOLO26-pose L). AP50:95 is the OKS-based average precision from step 9. We measure latency on an NVIDIA T4 GPU with TensorRT at half precision (FP16), one image at a time.\n\nIn the RF-DETR benchmarks, RF-DETR Keypoint scores 71.8 AP50:95 at 9.7 ms, against 71.0 at 9.8 ms for YOLO26-pose X, with 40.7 million parameters against 57.6 million. Compared with RF-DETR, YOLO26-pose L gives up 2.6 points for about 40% less latency.\n\nFour caveats, and the second one matters. First, the team that built RF-DETR measured these numbers. Second, Ultralytics reports higher accuracy and higher latency for its own models: [71.6 AP50:95 for YOLO26x-pose](https://docs.ultralytics.com/tasks/pose/), at 12.2 ms on a T4 with TensorRT10, using the NMS-free head. The RF-DETR benchmarks re-run every model in one setup and note that [“numbers may therefore differ from vendor-reported figures”](https://rfdetr.roboflow.com/latest/learn/benchmarks/). If you take Ultralytics’ figure for YOLO26x-pose instead, the AP gap shrinks from 0.8 to 0.2 points, and the latency comparison depends on whose setup you trust. Third, RF-DETR runs at 576 pixels, while the YOLO models run at 640. Fourth, this is a preview: one size, one pretrained skeleton, and an API that may change. Ultralytics has also announced [YOLO27](https://docs.ultralytics.com/models/yolo27/) as an unreleased preview, with only preliminary pose results at the time of writing.\n\n**A 0.2-to-0.8-point lead is a reason to test, not a reason to switch.**\n\nBack to the question from the start: whose knee is it? Here is how the main families of multi-person pose models answer it.\n\n**Top-down** models such as [HRNet](https://arxiv.org/abs/1902.09212) first run a person detector, crop each person, and find the joints in the crop. Ownership is free, but the pose network runs once per person, so the cost grows with the crowd. **Bottom-up** models find every joint in the image first and group them afterward. [OpenPose](https://arxiv.org/abs/1812.08008) uses [“Part Affinity Fields (PAFs), to learn to associate body parts with individuals”](https://arxiv.org/abs/1812.08008); [Associative Embedding](https://arxiv.org/abs/1611.05424) gives each joint a tag and groups joints with similar tags. **One-stage** models such as [YOLO-Pose](https://arxiv.org/abs/2204.06806) predict joints from the same grid cell as the box, so [“each bounding box has an associated pose”](https://arxiv.org/abs/2204.06806); duplicates are then removed with NMS, though YOLO26 also offers an optional head that does without it. **Query-based** models such as [PETR](https://openaccess.thecvf.com/content/CVPR2022/html/Shi_End-to-End_Multi-Person_Pose_Estimation_With_Transformers_CVPR_2022_paper.html), [ED-Pose](https://arxiv.org/abs/2302.01593), and GroupPose let each query predict one whole pose and need no clean-up step.\n\nRF-DETR Keypoint belongs to the last family and builds on GroupPose. What it does differently:\n\nStrip away the details and RF-DETR Keypoint does one thing differently from most pose models: it never has to work out whose knee it is, because every knee is born inside one object’s question. Here is the path from photo to skeleton, one step per animation:\n\nIf you need poses and boxes from one fast model, or different skeletons for different classes, this design fits well. If you need several model sizes or a stable API today, remember that this is a preview, and YOLO26-pose offers a range of sizes. If you fine-tune, choose your OKS σ values intentionally before you compare anything.\n\nCo-authored by Claude. The model and how to run it are on the [RF-DETR keypoint page](https://rfdetr.roboflow.com/latest/learn/run/keypoints/). The same street, the same queries, and masks instead of joints are in [the segmentation post](https://claude.ai/chat/LINK-TO-SEGMENTATION-POST) if you want to see how the other heads work.\n\nWhat would you track with your own skeleton: a hand, a horse, a forklift? Tell me in the comments, and if you want to try it, the code is at [github.com/roboflow/rf-detr](https://github.com/roboflow/rf-detr).\n\n[Where the Knees Are: How a Detection Transformer Reads Poses on a Street](https://pub.towardsai.net/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street-de2baccd3a96) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street", "canonical_source": "https://pub.towardsai.net/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street-de2baccd3a96?source=rss----98111c9905da---4", "published_at": "2026-10-25 15:01:02+00:00", "updated_at": "2026-10-25 15:13:44.710119+00:00", "lang": "en", "topics": ["computer-vision", "artificial-intelligence", "machine-learning", "ai-research"], "entities": ["RF-DETR", "Roboflow", "RFDETRKeypointPreview", "DINOv2", "Meta", "PyPI", "COCO"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street", "markdown": "https://wpnews.pro/news/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street.md", "text": "https://wpnews.pro/news/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street.txt", "jsonld": "https://wpnews.pro/news/where-the-knees-are-how-a-detection-transformer-reads-poses-on-a-street.jsonld"}}