{"slug": "human-object-interaction-detection-with-rf-detr", "title": "Human-Object Interaction Detection with RF-DETR", "summary": "Roboflow published a tutorial demonstrating human-object interaction (HOI) detection without training an interaction model, using RF-DETR to detect workers, forklifts, pallets, and carts (77.5% mAP@50) and Gemini 2.5 Pro to infer the action, outputting an annotated warehouse image with bounding boxes and an interaction summary for safety review. The approach avoids expensive interaction labels and supports zero-shot detection of unseen interactions, though the output is a VLM judgment in text and should be reviewed rather than treated as ground truth.", "body_md": "*You can build human-object interaction detection without training an interaction model: train RF-DETR to detect workers, forklifts, pallets, and carts (77.5% mAP@50 here), then pass the annotated image to Gemini 2.5 Pro in a Roboflow Workflow to describe what each person appears to be doing. The output is one annotated warehouse image with bounding boxes and a short interaction summary suitable for safety review.*\n\n## What Is Human-Object Interaction Detection?\n\nHuman-object interaction (HOI) detection is a [computer vision task](https://blog.roboflow.com/key-tasks-in-computer-vision/) that identifies people, the objects around them, and the action connecting the two. A standard object detector tells you what is in an image. HOI detection tells you who is doing what with which object. The output is written as a triplet: human, verb, object - such as person, drive, forklift or person, push, cart.\n\nThe verb is the hard part. Two images can contain the same person and the same forklift at the same distance and show different situations: in one the person is operating the forklift, in the other they are walking past it. The visual evidence separating those cases is small: hand position, body orientation, whether the person is seated in the cab.\n\nResearch systems learn the verb from labeled interaction data. [The two standard benchmarks](https://arxiv.org/html/2305.09948v5?ref=blog.roboflow.com) are HICO-DET, which covers 600 interaction categories across 80 object classes, and V-COCO, a smaller benchmark built on COCO images with 29 action categories.\n\nModels trained on them fall into two families. Two-stage methods run an object detector first and then classify each person-object pair. One-stage methods predict the full triplet directly, most recently with transformers.\n\nThis tutorial takes a different route. Instead of training an interaction classifier, we train [RF-DETR](https://rfdetr.roboflow.com/latest/?ref=blog.roboflow.com) to detect the people and objects, then ask [Gemini 2.5 Pro](https://playground.roboflow.com/models/google/gemini-2-5-pro?ref=blog.roboflow.com) to infer the verb from the annotated scene.\n\n### HOI detection vs. action recognition\n\nHOI detection is a different task from human activity recognition. Activity recognition classifies what a person is doing, usually from video and often from pose, without tying the action to a specific object. Activity recognition returns \"lifting.\" HOI detection returns \"lifting that pallet.\" The distinction matters for [warehouse monitoring](https://roboflow.com/industries/warehousing?ref=blog.roboflow.com) because safety rules are written about person-equipment pairs (who is on the forklift, who is standing near it), not about body movements in isolation.\n\n### Why a detector and a VLM instead of an HOI model?\n\nThe classic approach is to label the verbs: collect warehouse images, annotate every person-object pair with an interaction class, and train a dedicated HOI model. That works, but interaction labels are expensive, and a model trained on a fixed verb list can only recognize the interactions you labeled.\n\nRecent HOI research has moved toward [zero-shot detection](https://blog.roboflow.com/what-is-zero-shot-object-detection/) with [vision-language models](https://playground.roboflow.com/models/task/vision-language?ref=blog.roboflow.com), and that is what this workflow does. RF-DETR does the localization, which VLMs are still weak at. Gemini names the interaction, and because it is not limited to a fixed verb list, the same workflow can report that a worker appears to be inspecting a fuse box or unloading a pallet without a single interaction label in the training data.\n\nThe trade-off is determinism. A trained HOI model returns a fixed class with a confidence score. A VLM returns a judgment in text. That is why the prompt in Step 6 limits Gemini to visible evidence and cautious wording, and why the output should be reviewed rather than treated as ground truth.\n\n## Why Human-Object Interaction Detection Matters\n\nWarehouse workers regularly interact with forklifts, pallets, carts, boxes, and other equipment. Standard object detection can identify these objects, but it cannot explain whether a worker is operating a forklift, pushing a cart, or simply standing nearby.\n\nThis context is important for both safety and operational monitoring. [ OSHA](https://www.osha.gov/laws-regs/federalregister/1995-03-14?ref=blog.roboflow.com) has historically estimated that powered industrial truck incidents are associated with approximately 34,900 serious injuries and 85 fatalities annually in the United States.\n\nThis article builds two warehouse human-object interaction systems in [Roboflow Workflows](https://roboflow.com/workflows/build?ref=blog.roboflow.com), both using RF-DETR for detection and Gemini for interpretation. The first analyzes the whole scene: RF-DETR detects workers and warehouse objects, and Gemini 2.5 Pro describes the likely interactions in the annotated image. The second narrows to one decision: it crops each detected person near a forklift and asks Gemini to classify the interaction as safe or unsafe, returning a structured safety report.\n\nTogether, the two workflows can detect and report:\n\n- Apparent relationships between workers, forklifts, pallets, and carts\n- Possible interactions such as operating a forklift, pushing a cart, or working near a pallet\n- Per-person safe or unsafe classifications with a JSON safety report\n- Interactions that may require human review, logged through Vision Events\n\nThe first tutorial's output is an annotated warehouse image with a Gemini-generated interaction summary. [Here's the workflow we'll build.](https://app.roboflow.com/workflows/embed/...?ref=blog.roboflow.com)\n\n### Step 1: Prepare the Dataset\n\nWe use the [warehouse computer vision dataset](https://universe.roboflow.com/jjjj-jmgpe/warehouse-buhqm?ref=blog.roboflow.com) from\n\n[. The project contains approximately 1,200 warehouse images and 12 original object-detection classes. For this workflow, we use five classes relevant to the selected warehouse interaction examples.](https://universe.roboflow.com/?ref=blog.roboflow.com)\n\n__Roboflow Universe__- person\n- forklift\n- pallet\n- cart\n- fuse_box\n\nThe dataset is suitable for this project because it contains people alongside multiple warehouse objects with which they may interact. The images include workers near forklifts, pallets, carts, and other equipment. This provides a broader foundation for human-object interaction analysis than a dataset containing only people and forklifts.\n\nFork the dataset into your Roboflow workspace. Open the Train tab, select Custom Training, choose RF-DETR, and set the model size to Small.\n\nGenerate a new dataset version and configure a 70/15/15 split for training, validation, and testing.\n\nEnable:\n\n- Auto-orientation\n- Resize to 512 × 512\n\nThese preprocessing steps ensure that all images have a consistent orientation and resolution, providing standardized inputs for RF-DETR training.\n\n### Step 2: Train the RF-DETR Model\n\nDuring training, RF-DETR learns to locate and classify each worker and warehouse object using the bounding-box annotations. The model returns a class label, confidence score, and bounding box for each detected person, forklift, pallet, cart, or fuse box.\n\nThese instance-level detections provide visual context for the interaction-analysis stage. The detector itself does not understand that a person is pushing a cart or operating a forklift. Instead, the workflow draws the detections on the complete image and sends the annotated scene to Gemini for contextual interpretation.\n\nThis separation keeps the system clear: RF-DETR identifies who and what are present, while Gemini interprets how they appear to be interacting.\n\n### Step 3: Evaluate Model Performance\n\nThe trained RF-DETR Small model achieved the following validation results on the warehouse object-detection task.\n\nThe model achieved 77.5% mAP@50, 92.1% precision, 71.5% recall, and an 80.5% F1 score on the validation set.\n\nThe high precision score indicates that most predicted bounding boxes correspond to real warehouse objects. This is important because false detections could cause Gemini to describe objects or interactions that are not actually present.\n\nRecall is lower than precision, meaning the model may miss some visible workers or objects. A missed person, cart, or pallet can reduce the context available to Gemini and lead to an incomplete interaction summary.\n\nThe mAP@50 and F1 scores indicate that the model provides a useful foundation for detecting workers and common warehouse objects. However, performance may vary by class. Larger and more common objects, such as people and forklifts, may be easier to detect than smaller or less frequent objects.\n\nBefore deployment, test the model on images from the target warehouse. Camera angle, lighting, object size, aisle congestion, partial occlusion, worker uniforms, and warehouse layout may differ from the training data.\n\nThese metrics evaluate only the RF-DETR detection stage. They do not measure whether Gemini correctly identifies the interaction between a worker and an object.\n\n### Step 4: Deploy to Roboflow Workflows\n\nAfter evaluating the model, deploy it in Roboflow Workflows to build the interaction-analysis pipeline.\n\nThe workflow accepts one warehouse image, runs RF-DETR, draws bounding boxes and class labels, sends the annotated image to Gemini 2.5 Pro, and overlays the generated interaction summary on the final image.\n\nOpen the trained model and click Deploy Model. Select Customize With Logic to open the Workflow editor with the model already connected.\n\nThe completed workflow follows this structure:\n\nThe image input connects to the RF-DETR model and the Bounding Box Visualization block. The model predictions connect to both visualization blocks. The labeled image is then passed to Gemini and used as the base image for the final text overlay. Gemini’s response is displayed on the image, which is returned as annotated_image.\n\nThis workflow does not use tracking, proximity calculations, zones, crops, or a Custom Python block. Gemini analyzes the complete annotated scene and describes the apparent interactions.\n\n### Step 5: Configure the Detection Visualizations\n\nAdd a Bounding Box Visualization block after the RF-DETR model.\n\n**Connect:**\n\nImage: inputs.image\n\nPredictions: rfdetr_object_detection_model.predictions\n\nSet the bounding-box thickness to 1 and the color axis to CLASS. Thin boxes keep the image readable, while class-based colors make it easier to distinguish people, forklifts, pallets, and other objects.\n\nNext, add a Label Visualization block. Use the Bounding Box Visualization output as the base image and connect the RF-DETR predictions.\n\nThe visualization blocks do not determine the interaction. They make the model detections visible before Gemini analyzes the scene. For example, Gemini may observe that a detected person is seated inside a detected forklift or standing beside a detected pallet.\n\n### Step 6: Configure Gemini 2.5 Pro\n\nAdd a Google Gemini block after Label Visualization.\n\n**Use these settings:**\n\nImage: label_visualization.image\n\nModel: Gemini 2.5 Pro\n\nTask type: Visual Question Answering\n\nTemperature: 0.1\n\n**Use this prompt:**\n\n```\nInspect the annotated warehouse image and describe each visible person’s apparent interaction with nearby objects, such as forklifts, pallets, carts, fuse boxes.\n\nOnly report interactions supported by the image. Do not treat proximity alone as an interaction. Use “appears to be” when uncertain.\n\nReturn:\n\nWarehouse Interaction Summary:\n- [interaction]\n- [interaction]\n\nReview:\n[State whether any interaction may need human review.]\n\nIf none are clear, return:\n- No clear human-object interaction detected.\n\nKeep the response under 70 words.\n```\n\nThe prompt asks Gemini to describe visible relationships rather than simply list detected objects. It also reduces overinterpretation by stating that proximity alone does not confirm an interaction.\n\nFor example, a person near a forklift may be operating it, inspecting it, waiting beside it, or walking past. When the action is unclear, Gemini should use cautious wording.\n\nAdd a Text Display block and use label_visualization.image as the base image. Connect the Gemini output as the displayed text.\n\nThe workflow returns the resulting image as annotated_image.\n\n### Step 7: Test the Workflow\n\nClick Run and upload a warehouse image containing at least one person and one warehouse object.\n\nThe final image should contain:\n\n- RF-DETR bounding boxes\n- Object class labels\n- Gemini’s interaction summary\n- A review statement, when applicable\n\nTest several types of scenes:\n\n- A worker operating a forklift\n- A worker pushing or standing beside a cart\n- A worker loading or unloading a pallet\n- A person near an object without clearly interacting with it\n- A warehouse image with no visible person\n- A scene containing multiple workers and objects\n\nThe no-interaction and uncertain examples are especially important. They help verify that Gemini does not interpret every nearby person-object pair as a definite interaction.\n\nAlso test difficult conditions such as distant workers, overlapping detections, crowded aisles, low lighting, partial occlusion, and unusual camera angles. When the summary is incorrect, first check whether RF-DETR detected the relevant person and object correctly.\n\n## Extending the Workflow\n\nThe workflow can be extended to process frames from fixed warehouse cameras. Adding object tracking would assign persistent IDs to workers and equipment, making it possible to analyze interactions across multiple frames.\n\nProximity or zone logic could also filter the scene before Gemini analysis. For example, the workflow could identify people entering a forklift operating zone or isolate nearby person-object pairs for more focused inspection.\n\nResults could be stored with [Roboflow Vision Events](https://docs.roboflow.com/deployment/monitoring-and-analytics/vision-events?ref=blog.roboflow.com) for later review. Each event could include the original image, annotated output, detected classes, timestamp, camera identifier, and Gemini summary. Selected events could also be sent to [Slack or an internal warehouse system](https://blog.roboflow.com/vision-detections-into-plc-mes-slack-alerts/).\n\n## Use Roboflow Agent\n\nYou can also build this workflow by describing it instead of wiring it block by block. Roboflow Agent builds Workflows from a natural language prompt: ask for a workflow that runs your trained RF-DETR model, draws boxes and labels, and sends the annotated image to Gemini 2.5 Pro with your interaction prompt, and it connects the same blocks covered in Steps 4 through 6.\n\nThe agent also debugs: if a visualization block is wired to the wrong input or Gemini returns nothing, it can inspect the workflow and fix the configuration.\n\n## A Second Human-Object Interaction Workflow: Detect, Crop, Classify\n\nThis tutorial builds a detect-crop-classify Workflow. An object detector finds forklifts and people, a Dynamic Crop block isolates each person, and a VLM analyzes the crop to classify the interaction as safe or unsafe. The VLM uses the surrounding scene to make decisions that object detection alone cannot.\n\nBy the end, you'll have a trained model and a Workflow that takes a single image, classifies each forklift-person interaction as safe or unsafe, and returns the annotated image.\n\n**Dataset**\n\nGo to [ Roboflow Universe](https://universe.roboflow.com/?ref=blog.roboflow.com) and search for the\n\n[. Universe hosts hundreds of thousands of open-source computer vision datasets covering a wide range of use cases.](https://universe.roboflow.com/luzuko/forklift-person-detector?ref=blog.roboflow.com)\n\n__Forklift-Person-Detector dataset__This dataset contains two annotated classes, **forklift** and **person**, with images from indoor and outdoor warehouses and lumber yards.\n\nThe dataset covers varied lighting, occlusion, distances, and real forklift-person scenarios. Fork it into your workspace with annotations included.\n\n### Train RF-DETR\n\nOpen the Versions tab in your forked project and generate a new version. Then click Custom Train and select [ RF-DETR (Small)](https://blog.roboflow.com/rf-detr/) as the training architecture.\n\nOnce training starts, Roboflow handles the entire process in the cloud, so no local setup or GPU is required.\n\nAs training progresses, you can monitor the model's performance in real time as the [ mAP](https://blog.roboflow.com/mean-average-precision/) improves and eventually levels off.\n\nOnce training is complete, the model is ready to use in a Workflow. [Here's the workflow we'll build.](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiTklhT0xrM3BtZUt4NHlQc0FOeVciLCJ3b3Jrc3BhY2VJZCI6Im5JRk5DOGRjbU5OOXZ4d29ybWpoWTdCNjdQZTIiLCJ1c2VySWQiOiJuSUZOQzhkY21OTjl2eHdvcm1qaFk3QjY3UGUyIiwiaWF0IjoxNzg1NzczNjczfQ.KCRkdBa25QvuYwyWsTkpg5-eyOs8v0P2bgq818Wcer8?ref=blog.roboflow.com)\n\n### Build the Workflow\n\nHere's what each block does in this Workflow.\n\n**Object Detection Model:** detects forklifts and people, with confidence scores.**Detections Filter:** keeps only person detections.**Detection Offset:** expands each person box with a margin for context.**Dynamic Crop:** crops each expanded person region.**VLM Classifier:** classifies each crop as safe or unsafe.**Bounding Box Visualization:** draws boxes on all detections.**Label Visualization:** adds class and confidence labels.**Dimension Collapse:** flattens per-crop results into one list.**Custom Python Block:** merges detections and VLM output into a safety report.**Text Display:** overlays the safety status on the image.**Roboflow Vision Events:** logs each inspection for monitoring.**Outputs:** returns the labeled image and JSON safety report.\n\n### Step 1: Add the trained model as an Object Detection block\n\nOpen the **Workflows** tab and create a new Workflow. Roboflow automatically adds the **Image Input** and **Outputs** blocks.\n\nAdd an **Object Detection Model** block named defect_detector, connect inputs.image, and select your trained model version.\n\nSet the confidence threshold to **0.5**. Adjust later if needed. The block returns bounding boxes and confidence scores for detected forklifts and people.\n\n### Step 2: Add the Detections Filter\n\nAdd a Detections Filter block named person_filter. Connect Predictions to defect_detector.predictions.\n\nConfigure the filter operation in JSON to keep only detections where class_name equals person (case-insensitive):\n\n```\n{\n  \"type\": \"roboflow_core/detections_filter@v1\",\n  \"name\": \"person_filter\",\n  \"predictions\": \"$steps.defect_detector.predictions\",\n  \"operations\": [\n    {\n      \"type\": \"DetectionsFilter\",\n      \"filter_operation\": {\n        \"type\": \"StatementGroup\",\n        \"operator\": \"or\",\n        \"statements\": [\n          {\n            \"type\": \"BinaryStatement\",\n            \"left_operand\": {\n              \"type\": \"DynamicOperand\",\n              \"operand_name\": \"_\",\n              \"operations\": [\n                {\n                  \"type\": \"ExtractDetectionProperty\",\n                  \"property_name\": \"class_name\"\n                },\n                {\n                  \"type\": \"StringToLowerCase\"\n                }\n              ]\n            },\n            \"comparator\": {\n              \"type\": \"==\"\n            },\n            \"right_operand\": {\n              \"type\": \"StaticOperand\",\n              \"value\": \"person\"\n            },\n            \"negate\": false\n          }\n        ]\n      }\n    }\n  ],\n  \"operations_parameters\": {}\n}\n```\n\nOutputs only person detections for cropping and VLM analysis, while forklift detections continue to the visualization steps.\n\n### Step 3: Add Detection Offset\n\nAdd a Detection Offset block named person_crop_margin. Connect Predictions to person_filter.predictions.\n\nSet **Units** to **Percent (%)** and both offsets to **60**. This expands each crop with surrounding context, helping the VLM judge risk more accurately.\n\n### Step 4: Add Dynamic Crop\n\nAdd a Dynamic Crop block named dynamic_crop. Connect Image to Crop to inputs.image, and Regions of Interest to person_crop_margin.predictions.\n\nThis crops out each expanded person region from the original image, producing one crop per detected person, ready to send to the VLM.\n\n### Step 5: Add the VLM Classifier\n\nAdd a **Google Gemini** block named vlm_classifier. Connect dynamic_crop.crops, set **Open Prompt**, choose **Gemini 2.5 Flash-Lite**, and set **Temperature** to **0**.\n\nPrompt:\n\n```\nYou are inspecting an expanded crop around one detected person on or near a forklift. Classify the detected person as exactly one of two values: safe or unsafe.\n\nImportant cab/operator rule: If the person appears seated inside the forklift cab, operator compartment, or driver's seat, classify as safe, even if forks or a load are raised elsewhere in the crop. A seated forklift operator is safe unless there is clear visual evidence they are not in the driver's seat or are riding on the load/forks/structure.\n\nClassify as unsafe only when there is clear visual evidence that the person is outside the driver seat and is on, above, standing on, sitting on, or gripping the forks, load, mast, overhead guard, side rails, or any forklift structure other than the driver's seat, OR is on the ground directly under a raised load or clearly in the vehicle's path.\n\nIf the crop is ambiguous between a seated cab operator and a person on the forklift structure, choose safe unless you can clearly see they are outside the driver seat. If elevated above ground level and clearly not in the driver's seat, classify unsafe.\n\nReturn only this exact JSON with no extra text, no markdown code fences, no explanation: {\"status\":\"safe\"} or {\"status\":\"unsafe\"}\n```\n\nClassifies each person crop as safe or unsafe, distinguishing normal operators from hazardous positions.\n\n### Step 6: Add Dimension Collapse\n\nAdd a Dimension Collapse block named dimension_collapse. Connect Data to vlm_classifier.output.\n\nFlattens VLM crop results into one collection for image-level analysis.\n\n### Step 7: Add the Custom Python Block (safety logic)\n\nAdd a **Custom Python Block** named **Risk Check**. Connect classifications and predictions inputs, then add outputs for report, display_text, and safety_status.\n\nClick **Edit Code** to open the editor and add a block description, such as the safety risk report summary.\n\n``` python\ndef run(self, classifications, predictions):\n    def flatten_items(value):\n        if value is None:\n            return []\n        if isinstance(value, (list, tuple, set)):\n            out = []\n            for item in value:\n                out.extend(flatten_items(item))\n            return out\n        return [value]\n\n    def parse_status(value):\n        raw = value\n        if isinstance(value, dict):\n            for key in [\"output\", \"parsed_output\", \"structured_output\", \"raw_output\", \"result\", \"value\"]:\n                if key in value:\n                    raw = value.get(key)\n                    break\n            if raw is value and \"status\" in value:\n                status = str(value.get(\"status\", \"\")).strip().lower()\n                return \"unsafe\" if status == \"unsafe\" else \"safe\"\n        if isinstance(raw, dict):\n            status = str(raw.get(\"status\", \"\")).strip().lower()\n            return \"unsafe\" if status == \"unsafe\" else \"safe\"\n        text = \"\" if raw is None else str(raw).strip()\n        if text.startswith(\"```\"):\n            text = text.strip(\"`\").strip()\n            if text.lower().startswith(\"json\"):\n                text = text[4:].strip()\n        try:\n            parsed = json.loads(text)\n            status = str(parsed.get(\"status\", \"\")).strip().lower()\n            return \"unsafe\" if status == \"unsafe\" else \"safe\"\n        except Exception:\n            lower = text.lower()\n            return \"unsafe\" if \"unsafe\" in lower else \"safe\"\n\n    items_raw = flatten_items(classifications)\n\n    try:\n        people_evaluated = len(predictions)\n    except Exception:\n        people_evaluated = 0\n\n    names = []\n    boxes = []\n    try:\n        names = list(predictions.data.get(\"class_name\", [])) if hasattr(predictions, \"data\") and predictions.data is not None else []\n    except Exception:\n        names = []\n    try:\n        boxes = predictions.xyxy.tolist() if getattr(predictions, \"xyxy\", None) is not None else []\n    except Exception:\n        boxes = []\n\n    classifications_list = []\n    unsafe_count = 0\n    for i in range(people_evaluated):\n        status = parse_status(items_raw[i] if i < len(items_raw) else None)\n        if status == \"unsafe\":\n            unsafe_count += 1\n        item = {\n            \"index\": i,\n            \"detected_class\": str(names[i]) if i < len(names) else \"person\",\n            \"status\": status,\n            \"bbox_xyxy\": [float(v) for v in boxes[i]] if i < len(boxes) else []\n        }\n        classifications_list.append(item)\n\n    safety_status = \"UNSAFE\" if unsafe_count > 0 else \"SAFE\"\n\n    report = {\n        \"safety_status\": safety_status,\n        \"people_evaluated\": people_evaluated,\n        \"unsafe_count\": unsafe_count,\n        \"classifications\": classifications_list\n    }\n    display_text = f\"Status: {safety_status} | People evaluated: {people_evaluated} | Unsafe: {unsafe_count}\"\n\n    return {\"report\": report, \"display_text\": display_text, \"safety_status\": safety_status}\n```\n\nOpen the full editor:\n\nConverts VLM results into safe/unsafe labels, links them to detections, and marks the image unsafe if needed.\n\n### Step 8: Add Bounding Box and Label Visualization\n\nAdd a **Bounding Box Visualization** block. Connect the image input and defect_detector.predictions to show all detections.\n\nAdd a **Label Visualization** block, connect defect_detector.predictions, and set **Text** to **Class**.\n\n* *Adds class labels to each box, showing exactly what the detector found before safety analysis.\n\n### Step 9: Add Text Display\n\nAdd a **Text Display** block. Connect label_visualization.image and risk_check.display_text, then set the bottom-left white text overlay style.\n\n* *The image now shows both the detected boxes from Step 8 and the safety status from risk_check, all in one frame.\n\n### Step 10: Add Vision Events and configure Outputs\n\nAdd a **Roboflow Vision Events** block. Set **Custom** event type to **Forklift Person Safety Monitoring** and use risk_check.safety_status for the value and metadata.\n\n* *Logs each inspection with detections and safety status. Set outputs to output_image and safety_report.\n\n### Step 11: Configure Outputs\n\nSet two outputs: output_image from text_display.image, and safety_report from risk_check.report.\n\n* *With everything connected, the full Workflow looks like this:\n\n* *From here, every image that comes in gets a labeled result, a structured safety report, and a logged record; no extra steps needed.\n\n## Results\n\n### Test case 1: Two operators, status safe\n\nBoth people in this frame are seated inside their own forklift cabs, operating normally. Neither is standing in a forklift's path, on the forks, or near a raised load.\n\nThe safety report confirms both detections evaluated as safe, with zero unsafe flags.\n\nBoth operators are correctly read as safe, with no false alarms despite each crop being small and partially occluded by cab glass.\n\n### Test case 2: Person on the forks, status unsafe\n\nA person is riding on the raised forks themselves, not in the driver's seat, a clear and well-documented forklift hazard.\n\nThe report marks one person as unsafe and correctly classifies the seated operator as safe.\n\n* *The model separates safe operator positions from hazardous placements, catching missed safety violations\n\n## Production Deployment\n\nThe safe/unsafe threshold is based on initial testing, but real deployments will uncover edge cases the prompt has not handled yet. Reviewing borderline cases helps improve the system by refining the prompt or adding new examples instead of relying on fixed assumptions.\n\n[ Vision Events](https://blog.roboflow.com/model-monitoring/) records every inspection, including the image, detections, and final safety status. Over time, this data can reveal incident trends by camera location, shift, or recurring risk zones.\n\nThe same Workflow can run through [ Roboflow's managed API or self-hosted with Roboflow Inference](https://docs.roboflow.com/deployment?ref=blog.roboflow.com), allowing deployment across multiple cameras while keeping the same safe/unsafe output format for monitoring dashboards and alert systems.\n\n## Conclusion\n\nBoth workflows use the same division of labor: RF-DETR finds the people and equipment, and Gemini interprets how they relate. The difference is scope. The scene-level workflow sends one annotated image to Gemini 2.5 Pro and returns a written interaction summary, which suits open-ended monitoring and review. The detect-crop-classify workflow isolates each person, asks a narrower question, and returns a structured safe or unsafe report, which suits a specific safety rule that needs a machine-readable answer.\n\nUse the first pattern to find out what is happening in a scene, and the second when you already know which interaction matters and need a consistent decision about it. Both analyze single images without tracking or deterministic proximity logic, so treat their output as an inspection aid and review flagged results before operational use. A good next step is running both on the same test images from your own warehouse, which shows quickly which pattern fits your monitoring needs.\n\n**Further reading**\n\n**Cite this Post**\n\nUse the following entry to cite this post in your research:\n\n[Mostafa Ibrahim](/author/mostafa/). (Aug 3, 2026).\nHuman-Object Interaction Detection with RF-DETR. Roboflow Blog: https://blog.roboflow.com/human-object-interaction-detection-with-rfdetr/", "url": "https://wpnews.pro/news/human-object-interaction-detection-with-rf-detr", "canonical_source": "https://blog.roboflow.com/human-object-interaction-detection-with-rfdetr/", "published_at": "2026-08-03 17:41:00+00:00", "updated_at": "2026-08-09 13:11:25.519341+00:00", "lang": "en", "topics": ["computer-vision", "artificial-intelligence", "generative-ai"], "entities": ["Roboflow", "RF-DETR", "Gemini 2.5 Pro", "HICO-DET", "V-COCO"], "alternates": {"html": "https://wpnews.pro/news/human-object-interaction-detection-with-rf-detr", "markdown": "https://wpnews.pro/news/human-object-interaction-detection-with-rf-detr.md", "text": "https://wpnews.pro/news/human-object-interaction-detection-with-rf-detr.txt", "jsonld": "https://wpnews.pro/news/human-object-interaction-detection-with-rf-detr.jsonld"}}