{"slug": "video-analytics-with-vision-ai", "title": "Video Analytics with Vision AI", "summary": "Roboflow released a tutorial demonstrating video analytics with vision AI, combining RF-DETR object detection, ByteTrack multi-object tracking, and Gemini 2.5 Pro multimodal inspection to detect and classify damaged boxes on a conveyor belt in real time. The system automates package inspection for manufacturing and logistics, replacing manual monitoring with AI-driven analysis.", "body_md": "Video analytics with vision AI uses computer vision models to detect, track, and interpret objects and activity inside a video stream, in real time or on recorded footage, without a person watching every frame. Instead of a camera that only records, the system understands what is moving through the scene: what each object is, where it goes, and whether it meets the standard you set.\n\nThis tutorial builds a working example end to end. We train an [RF-DETR](https://rfdetr.roboflow.com/latest/?ref=blog.roboflow.com) model to detect boxes on a conveyor belt, deploy it in [Roboflow Workflows](https://roboflow.com/workflows/build?ref=blog.roboflow.com), use ByteTrack to assign a unique ID to each box as it moves, and use [Gemini 2.5 Pro](https://playground.roboflow.com/models/google/gemini-2-5-pro?ref=blog.roboflow.com) to inspect each tracked box for visible damage. The result is an annotated video showing every detected box, its tracker ID, and its inspection label. Before the build, here is how video analytics with vision AI actually works and where it is used.\n\n## What Is Video Analytics with Vision AI?\n\nTraditional video analytics relied on hand-coded rules: motion in a zone, a line crossing, a pixel threshold. Vision AI replaces those brittle rules with trained models that recognize objects and conditions directly, so the same camera feed can answer far more specific questions, \"Is this package damaged?,\" not just \"Did something move?\"\n\nMost production systems combine three stages, and today's tutorial uses all three:\n\n**Object detection** locates every object of interest in each frame and draws a bounding box around it. Here, an RF-DETR model detects cardboard boxes on the conveyor. Detection is what makes the rest of the pipeline possible, because you cannot track or inspect what you have not localized.\n\n**Multi-object tracking** assigns a persistent ID to each object and follows it across frames, so one box stays one box as it travels through the scene. We use ByteTrack for this. Tracking is what turns a stack of independent frame predictions into a coherent view of objects moving over time, and it prevents the same package from being counted or inspected twice.\n\n**Multimodal inspection** applies a vision-capable model to each tracked object to make a judgment a plain detector cannot. We send each fully visible box crop to Gemini 2.5 Pro, which classifies it as Defective, Not Defective, or Opened. This is the layer that reads context, condition, and intent rather than just object class.\n\n### Where Video Analytics with Vision AI Is Used\n\nThe same detect, track, inspect pattern powers a wide range of real-time video analytics beyond the conveyor line:\n\n**Manufacturing and logistics:** package inspection,[on-shelf availability](https://blog.roboflow.com/automate-on-shelf-monitoring/), throughput and dwell-time analysis.**Workplace safety:** PPE compliance, restricted-zone monitoring, and unsafe-interaction alerts.**Retail:** shelf monitoring, queue length, and stock-out detection.**Sports:** player and ball tracking for[performance analytics](https://blog.roboflow.com/automate-tennis-analytics/).**Traffic and cities:** vehicle counting, incident detection, and flow analysis.\n\nIn every case the architecture is the same one we build below: detect the objects, track them across frames, and inspect each one. The conveyor box inspection system is a concrete, buildable version of that pattern.\n\n## Video Analytics with Vision AI Demo\n\nNow let's automate conveyor belt monitoring by combining object detection, multi-object tracking, and multimodal AI inspection into a single video analytics workflow.\n\nConveyor systems are widely used, and as boxes move along the conveyor, ensuring that each package is correctly handled and free from visible damage is essential for maintaining product quality and preventing downstream issues. For many manufacturers, the Cost of Poor Quality is [ estimated](https://www.autodesk.com/blogs/design-and-manufacturing/understanding-the-cost-of-poor-quality-copq-in-manufacturing/?ref=blog.roboflow.com) to consume 15–20% of revenue, underscoring the importance of reliable quality inspection.\n\nComputer vision enables manufacturers to automatically detect packages, track their movement, and perform visual quality inspection without stopping the conveyor.\n\nIn this tutorial, we will build an automated conveyor box inspection system using Roboflow. We will train an [ RF-DETR](https://rfdetr.roboflow.com/latest/?ref=blog.roboflow.com) model to detect boxes on a conveyor belt, deploy the model in\n\n[, use ByteTrack to assign a unique ID to each box as it moves through the scene, and leverage](https://roboflow.com/workflows/build?ref=blog.roboflow.com)\n\n__Roboflow workflows__[to inspect each tracked box for visible damage and packaging defects.](https://playground.roboflow.com/models/google/gemini-2-5-pro?ref=blog.roboflow.com)\n\n__Gemini 2.5 Pro__The system can help identify:\n\n- Boxes moving along the conveyor\n- Unique IDs for each tracked box\n- Crushed, torn, or damaged packaging\n- Boxes that require manual inspection\n\nThe final output is an annotated video showing each detected box, its tracker ID, and its Gemini-generated inspection label. [Here's the workflow we'll build.](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiOXk5cHNNN3Q3QzZIa0FtWkltb3YiLCJ3b3Jrc3BhY2VJZCI6Im5JRk5DOGRjbU5OOXZ4d29ybWpoWTdCNjdQZTIiLCJ1c2VySWQiOiJuSUZOQzhkY21OTjl2eHdvcm1qaFk3QjY3UGUyIiwiaWF0IjoxNzgzMjg2Mzc1fQ.a_oHRX3tvfZOexcFXA2R9CJTqgVCJcSI9LUCURXBjnc?ref=blog.roboflow.com)\n\n### Step 1: Prepare the Dataset\n\nFor this tutorial, we use a public conveyor box dataset from [ Roboflow Universe](https://universe.roboflow.com/?ref=blog.roboflow.com). Our goal is to train a model that detects boxes moving along a conveyor belt, providing the foundation for real-time tracking and AI-powered inspection. Rather than training the model to recognize multiple object categories, we focus on a single object of interest: cardboard boxes.\n\nThe original dataset contains several annotation classes, including multiple box-related labels and additional objects that are not required for our application. Since our workflow only requires detecting cardboard boxes, we fork the dataset into our Roboflow workspace and remove images containing annotations unrelated to our target object. The resulting dataset contains a single detection class, allowing the model to focus entirely on localizing boxes while downstream components handle tracking and inspection.\n\nThe resulting dataset contains 850 annotated images captured in an industrial conveyor environment. Boxes appear at different positions along the conveyor, under varying lighting conditions, and at different distances from the camera. This variety helps the model generalize to real-world production lines where package placement and environmental conditions naturally vary.\n\nHere are examples of the detection targets:\n\nTo begin, fork the dataset into your Roboflow workspace. Then navigate to the Train tab and select Custom Training. From the available architectures, choose RF-DETR and set the model size to Small.\n\nOnce the model architecture is selected, generate a new dataset version before starting training. Configure a 70/15/15 split for training, validation, and testing.\n\nEnable the following preprocessing steps:\n\n- Auto-orientation\n- Resize (512×512)\n\nThese preprocessing steps normalize the input images while providing a consistent input resolution for RF-DETR training.\n\n### Step 2: Train the RF-DETR Model\n\nOnce training begins, RF-DETR learns to localize cardboard boxes directly from the annotated examples. Unlike an image classification model that only determines whether a box is present, RF-DETR predicts a bounding box around every detected package, making it well-suited for real-time video analytics.\n\nThis localization capability forms the foundation of the remainder of our workflow. The detected bounding boxes are passed to ByteTrack, which assigns a persistent ID to every box as it moves through the conveyor.\n\nOnce training is complete, deploy the model in Roboflow Workflows, where it serves as the first stage of our video analytics pipeline. The detector identifies every box entering the scene, ByteTrack maintains a unique ID for each object throughout its journey on the conveyor, and Gemini 2.5 Pro performs AI-powered inspection on each tracked box.\n\n### Step 3: Deploy the Model to Workflows\n\nAfter training, deploy the RF-DETR model in Roboflow Workflows to build the conveyor inspection pipeline. The workflow detects boxes in each frame, assigns a tracker ID to every package, waits until the tracked bounding box is fully inside the frame, and sends the cropped box to Gemini 2.5 Pro for inspection.\n\nGemini classifies each package as:\n\n- Defective\n- Not Defective\n- Opened\n\nThe final output draws the box location, tracker ID, and inspection result onto each video frame.\n\nOpen the trained model and click Deploy Model. In the deployment window, select Customize With Logic. This opens the Workflow editor with the RF-DETR model already connected to the image input.\n\n### Step 4: Configure Box Tracking and Filtering\n\nAdd a ByteTrack Tracker block after the detection model. Connect the original frame to the image input and the model predictions to the detections input.\n\nByteTrack assigns a persistent ID to each package as it moves through the video. This allows the workflow to follow the same box across multiple frames and distinguish it from other packages on the conveyor.\n\nNext, add the custom Full Box Filter block. Connect the original frame to its image input and the ByteTrack tracked detections to its predictions input.\n\nBoxes often enter the frame partially. Inspecting them too early can produce an unreliable result because Gemini cannot see the full package. The filter only passes boxes whose bounding box is safely inside the frame and larger than 20 pixels in both dimensions.\n\nThe block also keeps a small margin between the box and the image border. Boxes touching the edge remain visible in the tracking output, but are not sent for inspection yet.\n\nThe code for the block:\n\n``` python\ndef run(self, image, predictions):\n    if hasattr(image, 'numpy_image'):\n        img = image.numpy_image\n    else:\n        img = np.array(image)\n    h, w = img.shape[:2]\n    margin = max(12, int(0.02 * min(h, w)))\n\n    xyxy = getattr(predictions, 'xyxy', [])\n    mask = []\n    for box in xyxy:\n        x1, y1, x2, y2 = [float(v) for v in box]\n        is_full = (\n            x1 > margin and\n            y1 > margin and\n            x2 < (w - margin) and\n            y2 < (h - margin) and\n            (x2 - x1) > 20 and\n            (y2 - y1) > 20\n        )\n        mask.append(is_full)\n\n    \n    full = predictions[np.array(mask, dtype=bool)]\n\n    return {'full_detections': full}\n```\n\nAdd a Dynamic Crop block after the filter. Use the original frame as the image input and full_box_filter.full_detections as the predictions input. The block uses each accepted bounding box to crop the corresponding package from the frame.\n\n### Step 5: Configure Gemini Inspection\n\nAdd a Google Gemini block after Dynamic Crop, it receives an individual box crop instead of the full conveyor frame. This keeps the inspection focused on the visible condition of one package.\n\nAdd a Detections Classes Replacement block next. It attaches the inspection result to the corresponding tracked detection while preserving its box coordinates and tracker ID.\n\n### Step 6: Draw and Test the Results\n\nAdd a Bounding Box Visualization block using all ByteTrack detections. This draws boxes around every tracked package, including boxes that have not yet passed the full-box filter.\n\nAdd a Label Visualization block after it and configure the text as Tracker Id. Place the label in the top-left corner of each box.\n\nFinally, add the custom Defect Status Text Overlay block. It receives all tracked detections and the inspected detections.\n\nThe code used for this block:\n\n``` python\ndef run(self, image, all_predictions, inspection_predictions):\n    from inference.core.workflows.execution_engine.entities.base import WorkflowImageData\n\n    if not hasattr(self, \"status_by_tracker_id\"):\n        self.status_by_tracker_id = {}\n\n    original = image if hasattr(image, \"numpy_image\") else None\n    img = image.numpy_image.copy() if original else np.array(image).copy()\n    img = np.clip(img, 0, 255).astype(np.uint8)\n\n    def get_tracker_ids(detections):\n        ids = getattr(detections, \"tracker_id\", None)\n\n        if ids is None:\n            data = getattr(detections, \"data\", {}) or {}\n            ids = data.get(\"tracker_id\", data.get(\"track_id\", []))\n\n        result = []\n        for tracker_id in ids:\n            try:\n                tracker_id = int(tracker_id)\n                result.append(tracker_id if tracker_id >= 0 else None)\n            except (TypeError, ValueError):\n                result.append(None)\n\n        return result\n\n    def get_labels(detections):\n        data = getattr(detections, \"data\", {}) or {}\n        labels = data.get(\"class_name\", [])\n\n        if len(labels) == 0:\n            labels = getattr(detections, \"class_name\", [])\n\n        return list(labels)\n\n    valid_labels = {\n        \"defective\": \"Defective\",\n        \"not defective\": \"Not Defective\",\n        \"opened\": \"Opened\",\n    }\n\n    inspected_ids = get_tracker_ids(inspection_predictions)\n    inspected_labels = get_labels(inspection_predictions)\n\n    for tracker_id, raw_label in zip(inspected_ids, inspected_labels):\n        if tracker_id is None or tracker_id in self.status_by_tracker_id:\n            continue\n\n        label = valid_labels.get(\n            str(raw_label).strip().lower().replace(\"_\", \" \")\n        )\n\n        if label:\n            self.status_by_tracker_id[tracker_id] = label\n    colors = {\n        \"Defective\": (0, 0, 255),\n        \"Not Defective\": (0, 180, 0),\n        \"Opened\": (0, 165, 255),\n    }\n\n    boxes = getattr(all_predictions, \"xyxy\", [])\n    tracker_ids = get_tracker_ids(all_predictions)\n    height, width = img.shape[:2]\n\n    for box, tracker_id in zip(boxes, tracker_ids):\n        label = self.status_by_tracker_id.get(tracker_id)\n        if not label:\n            continue\n        x1, _, _, y2 = [int(round(float(value))) for value in box]\n        x = max(0, min(width - 1, x1))\n        y = max(20, min(height - 1, y2 - 8))\n\n        font = cv2.FONT_HERSHEY_SIMPLEX\n        scale = 0.8\n        thickness = 2\n        color = colors[label]\n\n        (text_width, text_height), baseline = cv2.getTextSize(\n            label, font, scale, thickness\n        )\n        cv2.rectangle(\n            img,\n            (x, max(0, y - text_height - baseline - 8)),\n            (\n                min(width - 1, x + text_width + 10),\n                min(height - 1, y + baseline + 4),\n            ),\n            color,\n            -1,\n        )\n\n        cv2.putText(\n            img,\n            label,\n            (x + 5, y - 4),\n            font,\n            scale,\n            (255, 255, 255),\n            thickness,\n            cv2.LINE_AA,\n        )\n\n    output_image = (\n        WorkflowImageData.copy_and_replace(\n            origin_image_data=original,\n            numpy_image=img,\n        )\n        if original is not None\n        else img\n    )\n\n    return {\"image\": output_image}\n```\n\n### Step 7: Test the Workflow\n\nClick Run and upload a conveyor video. Check that:\n\n- Every visible package receives a tracker ID.\n- Partially visible boxes are not inspected.\n- The inspection label appears after the full box enters the frame.\n- Different boxes keep separate IDs and results.\n\n## Use Roboflow Agent for Video Analytics with Vision AI\n\nIf you'd rather not add each block by hand, use [Roboflow Agent](https://app.roboflow.com/solutions/chat/new?ref=blog.roboflow.com). Instead of configuring blocks one at a time, you describe the pipeline you want in plain text and the Agent builds it for you. Here's an example:\n\n## Real-Time Video Analytics with Vision AI Conclusion\n\nIn this tutorial, we built an automated conveyor box inspection workflow using Roboflow. We trained an RF-DETR model to detect packages, used ByteTrack to follow each box across video frames, and added Gemini 2.5 Pro to classify boxes as Defective, Not Defective, or Opened.\n\nThe workflow waits until a package is fully visible before inspection. It can be extended with fixed conveyor cameras, alert systems, or reporting tools to support continuous package monitoring.\n\n**Further reading**\n\n[How to Train RF-DETR on a Custom Dataset](https://blog.roboflow.com/train-rf-detr-on-a-custom-dataset/)[What is ByteTrack? A Deep Dive](https://blog.roboflow.com/what-is-bytetrack-computer-vision/)[Video Object Counting: A Step-by-Step Guide](https://blog.roboflow.com/video-object-counting/)\n\n## Video Analytics with Vision AI FAQ\n\n### How does AI video analytics work?\n\nMost systems run three stages. An object detection model localizes each object in every frame, a tracker assigns a persistent ID and follows each object across frames, and an inspection or classification model evaluates each tracked object. In this tutorial that maps to RF-DETR for detection, ByteTrack for tracking, and Gemini 2.5 Pro for inspection, all connected in a single Roboflow Workflow.\n\n### What is the difference between video analytics and computer vision?\n\nComputer vision is the broader field of getting models to understand images. Video analytics is the application of computer vision to video streams specifically, adding the time dimension, so tracking objects across frames and analyzing behavior over time matter as much as recognizing them in a single image.\n\n### Can video analytics with vision AI run in real time?\n\nYes. With an efficient detector like RF-DETR and a lightweight tracker like ByteTrack, the detection and tracking stages run frame by frame at production speed. Heavier multimodal inspection is applied selectively, in this workflow only once a box is fully inside the frame, which keeps the pipeline fast while still inspecting every package.\n\n### Do I need to write code to build a video analytics pipeline?\n\nNo. You can build the entire detect, track, inspect pipeline visually in [Roboflow Workflows](https://roboflow.com/workflows/build?ref=blog.roboflow.com) by connecting blocks, or describe it in plain language to Roboflow Agent and have it assembled for you. Custom logic blocks are available when you need them, but they are optional.\n\n**Cite this Post**\n\nUse the following entry to cite this post in your research:\n\n[Mostafa Ibrahim](/author/mostafa/). (Jun 29, 2026).\nVideo Analytics with Vision AI. Roboflow Blog: https://blog.roboflow.com/video-analytics-with-vision-ai/", "url": "https://wpnews.pro/news/video-analytics-with-vision-ai", "canonical_source": "https://blog.roboflow.com/video-analytics-with-vision-ai/", "published_at": "2026-06-29 19:27:00+00:00", "updated_at": "2026-07-07 01:24:57.532965+00:00", "lang": "en", "topics": ["computer-vision", "artificial-intelligence", "machine-learning", "ai-products", "ai-tools"], "entities": ["Roboflow", "RF-DETR", "ByteTrack", "Gemini 2.5 Pro", "Google"], "alternates": {"html": "https://wpnews.pro/news/video-analytics-with-vision-ai", "markdown": "https://wpnews.pro/news/video-analytics-with-vision-ai.md", "text": "https://wpnews.pro/news/video-analytics-with-vision-ai.txt", "jsonld": "https://wpnews.pro/news/video-analytics-with-vision-ai.jsonld"}}