{"slug": "build-agentic-computer-vision-with-roboflow-workflows", "title": "Build Agentic Computer Vision with Roboflow Workflows", "summary": "Roboflow published a guide describing agentic computer vision, an architecture in which a vision model's output feeds a reasoning step that decides and executes an action instead of stopping at a bounding box or label. The guide outlines a four-step loop — perceive, reason, act, verify — combining detectors such as RF-DETR for perception, a vision language model for judgment, and tool calls for action, and cites ReAct, Toolformer, and Reflexion as research foundations for interleaving reasoning and actions.", "body_md": "*Agentic computer vision is a system where a vision model's output feeds a reasoning step that decides what to do next and takes an action, instead of stopping at a box or a label. In practice, that means a detector such as RF-DETR for perception, a vision language model for judgment, and tool calls or integrations for action, chained in a loop that can check its own result, using Roboflow Workflows.*\n\nComputer vision has traditionally been built around prediction. An image enters a model and the system returns an object class, bounding box, segmentation mask, keypoint, count, or another structured result. That model output is useful, but it does not necessarily answer the operational question. For example:\n\n- A warehouse camera may detect a forklift\n- A manufacturing camera may identify a damaged carton\n- A security camera may find a person in a restricted area\n\nA conventional system then needs application code around the model to decide whether the observation matters and what should happen next. Agentic computer vision extends this architecture.\n\nInstead of treating perception as the endpoint, it treats perception as the first step in a system that can interpret what it sees, use additional context, select an allowed action, observe what happened, and continue from there. The result is not simply a computer vision model. It is a closed-loop computer vision application.\n\n## What Is Agentic Computer Vision?\n\nAgentic computer vision is a computer vision architecture in which visual perception feeds a reasoning layer that can use context and tools to decide what action to take, execute that action, and verify the result.\n\nThe idea closely follows the broader development of AI agents. Research such as [ReAct](https://research.google/blog/react-synergizing-reasoning-and-acting-in-language-models/?ref=blog.roboflow.com) showed how reasoning and actions can be interleaved rather than handled as isolated stages, while [Toolformer](https://arxiv.org/abs/2302.04761?ref=blog.roboflow.com) demonstrated the value of language models invoking external tools through APIs. [Reflexion](https://doi.org/10.52202/075280-0377?ref=blog.roboflow.com) explored another important part of the pattern, using feedback from an environment to improve the next decision.\n\nFor vision applications, the environment is physical rather than purely textual. [Cameras](https://ai1.roboflow.com/?ref=blog.roboflow.com) provide observations, [vision models](https://playground.roboflow.com/models?ref=blog.roboflow.com) convert those observations into structured evidence, multimodal models interpret situations, and integrations connect the resulting decision to software or industrial systems.\n\nA useful way to think about agentic computer vision is as a four-step loop:\n\n1. **Perceive:** A detector, segmenter, classifier, OCR model, tracker, or other vision model converts pixels into structured observations such as`person detected` ,`forklift at x=420` ,`defect present` , or`three pallets counted` .\n2. **Reason:** A vision language model or other reasoning model combines the visual evidence with context such as time, location, policy, previous observations, production state, or external data and determines what the situation means.\n3. **Act:** The system invokes an approved tool for example, send a Slack notification, call a webhook, create a maintenance ticket, write an event to a database, update an MES, send a value to an OPC UA server, or request human review.\n4. **Verify:** The system checks whether the action succeeded or whether the visual situation changed as expected. If not, it can retry, choose another allowed path, or escalate.\n\nThe last step is important because an agent is different from a one-shot multimodal prompt.\n\nConsider a camera monitoring a pedestrian area. A traditional object detector might return:\n\n`forklift - confidence: 0.97`\n\nThe output is accurate but incomplete from an operational perspective. An agentic system could instead establish that the forklift overlaps a pedestrian-only zone, retrieve the current shift or area policy, determine whether the condition requires escalation, send a message containing the relevant image or video clip to the supervisor responsible for that shift, record the incident, and later verify whether the forklift left the restricted area.\n\nThat is the difference between *seeing an object* and *using visual evidence to complete a task*.\n\n## The Four Components of an Agentic Vision System\n\nThe perceive, reason, act, and verify loop can be implemented as four practical system components.\n\n### 1. Perception: turn pixels into structured facts\n\nThe first component is the perception layer. For most production systems, this should still be a specialized computer vision model such as an object detector, instance segmentation model, classifier, keypoint model, or OCR model.\n\nFor object detection, a model such as [RF-DETR](https://rfdetr.roboflow.com/latest/?ref=blog.roboflow.com) can turn an image into structured predictions containing classes, confidence scores, bounding boxes, and coordinates. For example:\n\n```\n{\n  \"class\": \"forklift\",\n  \"confidence\": 0.97,\n  \"x\": 614,\n  \"y\": 355,\n  \"width\": 281,\n  \"height\": 214\n}\n```\n\nThe important architectural decision is that the reasoning model does not necessarily need to inspect every full-resolution frame. A specialist model can first answer narrow questions extremely efficiently:\n\n- Is a person present?\n- Is a trailer present?\n- Is there a defect?\n- Did an object enter a polygon?\n- How many boxes crossed a line?\n- Has an object remained in a region for more than 30 seconds?\n\nOnly observations that satisfy those conditions need to move to the more expensive reasoning layer. This architecture is especially useful for video. A 30 FPS stream produces 30 opportunities for inference every second. Asking a VLM to independently reason about every frame is unnecessary for many applications. Instead:\n\nThe perception layer acts as a visual filter. RF-DETR was designed around the accuracy-latency trade-off required by this kind of workload. The RF-DETR provides a family of lightweight specialist detection transformers that can be fine-tuned to generate specialist detectors. Hence it provide better deployment trade-offs than relying on heavyweight vision-language models for every domain-specific detection problem.\n\nThere is another reason to separate perception from reasoning. Current VLMs are powerful but should not automatically be treated as perfect localization engines. Research continues to identify visual hallucination and localization failure modes in multimodal models.\n\n**HallusionBench**, published at CVPR, demonstrated that visual reasoning models can produce answers influenced by language priors rather than the evidence in an image. More recent CVPR work such as **ORIC** similarly finds that unusual object-context combinations can degrade object recognition in large vision-language models.\n\nThat's why a strong production pattern is to use a specialist model for evidence and then a VLM for ambiguous interpretation, rather than using a VLM for every visual operation. The detector provides the agent's eyes. The reasoning model decides what the evidence means.\n\n### 2. Reasoning: interpret the scene in context\n\nPerception tells the application what is visible. Reasoning determines why it matters. A vision language model can receive some combination of the image, cropped regions, detector output, temporal state, textual policies, sensor values, or data retrieved from external applications. Models from families such as [Gemini](https://blog.roboflow.com/gemini-computer-vision/), [Qwen-VL](https://playground.roboflow.com/models/qwen/qwen-vl?ref=blog.roboflow.com), and GPT can perform tasks that are difficult to represent using fixed detection classes alone. For example, a detector might identify:\n\n```\nperson\nforklift\npallet\ndoor\n```\n\nA reasoning prompt can ask:\n\n```\nA forklift and a person have been detected inside the loading area.\n\nUsing the supplied crop and zone information, determine whether:\n1. normal loading is occurring,\n2. the pedestrian is safely separated from the forklift,\n3. the scene is ambiguous and requires human review.\n\nReturn only structured JSON.\n```\n\nThe VLM is not replacing the detector in this architecture. It is receiving a much smaller and better-defined reasoning problem. That distinction matters.\n\nA model that must first search an entire frame for every relevant object, remember which objects matter, determine their relationships, interpret a policy, and produce a decision has more opportunities to fail than a model receiving already-localized evidence and a constrained task.\n\nA production architecture should therefore not assume that a persuasive natural-language answer is necessarily a correct visual judgment. Evaluate the reasoning model on your own task.\n\n[Vision Playground](https://playground.roboflow.com/?ref=blog.roboflow.com) and [Vision Evals](https://playground.roboflow.com/evals?ref=blog.roboflow.com) are useful starting points because models can be compared across standardized visual tasks rather than selected solely from general language-model benchmarks. The current evaluation covers object detection, counting, identification, OCR, data extraction, and reasoning using the same samples and ground truth.\n\nThe production choice may not be the model with the highest overall benchmark score. A [warehouse](https://roboflow.com/industries/warehousing?ref=blog.roboflow.com) application may prioritize reasoning accuracy and cost. A document workflow may prioritize OCR and structured extraction. A near-real-time monitoring application may value response speed more heavily. The reasoning layer should therefore be benchmarked against your actual visual decisions, not just a generic leaderboard.\n\n### 3. Action: give the system controlled tools\n\nA reasoning model becomes operationally useful when its decision can cause something to happen. This is the action layer. An action can be entirely digital such as:\n\n- `send_slack_alert()`\n- `create_ticket()`\n- `post_webhook()`\n- `write_database_record()`\n- `send_email()`\n- `request_human_review()`\n\nOr it can interact with an industrial system:\n\n- `write_opcua_tag()`\n- `write_plc_value()`\n- `publish_mqtt_event()`\n\nRoboflow Workflows supports integration patterns including webhooks, [Slack notifications](https://blog.roboflow.com/slack-notification-workflows/), email, SQL Server, [MQTT](https://roboflow.com/build-a-workflow/roboflow-object-detection-model-to-mqtt-publisher?ref=blog.roboflow.com), OPC UA, and [PLC](https://blog.roboflow.com/computer-vision-plc-integration/)-oriented blocks. The precise integrations available depend on deployment and plan.\n\nFor example, the output of the reasoning layer could become:\n\n```\n{\n  \"status\": \"blocked\",\n  \"severity\": \"medium\",\n  \"reason\": \"pallet obstructing loading path\",\n  \"action\": \"create_ticket\"\n}\n```\n\nThe system can then map `create_ticket` to a specific approved integration. This is preferable to giving a model unrestricted control over arbitrary APIs. The agent should choose from a small set of actions that the application explicitly exposes. A good action contract might allow:\n\n```\nNO_ACTION\nNOTIFY_SUPERVISOR\nCREATE_MAINTENANCE_TICKET\nREQUEST_HUMAN_REVIEW\n```\n\nbut not arbitrary commands. This principle becomes even more important when computer vision interacts with physical equipment.\n\nRoboflow supports industrial integrations such as an [OPC UA Writer Sink](https://blog.roboflow.com/integrate-roboflow-with-ignition/), PLC Writer, and Modbus-oriented workflow integrations. An OPC UA workflow can, for example, convert a vision result into a Boolean, count, status, or defect code and publish that value to an OPC UA server used by PLCs, SCADA applications, or dashboards.\n\nThe reasoning model, however, should not become the machine's safety controller. For [industrial systems](https://roboflow.com/industries/industrial-manufacturing?ref=blog.roboflow.com), an agent can recommend or publish a bounded state such as:\n\n```\ninspection_result = REVIEW\n```\n\nwhile deterministic PLC logic retains responsibility for machine timing, interlocks, watchdogs, safety conditions, and physical actuation. Agentic reasoning should expand what the vision system understands, not remove the safeguards already built into the control system.\n\n### 4. Memory and verification: know what happened before and what happened next\n\nA single frame has no memory, a real operational process does. Suppose a camera sees a trailer at a dock door. One frame cannot tell you whether the trailer arrived one second ago or has remained idle for 45 minutes. Similarly, detecting a person inside a region does not tell you whether that person just entered, has been there continuously, or has crossed the same area repeatedly. This is where tracking and state become part of the agent.\n\nRoboflow Workflows includes tracking and temporal-processing components that can maintain object identity and compute events across frames. The available block ecosystem includes [Byte Track](https://trackers.roboflow.com/latest/trackers/bytetrack/?ref=blog.roboflow.com), [BoT-SORT](https://trackers.roboflow.com/latest/trackers/botsort/?ref=blog.roboflow.com), [Time in Zone](https://docs.roboflow.com/workflows/blocks/blocks/video-processing/timein-zone?ref=blog.roboflow.com), counters, caches, filters, and other stateful processing blocks. A tracked observation might evolve like this:\n\n```\nFrame 120:\ntruck_17 enters dock_zone_4\n\nFrame 1,920:\ntruck_17 still present\ntime_in_zone = 60 seconds\n\nFrame 10,920:\ntruck_17 still present\ntime_in_zone = 360 seconds\n```\n\nOnly after the temporal condition is satisfied does the reasoning layer need to run. Memory can also include non-visual state:\n\n```\n{\n  \"camera\": \"dock_04\",\n  \"track_id\": 17,\n  \"first_seen\": \"14:03:11\",\n  \"time_in_zone_seconds\": 367,\n  \"previous_state\": \"loading\",\n  \"previous_action\": \"none\",\n  \"shift\": \"afternoon\"\n}\n```\n\nThe second part of this component is verification. After the agent acts, ask a simple question:\n\n**Did the intended result actually occur?**\n\nVerification might be digital. A ticket API returns a ticket ID. Slack returns a successful response. An OPC UA write reports success. A database confirms that the event was inserted. Or verification can be visual.\n\nThe system sends an alert because an access route is obstructed. Thirty seconds later, it inspects the next observation. If the obstruction remains, it escalates. If it disappears, it closes the event. That closes the loop:\n\nResearch on agent architectures has repeatedly shown the value of feedback rather than assuming the first generated action is final. Reflexion, for example, formalizes the use of environmental feedback and memory to guide later decisions.\n\nFor production vision systems, the implementation can be much simpler than a research agent. Verification may be nothing more than checking an API status code, querying an acknowledgement flag, or looking at the next relevant camera frame.\n\n## Why Agentic Computer Vision Is Possible Now\n\nAgentic computer vision combines these components (i.e. object detection, tracking, workflow automation, multimodal reasoning, and APIs) into one deployable system.\n\n### Vision language models can reason about visual context\n\nModern VLMs can answer open-ended questions about images, extract structured information, compare visual evidence, interpret text inside scenes, and reason about relationships that would otherwise require many narrowly trained classes.\n\nRoboflow's current Vision Evals compares dozens of models across six grounded visual tasks and records accuracy, token usage, estimated cost, and speed. This makes model selection increasingly measurable rather than anecdotal.\n\nAt the same time, research on hallucinations makes an important limitation clear i.e. VLM output must still be evaluated and constrained. The existence of strong reasoning does not eliminate the need for reliable perception, structured output, and verification.\n\nThis is exactly why the hybrid architecture works well. Use a specialist model to establish visual facts, then let the VLM reason about the part that is difficult to hard-code.\n\n### Detectors are fast enough to gate VLM reasoning\n\nThe second enabling technology is fast local perception. A specialist detector can run continuously while the slower reasoning model runs only when needed.\n\nRF-DETR demonstrated single-image detection latency in the low-millisecond range on the specified T4/TensorRT configuration, allowing perception to operate at a fundamentally different cadence from VLM reasoning.\n\nThis creates an architecture such as:\n\nThis gating step is also what makes the economics more practical. VLM APIs are generally priced per input and output token or equivalent model usage. If a camera generates 108,000 frames per hour at 30 FPS, sending every frame to a remote reasoning model is usually unnecessary. If the detector and temporal logic reduce that stream to a handful of meaningful events, the VLM is paying attention only when semantic reasoning is useful. The detector therefore acts not only as a perception layer, but also as a cost and latency gate.\n\n### Workflows can orchestrate the system without custom glue code\n\nThe third change is orchestration. Building this system required developers to separately implement video capture, model serving, tracking, event logic, VLM APIs, JSON parsing, retry behavior, webhooks, notification services, and deployment.\n\nRoboflow Workflows provides a visual environment for chaining these operations into a multi-step computer vision application. A Workflow can then be deployed in the cloud or on compatible local hardware using [Roboflow Inference](https://inference.roboflow.com/?ref=blog.roboflow.com) and can process images, videos, and live streams.\n\nThe available Workflow block ecosystem includes object detection, tracking, conditions, dynamic crops, VLMs, JSON parsing, Slack and email notification, webhooks, databases, MQTT, and industrial integrations. The architecture can therefore be represented directly:\n\nThis makes agentic vision useful for applications where the visual observation itself is straightforward but interpreting its operational significance is not. Examples include safety-event triage, [logistics](https://roboflow.com/industries/logistics?ref=blog.roboflow.com) exception monitoring, maintenance assessment, [retail shelf exceptions](https://blog.roboflow.com/retail-store-object-detection/), visual inspection escalation, site monitoring, asset condition reporting, document workflows, and situations where an operator currently has to look at an image before deciding what business process should happen next.\n\n## Building An Agentic Computer Vision System in Roboflow Workflows\n\n[Roboflow Workflows](https://roboflow.com/workflows/build?ref=blog.roboflow.com) provides the orchestration layer needed to connect perception, reasoning, structured outputs, control flow, actions, and deployment into one computer vision application.\n\nThe [Roboflow Vision Agents](https://blog.roboflow.com/vision-agents/) tutorial demonstrates the general vision-agent pattern using a specialist perception model, conditional gating, Gemini-based reasoning, JSON parsing, and an automated notification.\n\nTo demonstrate this architecture, we will build a [dock-door monitoring system](https://roboflow.com/ai/dock-safety?ref=blog.roboflow.com) that watches a video feed, detects trucks and people, tracks how long each truck remains near the loading bay, and triggers an AI assessment when a truck exceeds a configurable dwell-time threshold. The completed system uses two Roboflow Workflows:\n\n1. [Dock Door Monitor](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiNjZqY0VjSjRVaEl0dkJ1eDV3engiLCJ3b3Jrc3BhY2VJZCI6InZjQmw1Y0x3bUtQallLTGNRemV1VkE4UlRhNjIiLCJ1c2VySWQiOiJ2Y0JsNWNMd21LUGpZS0xjUXpldVZBOFJUYTYyIiwiaWF0IjoxNzg5ODA1MDY5fQ.PNMpnUu4GRlDiyJby8aHLE2uiAOoijHwFIanj9ERHU0?ref=blog.roboflow.com) processes the video, detects and tracks trucks, measures dwell time, and emits a single trigger frame.\n2. [Dock Door Frame Analyzer](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiY0g0QklpZklQSkhhZXpGaXpLdmYiLCJ3b3Jrc3BhY2VJZCI6InZjQmw1Y0x3bUtQallLTGNRemV1VkE4UlRhNjIiLCJ1c2VySWQiOiJ2Y0JsNWNMd21LUGpZS0xjUXpldVZBOFJUYTYyIiwiaWF0IjoxNzg5ODA1MTExfQ.iDt5wUN-IgB23E2J9Q6ol6_bTVqQB-DnWqmicMdsDmQ?ref=blog.roboflow.com) sends that trigger frame to Gemini and returns a structured assessment of whether the dock is active, idle, or blocked.\n\nA small Python application connects the two Workflows:\n\n### Workflow 1: Detect, track, and decide when an event is worth investigating\n\nThe first Workflow, [Dock Door Monitor](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiNjZqY0VjSjRVaEl0dkJ1eDV3engiLCJ3b3Jrc3BhY2VJZCI6InZjQmw1Y0x3bUtQallLTGNRemV1VkE4UlRhNjIiLCJ1c2VySWQiOiJ2Y0JsNWNMd21LUGpZS0xjUXpldVZBOFJUYTYyIiwiaWF0IjoxNzg5ODA3MTUwfQ.nb_uKkjmKoArgSvO7TqLyFW896UK2EuknnahudytM88?ref=blog.roboflow.com), is responsible for the fast perception and memory parts of the agent. It starts with two inputs:\n\n- an `image` , which receives each video frame,\n- and a `dwell_seconds` parameter.\n\nThe second input makes the dwell threshold configurable at runtime, so the same published Workflow can be tested with a one-second threshold or deployed with a much longer production threshold without rebuilding the graph.\n\n#### RF-DETR Object Detection\n\nThe first processing block is an Object Detection Model using `rfdetr-small`. For this example, the model is configured with a confidence threshold of `0.4` and restricted to the classes:\n\n```\ntruck\nperson\n```\n\nRF-DETR therefore answers the first question in the system:\n\nWhat objects are visible in this frame?\n\nFiltering to trucks and people also prevents unrelated detections from being passed through the rest of the Workflow. The pipeline begins as:\n\n#### ByteTrack Tracker\n\nObject detection works frame by frame. It can detect a truck in two consecutive frames, but by itself it does not know that both detections represent the same truck. The ByteTrack Tracker solves this by assigning a persistent `tracker_id` to objects as they move through the video. This gives the Workflow memory across frames.\n\nThe example uses a minimum of two consecutive frames before establishing a track, a lost-track buffer of 30 frames, and detection thresholds of `0.4`. The buffer allows an object's identity to survive short detection gaps caused by blur or temporary occlusion. This persistent identity is what makes dwell-time measurement possible.\n\n#### Time in Zone\n\nNext, the tracked detections are passed to Time in Zone. A polygon is drawn around the dock-door region. When the center of a tracked object's bounding box enters that polygon, the block begins measuring how long it remains there. For the example camera, the polygon is:\n\n```\n[\n  [267, 173],\n  [48, 464],\n  [337, 552],\n  [506, 250]\n]\n```\n\nThese coordinates are camera-specific and should be redrawn for a different dock view. The block uses the bounding-box `CENTER` as the triggering anchor and resets the timer after an object leaves the zone. The agent can now answer a second question:\n\nHow long has this particular truck been at the dock?\n\nThe sequence has become:\n\n#### Detections Filter\n\nThe [Detections Filter](https://docs.roboflow.com/workflows/blocks/blocks/logic-and-branching/detections-filter?ref=blog.roboflow.com) determines when a truck becomes operationally interesting. It retains a detection only when:\n\n```\nclass_name == \"truck\"\nAND\ntime_in_zone >= dwell_seconds\n```\n\nThe runtime `dwell_seconds` value may arrive as a string, so the comparison uses a `ToNumber` operation to cast it to a float before comparing it with `time_in_zone`. This block therefore answers:\n\n**Has a truck remained at the door long enough to require attention?**\n\nInstead of asking Gemini to inspect every truck immediately, the deterministic pipeline waits until the dwell condition has actually been met.\n\n#### Property Definition: count qualifying trucks\n\nThe filtered detections are then passed to a [Property Definition](https://docs.roboflow.com/workflows/blocks/blocks/advanced-blocks/property-definition?ref=blog.roboflow.com) block using the `SequenceLength` operation. This produces:\n\n```\nqualifying_truck_count\n```\n\nFor example:\n\n```\n0 → no truck is currently over the dwell threshold\n1 → one truck is over the threshold\n2 → two trucks satisfy the condition\n```\n\n`SequenceLength` is important here because the task is to count detections, rather than extract one of their properties.\n\n#### Delta Filter and Continue If\n\nWithout another control step, the Workflow would continue triggering on every subsequent frame while the same truck remained over the threshold. That would result in repeated Gemini requests. A [Delta Filter](https://docs.roboflow.com/workflows/blocks/blocks/logic-and-branching/delta-filter?ref=blog.roboflow.com) watches `qualifying_truck_count` and allows the next branch to execute only when the value changes. A [Continue If](https://docs.roboflow.com/workflows/blocks/blocks/logic-and-branching/continue-if?ref=blog.roboflow.com) block then checks whether:\n\n```\nqualifying_truck_count > 0\n```\n\nTogether, these blocks convert a continuous condition into an event. For example:\n\nThis is an important part of the agentic architecture. The perception system does not simply report detections continuously; it identifies when a meaningful change has occurred and only then allows the reasoning or action stage to continue.\n\n#### Select the qualifying truck\n\nOnce the event branch is triggered, a [Detections Transformation](https://docs.roboflow.com/workflows/blocks/blocks/transformations/detections-transformation?ref=blog.roboflow.com) block selects the first truck that satisfies the dwell-time condition.\n\nThis selected detection identifies the specific truck that caused the event and is passed to the trigger-frame visualization. There is no separate truck crop in this version of the Workflow. Instead, the complete dock-door frame is preserved so that the later reasoning stage can see both the truck and its surrounding environment.\n\n#### Create the trigger frame\n\nA [Bounding Box Visualization](https://inference.roboflow.com/workflows/blocks/bounding_box_visualization?ref=blog.roboflow.com) block creates the event image called:\n\n```\ntrigger_frame\n```\n\nIt uses the annotated dwell-time frame as the image and highlights the selected qualifying truck. The full trigger frame is particularly useful for VLM reasoning because the operational cause of a delay may not be inside the truck's bounding box. For example, the surrounding scene may contain a person, forklift, pallet, equipment, closed access point, or another condition affecting dock activity. The event path is therefore:\n\nThis `trigger_frame` becomes the visual evidence sent to the second Workflow for semantic analysis.\n\n#### Continuous video visualization\n\nThe trigger frame is intentionally sparse: it is produced only when the event branch executes. The Workflow therefore has a separate visualization path that remains populated during normal video processing. A Bounding Box Visualization draws the objects returned by Time in Zone, and a [Label Visualization](https://docs.roboflow.com/workflows/blocks/blocks/visualize-predictions/label-visualization?ref=blog.roboflow.com) displays their `Time In Zone` values. Using the tracker ID as the color axis also makes it easier to visually follow the same object across frames. This produces the continuously available:\n\n```\noutput_image\n```\n\nThe two visualization paths therefore serve different purposes:\n\n`output_image` is the normal monitored video, while `trigger_frame` is generated only when a qualifying dwell event occurs.\n\n#### Workflow outputs\n\nThe updated Dock Door Monitor workflow exposes:\n\n- Continuous video with detections and dwell-time labels\n- ByteTrack tracked detections\n- Time in Zone detections\n- Trucks satisfying the dwell-time rule\n- Number of trucks currently satisfying the rule\n- Context-rich event frame highlighting the qualifying truck\n- Error status returned by the optional Slack action\n- Message produced by the optional Slack action\n\nThe distinction between continuous and event-driven outputs is important. `output_image` remains available on every processed frame, whereas `trigger_frame` is populated only when the Delta Filter and Continue If branch execute. For this reason, `output_image` should remain the primary video output. The sparse `trigger_frame` is used as evidence for the reasoning Workflow.\n\n#### Optional Slack action\n\nThe Workflow also contains a [Slack Notification](https://docs.roboflow.com/workflows/blocks/blocks/notifications/slack-notification?ref=blog.roboflow.com) block connected to the event path.  A notification could contain information such as:\n\n```\nDock-door alert:\n1 truck exceeded the configured 60-second dwell threshold.\nReview the trigger frame in the monitoring application.\n```\n\nThe message can use `qualifying_truck_count` and the configured `dwell_seconds` value, with a cooldown to prevent excessive notifications. When Slack integration is required in production, the block can be enabled after providing the appropriate Slack credentials and destination channel. The first Workflow now represents the **perceive → remember → decide → trigger action** part of the agent:\n\nThis design keeps continuous perception fast and deterministic. Gemini/ VLM is not called for every frame. The first Workflow detects and tracks activity, maintains temporal state, applies the dwell-time rule, and produces a single context-rich trigger frame only when an event becomes important enough to investigate.\n\n### Workflow 2: Let Gemini interpret the event\n\nThe second Workflow, [Dock Door Frame Analyzer](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiY0g0QklpZklQSkhhZXpGaXpLdmYiLCJ3b3Jrc3BhY2VJZCI6InZjQmw1Y0x3bUtQallLTGNRemV1VkE4UlRhNjIiLCJ1c2VySWQiOiJ2Y0JsNWNMd21LUGpZS0xjUXpldVZBOFJUYTYyIiwiaWF0IjoxNzg5ODM2NTgzfQ.c67TkN822sYo4B_iuW-ZF32BQSRDxNBWod8Rtkf7IWI?ref=blog.roboflow.com), is deliberately much smaller. It processes only the event image produced by the first Workflow.\n\nThis Workflow receives one `image` input and passes it to a [Google Gemini](https://docs.roboflow.com/workflows/blocks/blocks/run-a-model/google-gemini?ref=blog.roboflow.com) block configured for visual question answering using `gemini-2.5-flash`. The prompt asks Gemini to determine whether the trailer is:\n\n```\nloaded\nidle\nblocked\n```\n\nand return only:\n\n```\n{\n  \"status\": \"loaded|idle|blocked\",\n  \"blocked_by\": \"short description or none\",\n  \"reason\": \"one concise sentence\"\n}\n```\n\nThe instructions further define the meanings:\n\n- `loaded` means visible loading or unloading activity is occurring.\n- `idle` means the trailer is docked but there is no visible work or obstruction.\n- `blocked` means a visible person, vehicle, object, equipment, closed access point, or another condition appears to prevent work.\n\nThe explicit schema is important because the output is intended for software, not just human reading.\n\n#### JSON Parser\n\nGemini's text response is passed directly to a [JSON Parser](https://docs.roboflow.com/workflows/blocks/blocks/advanced-blocks/json-parser?ref=blog.roboflow.com). The parser extracts:\n\n```\nstatus\nblocked_by\nreason\n```\n\nand exposes them as:\n\n```\ndock_status\nblocked_by\nassessment_reason\n```\n\nThe Workflow also returns `gemini_raw_output` and `json_parse_error`. These are useful when validating the application because they make malformed or incomplete VLM responses visible instead of silently hiding them. The final result can therefore look like:\n\n```\n{\n  \"dock_status\": \"blocked\",\n  \"blocked_by\": \"person standing between trailer and dock\",\n  \"assessment_reason\": \"A worker appears to obstruct access to the loading area.\",\n  \"json_parse_error\": false\n}\n```\n\nThis illustrates the reasoning layer clearly.\n\n### Connecting the two Workflows\n\nA small Python application connects [Dock Door Monitor](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiNjZqY0VjSjRVaEl0dkJ1eDV3engiLCJ3b3Jrc3BhY2VJZCI6InZjQmw1Y0x3bUtQallLTGNRemV1VkE4UlRhNjIiLCJ1c2VySWQiOiJ2Y0JsNWNMd21LUGpZS0xjUXpldVZBOFJUYTYyIiwiaWF0IjoxNzg5OTEzMTExfQ.h2F5PYW2iA5OVwyej4WbVW7C7gvHc2b2TJD2UD5GlTA?ref=blog.roboflow.com) and [Dock Door Frame Analyzer](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoiY0g0QklpZklQSkhhZXpGaXpLdmYiLCJ3b3Jrc3BhY2VJZCI6InZjQmw1Y0x3bUtQallLTGNRemV1VkE4UlRhNjIiLCJ1c2VySWQiOiJ2Y0JsNWNMd21LUGpZS0xjUXpldVZBOFJUYTYyIiwiaWF0IjoxNzg5OTEzMTM1fQ.bEGYLTuX-IP_tuDsr8MkTf6jCeC-LAO9luro4uZv_xA?ref=blog.roboflow.com). The first Workflow runs continuously over the video using WebRTC. It performs RF-DETR detection, ByteTrack tracking, dwell-time measurement, and event gating. The application displays the continuous `output_image` stream and listens for the sparse `trigger_frame` output.\n\nWhen `trigger_frame` is produced, Python sends that single image to Dock Door Frame Analyzer. Gemini interprets the scene, the JSON Parser converts the result into structured fields such as `dock_status`, `blocked_by`, and `assessment_reason`, and the application prints the result, stores it as JSON, and overlays the latest assessment on the video.\n\nThe Gemini request runs in a background thread so the slower reasoning step does not block the continuous video-processing callback. This preserves the separation between fast, stateful perception and event-driven reasoning.\n\n[.](https://github.com/tim3in/cv-examples/blob/main/dock_monitor.py?ref=blog.roboflow.com)\n\n**dock_monitor.py**\nOutputs generated by the application\n\n**Live annotated video:** Displays the continuous `output_image` from Dock Door Monitor, with tracked detections, dwell-time information, and the latest Gemini assessment overlaid on the video.\n\n**Trigger frame:** Saves `trigger_frame_<frame_id>.jpg` when a truck first exceeds the configured dwell-time threshold; this is the same contextual frame sent to Dock Door Frame Analyzer.\n\n**Gemini assessment JSON:** Saves `gemini_assessment.json` containing the structured `dock_status`, `blocked_by`, `assessment_reason`, parse status, frame ID, and video timestamp.\n\n```\n{\n  \"source_frame_id\": 139,\n  \"video_time_seconds\": 5.75,\n  \"dock_status\": \"blocked\",\n  \"blocked_by\": \"closed dock door\",\n  \"assessment_reason\": \"The trailer is positioned at dock door 01, but the dock door is closed, preventing any loading or unloading activity.\",\n  \"json_parse_error\": false,\n  \"gemini_raw_output\": \"``` json\\n{\\n  \\\"status\\\": \\\"blocked\\\",\\n  \\\"blocked_by\\\": \\\"closed dock door\\\",\\n  \\\"reason\\\": \\\"The trailer is positioned at dock door 01, but the dock door is closed, preventing any loading or unloading activity.\\\"\\n}\\n```\"\n}\n```\n\n**Annotated output video:** Saves `door_annotated.mp4`, containing the Workflow visualization together with the Gemini assessment displayed after analysis completes.\n\n**Startup console output:** Prints the input video, Dock Door Monitor Workflow ID, Dock Door Frame Analyzer Workflow ID, dwell threshold, and output directory when the program starts.\n\n**Trigger console output:** Reports the frame number and video timestamp when the dwell-time condition generates a `trigger_frame` and starts Gemini analysis.\n\n**Saved-frame console output:** Prints the path of the saved trigger image, for example `dock_monitor_results/trigger_frame_245.jpg`.\n\n**Gemini console output:** Prints the returned dock status, blocking object or condition, reasoning text, and JSON parsing status.\n\n**Completion console output:** Reports whether a trigger was detected and confirms the location of the final annotated video after processing finishes.\n\n## When Not to Use an Agentic Approach\n\nThe fact that a task *can* use a VLM does not mean that it *should*. The easiest way to over-engineer a computer vision application is to put reasoning into a decision that is already well defined. \n\nAgentic systems introduce additional latency, cost, variability, failure modes, external dependencies, and validation requirements. Those costs are justified only when model reasoning solves genuine ambiguity.\n\n### Hard real-time machine decisions\n\nSuppose a production line gives the vision system 200 ms from image acquisition to a reject decision. The system does not need philosophical reasoning about whether a component appears acceptable. It needs a deterministic answer before the product reaches the reject mechanism. That path should remain something like:\n\nRoboflow's edge-inference architecture is designed for this type of problem: keep inference close to the camera and hand the resulting machine-readable decision to the control system without putting a remote language-model request in the critical timing path.\n\nAn agent might still operate above that system. For example, the deterministic pipeline rejects the defective product immediately. Separately, an agent sees that ten similar defects occurred in five minutes, reviews representative images, summarizes the pattern, and opens a maintenance ticket. The agent is valuable because it reasons about the event history. It is not responsible for firing the reject gate.\n\n### Regulated or tightly validated inspections\n\nSome inspection environments require outputs that are reproducible, auditable, and validated against a defined procedure. Open-ended VLM reasoning can make this harder because the output may vary with model version, prompt wording, sampling parameters, API changes, or visual ambiguity.\n\nWhere compliance requires a fixed inspection procedure, the validated model and deterministic decision logic should remain the system of record.\n\nA reasoning model may still assist with explanation, operator support, report generation, or escalation, but it should not silently redefine the acceptance criteria.\n\n### High-volume, low-ambiguity tasks\n\nCounting boxes does not require an agent if a detector, tracker, and line counter already solve the problem. Neither does:\n\n- detect whether a helmet is present\n- count cars entering a parking lot\n- read a barcode\n- measure an object's width\n- reject products containing a known defect\n- count pallets crossing a line\n\nIf the problem can be described as a stable mathematical rule over structured predictions, implement the rule. Adding a language model creates complexity without adding useful intelligence. The practical rule is:\n\nPut the agent above the line, not in it.\n\nKeep deterministic, high-frequency, safety-critical operations in the fast vision pipeline. Use the agent above that layer to interpret exceptions, combine evidence, coordinate software systems, explain unusual events, and decide when a human should become involved.\n\n## Evaluating and Guardrailing an Agentic Vision System\n\nAn agentic vision system contains several stages, so one overall accuracy number is not enough. Evaluate perception, reasoning, actions, and verification separately.\n\n### Evaluate perception separately from reasoning\n\nFirst evaluate the perception model using metrics such as mAP50:95, precision, recall, false positives, and false negatives on data similar to the real deployment environment. Then evaluate the VLM on a separate labeled set using the actual task classes, for example:\n\n```\nNORMAL\nBLOCKED\nIDLE\nUNCERTAIN\n```\n\nMeasure accuracy, precision, recall, confusion between classes, and how often the model gives unsupported answers. An `UNCERTAIN` option is useful because the system can request human review instead of forcing a confident answer when the evidence is weak.\n\n### Evaluate the complete decision path\n\nAfter testing perception and reasoning separately, test the whole system. For an alerting application, measure things such as:\n\n- correct actions\n- false alerts\n- missed events\n- incorrect actions\n- human escalations\n- action failures\n- time from event to action\n\nThe system should be judged by the final operational result. Correct detection is not enough if the alert, ticket, or other action fails.\n\n### Log the evidence behind every important decision\n\nImportant agent decisions should be traceable. Store the image or frame, model predictions, VLM response, prompt version, selected action, action result, and any later verification. Roboflow Vision Events can store production observations together with timestamps, images, predictions, Workflow information, and custom metadata.\n\nFor example:\n\n```\n{\n  \"camera_id\": \"dock_04\",\n  \"decision\": \"BLOCKED\",\n  \"action\": \"CREATE_TICKET\",\n  \"verification\": \"OPEN\"\n}\n```\n\nThis makes it easier to review what the system saw and why it acted.\n\n### Constrain the action space\n\nThe VLM should not be allowed to perform arbitrary actions. Instead, define a small allowlist such as:\n\n```\nNO_ACTION\nSEND_NOTIFICATION\nCREATE_TICKET\nREQUEST_HUMAN_REVIEW\n```\n\nThe application defines which actions are allowed, the model selects one, and the system executes it. This makes the agent easier to test and limits the effect of incorrect reasoning.\n\n### Use thresholds before consequential actions\n\nDo not let every VLM response immediately trigger an external action. A practical rule can require:\n\n```\ndetector confidence > threshold\nAND\nevent duration > minimum duration\nAND\nreasoning result != uncertain\nAND\naction is allowlisted\n```\n\nFor higher-risk actions, require human confirmation or use stricter thresholds. Also, do not assume a confidence value produced by a language model is a calibrated probability unless you have tested it on your own data.\n\n### Keep humans in the loop for consequential decisions\n\nHuman review is appropriate when an action can have important financial, operational, safety, or regulatory consequences. The agent can still reduce workload by collecting the evidence and presenting only the important events, for example:\n\n```\nCamera: Dock 04\nEvent: Possible blocked loading path\nDuration: 8 min 12 sec\nAgent assessment: obstruction likely\nRecommended action: inspect Dock 04\n```\n\nThe system performs the repetitive monitoring, while the operator makes the final high-impact decision. The core principle remains simple:\n\nPerceive what happened. Reason about what it means. Take an allowed action. Verify what happened next.\n\nThe goal is not to replace deterministic computer vision, but to add reasoning only where it provides useful context.\n\n## Conclusion\n\nAgentic computer vision extends vision systems beyond detection by combining perception, reasoning, action, and verification in a controlled loop. The strongest designs keep fast, deterministic vision in the critical path and use VLM reasoning only when context or judgment is actually needed.\n\nBuild your own vision agent with [Roboflow Workflows](https://docs.roboflow.com/workflows?ref=blog.roboflow.com) today by combining detection, tracking, VLM reasoning, and integrations in a single deployable pipeline.\n\n### **Cite this Post**\n\nUse the following entry to cite this post in your research:\n\n[Timothy M](https://blog.roboflow.com/author/timothy/). (Sep 22, 2026).\n      Build Agentic Computer Vision with Roboflow Workflows. Roboflow Blog: https://blog.roboflow.com/agentic-computer-vision/", "url": "https://wpnews.pro/news/build-agentic-computer-vision-with-roboflow-workflows", "canonical_source": "https://blog.roboflow.com/agentic-computer-vision/", "published_at": "2026-09-22 12:47:32+00:00", "updated_at": "2026-09-22 12:53:14.014056+00:00", "lang": "en", "topics": ["computer-vision", "ai-agents", "ai-tools", "large-language-models", "ai-products"], "entities": ["Roboflow", "RF-DETR", "ReAct", "Toolformer", "Reflexion"], "alternates": {"html": "https://wpnews.pro/news/build-agentic-computer-vision-with-roboflow-workflows", "markdown": "https://wpnews.pro/news/build-agentic-computer-vision-with-roboflow-workflows.md", "text": "https://wpnews.pro/news/build-agentic-computer-vision-with-roboflow-workflows.txt", "jsonld": "https://wpnews.pro/news/build-agentic-computer-vision-with-roboflow-workflows.jsonld"}}