{"slug": "how-to-use-a-small-vision-model-as-a-filter-for-an-ai-agent", "title": "How to Use a Small Vision Model as a Filter for an AI Agent", "summary": "A Roboflow tutorial demonstrates using a small object detection model, the RF-DETR Medium checkpoint fabric-kruoy/18 from Roboflow Universe, as a first-pass filter so an AI agent only sends flagged fabric images to a larger vision model. The Workflow returns a selection flag, class labels, confidence scores and bounding boxes, retaining the fabric_broken_pick, fabric_hole, fabric_missing_pick and fabric_thick_weft classes while excluding fabric_perfect, and can be wired to an assistant in ChatGPT or Claude via workflows_run and media_analyze, or to a Python agent built with the OpenAI Agents SDK using gpt-6-sol via screen_fabric_images and view_defect. The approach aims to cut token usage and surface missed defects by having the agent review only flagged regions rather than every inspection image.", "body_md": "To use a small vision model as a filter for an AI agent, run an object detection model as a tool the agent calls first. The tool marks which images need review and returns boxes for the suspected issues, so the large vision model only looks at what was flagged. Build the filter in Roboflow Workflows and connect it to ChatGPT, Claude, or a custom Python agent.\n\nIn a manufacturing setting, inspection cameras capture images of products and components during production. Computer vision models can analyze these images for visible issues such as surface damage, missing components, or incorrect assembly. An AI agent can use the results to request further visual analysis and prepare a quality-control report.\n\nAs the number of inspection images grows, the agent needs a way to select images for detailed review. We can use an object detection model (opens in a new tab) as an image filter and expose it through a Workflow tool. The Workflow returns a selection flag, class labels, confidence scores, and bounding boxes. The agent uses these results to decide which images to review and which regions to inspect.\n\nIn this tutorial, we will build this filtering pipeline with Roboflow Workflows and use it as a tool for an AI agent. We will connect the Workflow through an existing agent application or a custom Python agent, then evaluate image filtering, token usage, and missed defects.\n\nHow to Use a Small Vision Model as a Filter for an AI Agent\n\nWe will use textile manufacturing as our example. In fabric inspection, computer vision models can help locate holes and irregularities in the weave. Our example processes a batch of woven-fabric images to select regions that need closer review.\n\nThe Workflow runs object detection on each image and selects it for agent review when at least one retained defect prediction meets the confidence threshold. The agent then requests visual analysis and reports an assessment for each reviewed region.\n\nBoth options use an AI agent. In Option 1, an existing assistant in ChatGPT or Claude calls the Workflow and requests review through a separate VLM tool. In Option 2, a Python agent built with the OpenAI Agents SDK uses gpt-6-sol to call the tools and analyze the returned region crops.\n\nWe will use the public fabric model on Roboflow Universe (opens in a new tab). The checkpoint is fabric-kruoy/18, listed as RF-DETR Medium. The Workflow retains fabric_broken_pick, fabric_hole, fabric_missing_pick, and fabric_thick_weft. It excludes fabric_perfect from the review decision.\n\nAfter building and testing the Workflow, we will use it as an AI agent tool through either of these methods:\n\nMethod and agent\n\nHow the agent screens images\n\nHow visual review works\n\nOption 1: An assistant in ChatGPT or Claude, connected to Roboflow.\n\nThe assistant calls workflows_run to run the Workflow.\n\nThe assistant calls media_analyze for selected images. A separate vision model reviews the original image, and the assistant uses its analysis to prepare a report.\n\nOption 2: A Python agent built with the OpenAI Agents SDK, using gpt-6-sol as its model.\n\nThe agent calls screen_fabric_images to run the same Workflow.\n\nThe agent calls view_defect to receive padded region crops. Its own model analyzes the crops and writes the report.\n\nChoose Option 1 to use an assistant in ChatGPT or Claude with Roboflow tools for filtering and visual review. Choose Option 2 to build the agent in Python and process a local image folder. In both options, the Roboflow Workflow is the computer vision tool called by the agent. You can follow either option after completing the shared setup.\n\nThe Workflow contains the object detector and output-processing logic. The AI agent runs outside the Workflow. No large language model (LLM) or vision-language model (VLM) block is required inside it, and its output is JSON. A detection is a model prediction that still needs visual review. An image with no retained defect predictions is marked not_selected; this does not establish that the fabric is defect-free.\n\nThe example Workflow is textile-defect-agent-filter in workspace tim-4ijf0. That link opens the example workspace's editor. Readers using another workspace should build their own copy and replace the workspace and Workflow identifiers in the examples.\n\nStep 1: Prepare a fabric image test set\n\nSelect a small set of fabric images with verified defect labels and a consistent orientation. Include clear defects, subtle defects, and images verified as acceptable by an inspector. Keep the original image files unchanged after inference so the returned bounding-box coordinates remain valid.\n\nFor Option 1, use a direct HTTPS image URL that Roboflow can access for both Workflow inference and VLM review. Provide the URL as text. Attaching the image directly to the conversation may send it to the chat model before filtering.\n\nFor Option 2, place JPG, JPEG, PNG, or WEBP files in a local folder named fabric-images. The supplied program also reads subfolders. It uses each file's relative path as image_id, such as roll-01/frame-003.jpg, so files with the same name in different subfolders remain distinct.\n\nKeep the verified labels and inspection notes in a separate record. We will use them to evaluate the complete pipeline.\n\nStep 2: Build the textile defect detection Workflow\n\nIn your Roboflow workspace, select Workflows, create a new Workflow, and choose Build from scratch. Name it textile-defect-agent-filter. The Workflow creation guide (opens in a new tab) explains how to open a blank Workflow.\n\nAdd the components shown below. The main branch runs object detection, counts the retained predictions, and creates the review flag. The remaining connections supply the original image dimensions and image identifier to the result formatter.\n\nConfigure the inputs\n\nAdd one image input and two parameters:\n\nName\n\nType\n\nPurpose and default\n\nimage\n\nImage\n\nThe original fabric image used for inference.\n\nimage_id\n\nText parameter\n\nThe identifier returned with the result. Default: an empty string.\n\nconfidence_threshold\n\nNumber parameter\n\nThe minimum detection score to retain. Default: 0.25.\n\nSupply a unique image_id on every request to associate each result with its input image. The Workflow accepts an empty default, but a unique identifier makes batch results easier to track. Start with a confidence threshold of 0.25 and evaluate it on your test set.\n\nThe model block retains qualifying predictions for these four classes. fabric_perfect does not contribute to the review decision. If an image contains both a retained defect and a fabric_perfect prediction, the retained defect still selects it for review.\n\nThe class and confidence filters are configured in this model block, so this Workflow does not require a separate Detections Filter block. Its prediction output is $steps.detect_textile_defects.predictions.\n\nThe confidence value is a model score. A threshold of 0.25 does not mean the material has a 25% chance of being defective.\n\nSet its data input to $steps.detect_textile_defects.predictions. Use Count Items, represented by the SequenceLength operation in the saved definition. Its output, $steps.count_defect_detections.output, is the number of retained defect predictions.\n\nThis count determines whether the image is selected for review. It represents the number of retained bounding boxes. It does not measure defect severity or the number of defective fabric rolls.\n\nCreate the review flag and selection reason\n\nAdd two Expression blocks (opens in a new tab). In both blocks, create an input variable named count and connect it to $steps.count_defect_detections.output.\n\nName the first block select_for_agent_review. Add a case that checks whether count is greater than 0. Return the Boolean value true when it matches, and set the default result to the Boolean value false.\n\nName the second block set_review_reason. Use the same condition, but return the text possible_defect when it matches and not_selected as the default result.\n\nRetained defect count\n\nReview flag\n\nSelection reason\n\nGreater than 0\n\ntrue\n\npossible_defect\n\n0\n\nfalse\n\nnot_selected\n\nUse Boolean values for the flag, rather than the strings \"true\" and \"false\". The formatter receives the flag from $steps.select_for_agent_review.output and the reason from $steps.set_review_reason.output.\n\nBoth outcomes continue to the formatter. Keeping a result for an image with zero retained detections allows the caller to account for that image in the batch.\n\nRead the original image dimensions\n\nAdd two more Property Definition blocks, both reading $inputs.image:\n\nBlock name\n\nOperation\n\nProperty\n\nget_original_image_width\n\nExtractImageProperty\n\nwidth\n\nget_original_image_height\n\nExtractImageProperty\n\nheight\n\nThe width and height describe the original input image and provide the reference dimensions for the bounding boxes. The Workflow passes the original image to the detector without an earlier crop or transformation block.\n\nCreate the following inputs. The prediction input accepts object detection predictions. The remaining inputs receive the parameter or step outputs shown in the table.\n\nFormatter input\n\nConnect to\n\npredictions\n\n$steps.detect_textile_defects.predictions\n\nimage_id\n\n$inputs.image_id\n\nwidth\n\n$steps.get_original_image_width.output\n\nheight\n\n$steps.get_original_image_height.output\n\ndefect_count\n\n$steps.count_defect_detections.output\n\nselected\n\n$steps.select_for_agent_review.output\n\nreason\n\n$steps.set_review_reason.output\n\nDefine one output named result with the kind dictionary, then use this run function in format_agent_review_json. It matches the uploaded formatter code, laid out for readability. Renaming the step changes its selectors; the function arguments and returned JSON fields stay the same. The dynamic Python block reference (opens in a new tab) explains the input/output definitions and supplies math as a baseline import.\n\nPython\n\n``` python\ndef run(self, predictions, image_id, width, height,\n        defect_count, selected, reason):\n    w, h = int(width), int(height)\n    if w <= 0 or h <= 0:\n        raise ValueError(\"Invalid original image dimensions\")\n\n    n = len(predictions)\n    if n != int(defect_count):\n        raise ValueError(\"Detection count mismatch\")\n\n    names = list(predictions.data.get(\"class_name\", []))\n    scores = predictions.confidence\n    allowed = {\n        \"fabric_broken_pick\", \"fabric_hole\",\n        \"fabric_missing_pick\", \"fabric_thick_weft\"\n    }\n    rows = []\n    if n > 0 and (len(names) != n or scores is None or len(scores) != n):\n        raise ValueError(\"Missing prediction classes or confidence scores\")\n\n    for i, box in enumerate(predictions.xyxy):\n        name = str(names[i])\n        confidence = float(scores[i])\n        coords = [float(v) for v in box]\n        if (name not in allowed or not math.isfinite(confidence)\n                or not all(math.isfinite(v) for v in coords)):\n            raise ValueError(\"Invalid defect prediction\")\n        rows.append({\n            \"class\": name,\n            \"confidence\": confidence,\n            \"bbox\": {\n                \"x_min\": coords[0], \"y_min\": coords[1],\n                \"x_max\": coords[2], \"y_max\": coords[3]\n            }\n        })\n\n    return {\n        \"result\": {\n            \"processing_status\": \"success\",\n            \"image_id\": str(image_id),\n            \"selected_for_review\": bool(selected),\n            \"reason\": str(reason),\n            \"defect_count\": n,\n            \"image_dimensions\": {\"width\": w, \"height\": h},\n            \"coordinate_format\": {\n                \"format\": \"xyxy\", \"units\": \"pixels\",\n                \"reference\": \"original_image\", \"origin\": \"top_left\",\n                \"x_direction\": \"right\", \"y_direction\": \"down\"\n            },\n            \"predictions\": rows\n        }\n    }\n```\n\nThe formatter validates the dimensions, prediction count, class labels, and numeric values before creating the output dictionary. These checks validate the output structure. Visual analysis is still required to assess each predicted defect.\n\nEach prediction includes its class, confidence, and an xyxy box. The two corners are x_min, y_min and x_max, y_max, measured in original-image pixels from the top-left. The final two values are corner coordinates, not the box's width and height.\n\nConfigure the Workflow output\n\nAdd one Workflow output named result and connect it to $steps.format_agent_review_json.result.\n\nThe result contains processing_status, image_id, selected_for_review, reason, defect_count, image_dimensions, coordinate_format, and predictions. Return JSON only from the Workflow. Keep the original image URL or local file path available so the agent can access the image after selection.\n\nSave the Workflow before moving to the image tests.\n\nStep 3: Test the Workflow and inspect the JSON\n\nOpen the Workflow editor's Run panel, supply an image and an image_id, and run the Workflow with confidence_threshold set to 0.25. The Workflow testing guide (opens in a new tab) explains how to inspect the structured output.\n\nHere is a previously recorded response from the example Workflow for image 26f5127e5487946d0079. This is a different input from the public Universe test image used in Step 4, so its predictions are an example of the output format:\n\nJSON\n\n```\n[\n  {\n    \"result\": {\n      \"processing_status\": \"success\",\n      \"image_id\": \"26f5127e5487946d0079\",\n      \"selected_for_review\": true,\n      \"reason\": \"possible_defect\",\n      \"defect_count\": 1,\n      \"image_dimensions\": {\n        \"width\": 2464,\n        \"height\": 2048\n      },\n      \"coordinate_format\": {\n        \"format\": \"xyxy\",\n        \"units\": \"pixels\",\n        \"reference\": \"original_image\",\n        \"origin\": \"top_left\",\n        \"x_direction\": \"right\",\n        \"y_direction\": \"down\"\n      },\n      \"predictions\": [\n        {\n          \"class\": \"fabric_hole\",\n          \"confidence\": 0.26379328966140747,\n          \"bbox\": {\n            \"x_min\": 1626,\n            \"y_min\": 66,\n            \"x_max\": 2085,\n            \"y_max\": 494\n          }\n        }\n      ]\n    }\n  }\n]\n```\n\nThe outer list contains one image result. Inside result, processing_status: \"success\" means screening completed. The Workflow retained one fabric_hole prediction with a confidence score of approximately 0.264, which exceeds the configured 0.25 threshold. It therefore returned selected_for_review: true.\n\nThe box runs from (1626, 66) to (2085, 494) in the original 2464 × 2048 image. These are two corner coordinates, not a position followed by width and height. The confidence score belongs to the detector; it does not confirm that a hole is present.\n\nTest an image verified as acceptable and an image with a subtle defect. Compare the predicted bounding boxes with the verified defect locations. When inference succeeds with no retained predictions, the result should contain selected_for_review: false, reason: \"not_selected\", defect_count: 0, and an empty predictions list. A processing failure must produce an execution error.\n\nThis response demonstrates one successful Workflow inference run. Test agent tool execution and visual analysis separately. Measuring accuracy or token reduction requires an evaluation dataset and comparison runs.\n\nThe published Workflow is ready to run object detection and return structured results. The next step is to make it callable by an AI agent. The agent sends an inference request through a tool, reads the JSON response, and uses the selected predictions to guide visual analysis.\n\nThe output has two roles. selected_for_review controls which images enter the analysis stage. The predictions array supplies the class labels, confidence scores, and bounding boxes for each selected image.\n\nChoose one of the following integrations. In Option 1, an assistant in ChatGPT or Claude calls the Workflow and requests VLM review through a second Roboflow tool. In Option 2, a Python agent built with the OpenAI Agents SDK calls the Workflow through a function tool, then uses its own model, gpt-6-sol, to analyze returned region crops. Both rely on the retained predictions: a missed detection can exclude a defective image from review, and region crops can omit other defects in the same image.\n\nOption 1: Use ChatGPT or Claude as the AI agent\n\nWe use an existing AI assistant in ChatGPT or Claude as the agent, with the Roboflow connection enabled. The agent calls workflows_run to screen the image. When the result selects the image, it calls the separate media_analyze VLM tool for visual review. That tool loads the original image and asks a vision model to assess it. The agent uses the returned analysis to prepare the inspection report.\n\nThis method does not depend on the chat client's ability to open an external image directly. The visual review happens through a separate tool, outside the detection Workflow. The Roboflow MCP guide (opens in a new tab) lists both tools and the supported connection methods.\n\nThe review tool receives the full original image. The bounding boxes in its question guide the visual analysis. Option 2 provides explicit region crops when you need that control over the image input.\n\n1. Enable the Roboflow tools\n\nIn ChatGPT, install the Roboflow plugin (opens in a new tab) and sign in when prompted. In Claude, use the Roboflow connector linked in the MCP setup guide (opens in a new tab). If your client is already connected to the correct workspace, check that it exposes both workflows_run and media_analyze. Tool names may include a client-specific prefix.\n\nThe connected account must have access to your published Workflow and permission to use both tools. If you built your own copy, replace the workspace and Workflow identifiers below. Roboflow must also be able to retrieve the image URL. Pasting the prompt into a chat without the connected tools will not execute the process.\n\nmedia_analyze uses workspace credits for a vision-model call. We call it only for selected images. The detection Workflow remains JSON-only and has no LLM or VLM block.\n\n2. Request filtering followed by visual review\n\nUse this prompt with an original image from the fabric dataset on Roboflow Universe (opens in a new tab). The dataset lists image NM0YAb5kjVesmskIj3Be in its test split with one fabric_hole annotation. This is a reference label. Inference and visual review must produce their own results.\n\nImage URL points directly to the JPEG file. image_id is a tracking label returned unchanged by the Workflow. We use the dataset image ID here, but another unique label also works. Enable the Roboflow tools for the conversation and paste this prompt:\n\n```\nUse the Roboflow tools to inspect this fabric image.\n\nWorkspace: tim-4ijf0\nWorkflow ID: textile-defect-agent-filter-1793090127138\nImage URL: https://source.roboflow.com/jCsmAr4784Mgz4kk6OH9ETCa6BD2/NM0YAb5kjVesmskIj3Be/original.jpg\nimage_id: NM0YAb5kjVesmskIj3Be\nconfidence_threshold: 0.25\n\nFirst call workflows_run with the image URL, image_id, and threshold.\nDo not request visual analysis before reading its result.\n\nIf processing fails, report the execution error separately.\nIf selected_for_review is false, report not_selected and stop.\nDo not describe an unselected image as verified defect-free.\n\nIf selected_for_review is true, call media_analyze with the same URL\nand media_type=image. In its question, include the original image\ndimensions and every returned class, confidence score, and xyxy box.\nAsk the vision model to inspect those regions and surrounding texture,\ndescribe the visible evidence, and assess each prediction as\nvisually_supported, not_supported, or uncertain. Treat the detector\noutputs as unverified predictions, including high-confidence ones.\n\nUse the returned visual analysis to write the report. If the tool is\nunavailable, fails, or cannot inspect a region, mark that region\nnot_reviewed and explain why. Use uncertain only when an image was\ninspected but the visual evidence was inconclusive.\n\nKeep detector results separate from VLM assessments. Do not infer\nappearance from JSON or treat a model assessment as physical confirmation.\nReturn short bullet points with the image ID, screening result, detector\npredictions, review tool, visual assessment, evidence, and next action\nfor a human inspector.\n```\n\nThe Workflow sets the selection flag. The agent's instructions use that flag to request the second tool call. The Workflow itself does not start the visual review.\n\n3. Check both tool calls\n\nThe first call runs the published Workflow:\n\n```\n{\n  \"workflow_id\": \"textile-defect-agent-filter-1793090127138\",\n  \"images\": {\n    \"image\": \"https://source.roboflow.com/jCsmAr4784Mgz4kk6OH9ETCa6BD2/NM0YAb5kjVesmskIj3Be/original.jpg\"\n  },\n  \"parameters\": {\n    \"image_id\": \"NM0YAb5kjVesmskIj3Be\",\n    \"confidence_threshold\": 0.25\n  }\n}\n```\n\nThe workspace comes from the Roboflow connection. The Workflow ID is its URL slug. After a successful result with selected_for_review: true, the conversation should show a media_analyze call. Its question should use the dimensions and predictions from that execution.\n\nFor the screening result reported for this test image, the review call has this form:\n\n```\n{\n  \"url\": \"https://source.roboflow.com/jCsmAr4784Mgz4kk6OH9ETCa6BD2/NM0YAb5kjVesmskIj3Be/original.jpg\",\n  \"media_type\": \"image\",\n  \"question\": \"Visually review the predicted region in this 2048 x 1536 fabric image. The detector suggested fabric_hole with confidence 0.6238 at xyxy [328, 640, 565, 914], measured in original-image pixels from the top-left. Inspect this region and the surrounding weave. Describe the visible evidence and assess the prediction as visually_supported, not_supported, or uncertain. Treat the detector output as an unverified prediction. If you cannot inspect the image or region, report not_reviewed and explain why. Do not assess the rest of the fabric as defect-free.\"\n}\n```\n\nThis second JSON example uses values reported for this input. For another image or a new inference result, use its returned values. The coordinates in Step 3 belong to a different image.\n\nmedia_analyze returns a structured analysis with a summary. The requested assessment labels belong in the agent's report; they are not new fields in the Workflow output. Use the tool's visual observations to assess the selected predictions. Keep the Workflow's class labels and detection count separate from any additional model recommendations.\n\n4. Interpret the review result\n\nFor this image, the reported Workflow run returned one fabric_hole prediction with confidence 0.6238, a bounding box of [328, 640, 565, 914], and selected_for_review: true. Its dimensions were 2048 × 1536 pixels.\n\nA subsequent live media_analyze call successfully loaded the image. The vision model assessed the region as visually supported and described a localized disruption in the weave, broken yarn, and a small opening. This is a VLM assessment for human review, not confirmation from a physical inspection.\n\nThe earlier image-access error occurred after Workflow inference had succeeded. It meant that the client could not complete its own image-viewing step. Calling the review tool resolved that step in this test. The successful review call is recorded in visual-review-test-result.json in the tutorial files. This does not establish accuracy across a dataset or compatibility in every client.\n\nWhen image loading or review fails, report not reviewed. Use uncertain when the review actually ran but the visual evidence was inconclusive. Keep either outcome separate from processing_status: \"success\", which only describes the Workflow execution.\n\nOption 2: Build a Python agent with the OpenAI Agents SDK\n\nWe create a custom agent in Python using the OpenAI Agents SDK (opens in a new tab). In our example, gpt-6-solis the agent's model. It calls screen_fabric_images to run the Roboflow Workflow on images in a local folder, then calls view_defect to receive padded crops of the selected regions. The same agent model analyzes those crops and writes the inspection report.\n\nThe SDK manages the agent's tool-calling loop: it executes each function requested by the model and returns the tool output for the next model turn. The Workflow supplies the detection results, and the viewing tool supplies the image crops.\n\nThe first tool sends image data to Roboflow for inference. It returns counts, image identifiers, and detection metadata to the agent model. The second tool supplies image data when the agent requests a selected region.\n\n1. Prepare the Python project\n\nUse a Python 3.12 environment for this example. Create a project folder containing fabric-images, then install the packages:\n\nShell\n\n```\npython -m pip install openai-agents inference-sdk\n```\n\nTo use the same Universe sample, save the original test image (opens in a new tab) as fabric-images/NM0YAb5kjVesmskIj3Be.jpg. This Python example reads local image files. It uses each relative filename as image_id, so this sample will be reported as NM0YAb5kjVesmskIj3Be.jpg.\n\nConfigure ROBOFLOW_API_KEY and OPENAI_API_KEY privately in the environment used to run the program. The Roboflow key must access the workspace containing your Workflow. These API credentials are separate from signing into an agent application's plugin. See the Roboflow API-key guide (opens in a new tab) and the Agents SDK quickstart (opens in a new tab).\n\nThe code uses gpt-6-sol, which supports image input and function calling through the Responses API. Use an API account with access to that model. The model page (opens in a new tab) lists its capabilities and current pricing.\n\nCreate fabric_agent.py and add the following code sections in order. The complete file and an optional comparison script are also included in the accompanying tutorial files.\n\n2. Add the imports and response validation\n\nThe settings identify the published Workflow. validate_result checks the image identifier, prediction count, selection flag, and bounding-box coordinates. An invalid response is recorded as a processing error.\n\nPython\n\n```\n\"\"\"Use a published Roboflow Workflow as a tool for a Python inspection agent.\"\"\"\n\nimport argparse\nimport base64\nimport io\nimport json\nimport math\nimport os\nimport time\nfrom datetime import datetime\nfrom pathlib import Path\n\nfrom agents import (\n    Agent, ModelSettings, RunConfig, Runner, ToolOutputImage, ToolOutputText,\n    function_tool, set_default_openai_api,\n)\nfrom inference_sdk import InferenceConfiguration, InferenceHTTPClient\nfrom PIL import Image\n\nWORKSPACE = \"tim-4ijf0\"\nWORKFLOW_ID = \"textile-defect-agent-filter-1793090127138\"\nIMAGE_TYPES = {\".jpg\", \".jpeg\", \".png\", \".webp\"}\nDEFECT_CLASSES = {\n    \"fabric_broken_pick\", \"fabric_hole\",\n    \"fabric_missing_pick\", \"fabric_thick_weft\",\n}\nCROP_MAX_SIDE = 1024\n\ndef validate_result(response, image_id):\n    \"\"\"Reject inconsistent screening data before offering it to the agent.\"\"\"\n    if not isinstance(response, list) or len(response) != 1:\n        raise ValueError(\"Expected one Workflow result\")\n    result = response[0][\"result\"]\n    if (result[\"processing_status\"] != \"success\"\n            or result[\"image_id\"] != image_id):\n        raise ValueError(\"Unsuccessful or mismatched result\")\n    predictions, count = result[\"predictions\"], result[\"defect_count\"]\n    if (not isinstance(predictions, list) or type(count) is not int\n            or count != len(predictions)):\n        raise ValueError(\"Invalid prediction count\")\n    selected = result[\"selected_for_review\"]\n    if type(selected) is not bool or selected != (count > 0):\n        raise ValueError(\"Invalid review decision\")\n    if result[\"reason\"] != (\"possible_defect\" if selected else \"not_selected\"):\n        raise ValueError(\"Invalid selection reason\")\n    dims = result[\"image_dimensions\"]\n    if any(type(dims[k]) is not int or dims[k] <= 0 for k in (\"width\", \"height\")):\n        raise ValueError(\"Invalid image dimensions\")\n    expected = {\"format\": \"xyxy\", \"units\": \"pixels\", \"reference\": \"original_image\",\n                \"origin\": \"top_left\", \"x_direction\": \"right\", \"y_direction\": \"down\"}\n    if any(result[\"coordinate_format\"].get(k) != v for k, v in expected.items()):\n        raise ValueError(\"Unexpected coordinate format\")\n    for hint in predictions:\n        score = float(hint[\"confidence\"])\n        box = hint[\"bbox\"]\n        x1, y1, x2, y2 = [float(box[k]) for k in (\"x_min\", \"y_min\", \"x_max\", \"y_max\")]\n        if (hint[\"class\"] not in DEFECT_CLASSES or not math.isfinite(score)\n                or not 0 <= score <= 1\n                or not all(math.isfinite(v) for v in (x1, y1, x2, y2))\n                or not x1 < x2 or not y1 < y2\n                or x2 <= 0 or y2 <= 0 or x1 >= dims[\"width\"] or y1 >= dims[\"height\"]):\n            raise ValueError(\"Invalid defect hint\")\n    return result\n```\n\nThe returned bbox is a dictionary. Read its named fields, such as box[\"x_min\"]; unpacking the dictionary itself would read its keys instead of the numeric coordinates.\n\n3. Add the inference and image-viewing tools\n\nbuild_tools configures both tools with the image folder, confidence threshold, and Roboflow client. The agent receives the tool definitions. The Python program stores local file paths and inference results, then returns the requested data through tool outputs.\n\nPython\n\n``` python\ndef build_tools(folder, client, confidence=0.25,\n                workspace=WORKSPACE, workflow_id=WORKFLOW_ID):\n    \"\"\"Keep each run's files and results in its own local state.\"\"\"\n    folder = Path(folder).resolve()\n    if not folder.is_dir() or not math.isfinite(confidence) or not 0 <= confidence <= 1:\n        raise ValueError(\"Provide an existing folder and confidence from 0 to 1\")\n    paths = sorted(p for p in folder.rglob(\"*\")\n                   if p.is_file() and p.suffix.lower() in IMAGE_TYPES)\n    if not paths:\n        raise ValueError(\"No supported image files found\")\n    selected_files = {}\n    state = {\"summary\": None, \"records\": [], \"views\": [], \"view_errors\": []}\n\n    @function_tool\n    def screen_fabric_images() -> str:\n        \"\"\"Screen the configured folder with Roboflow before viewing any image.\n\n        Returns JSON counts, selected-image hints and processing errors.\n        No image pixels are returned. Repeated calls reuse this run's results.\n        \"\"\"\n        if state[\"summary\"] is not None:\n            return json.dumps(state[\"summary\"])\n        print(\"[tool] screen_fabric_images\")\n        selected, errors = [], []\n        for path in paths:\n            image_id = path.relative_to(folder).as_posix()\n            try:\n                response = client.run_workflow(\n                    workspace_name=workspace, workflow_id=workflow_id,\n                    images={\"image\": str(path)},\n                    parameters={\"image_id\": image_id,\n                                \"confidence_threshold\": confidence},\n                    use_cache=False,\n                )\n                result = validate_result(response, image_id)\n                state[\"records\"].append({\"path\": str(path), \"result\": result})\n                if result[\"selected_for_review\"]:\n                    selected_files[image_id] = (path, result)\n                    selected.append({\n                        \"image_id\": image_id,\n                        \"image_dimensions\": result[\"image_dimensions\"],\n                        \"hints\": [{\"defect_index\": i, **hint}\n                                  for i, hint in enumerate(result[\"predictions\"])],\n                    })\n            except Exception as exc:\n                errors.append({\"image_id\": image_id, \"error_type\": type(exc).__name__})\n        state[\"summary\"] = {\n            \"images_total\": len(paths), \"screened\": len(state[\"records\"]),\n            \"selected_count\": len(selected),\n            \"not_selected\": len(state[\"records\"]) - len(selected),\n            \"selected\": selected, \"processing_errors\": errors,\n        }\n        return json.dumps(state[\"summary\"])\n\n    @function_tool\n    def view_defect(image_id: str, defect_index: int) -> list[ToolOutputText | ToolOutputImage]:\n        \"\"\"Show one padded region from an image selected by screen_fabric_images.\n\n        Args:\n            image_id: An identifier in the screening tool's selected list.\n            defect_index: The zero-based hint index returned by that tool.\n        \"\"\"\n        print(f\"[tool] view_defect: {image_id}, hint {defect_index}\")\n        try:\n            path, result = selected_files[image_id]\n            data_url, bounds, size = make_crop(path, result, defect_index)\n            record = {\"image_id\": image_id, \"defect_index\": defect_index,\n                      \"crop_xyxy\": list(bounds), \"display_size\": list(size)}\n            state[\"views\"].append(record)\n            return [ToolOutputText(text=json.dumps(record)),\n                    ToolOutputImage(image_url=data_url, detail=\"high\")]\n        except Exception as exc:\n            record = {\"image_id\": image_id, \"defect_index\": defect_index,\n                      \"error_type\": type(exc).__name__}\n            state[\"view_errors\"].append(record)\n            return [ToolOutputText(text=json.dumps({\"view_error\": record}))]\n\n    return [screen_fabric_images, view_defect], state\n```\n\nThe @function_tool decorator makes each function available to the agent. The function name, parameter types, and docstring describe the tool's inputs and purpose. The tools become callable by this agent when we pass them to its tools list below. See the agent definitions guide (opens in a new tab).\n\nThe screening tool processes one image at a time through the Roboflow Inference SDK (opens in a new tab). use_cache=False avoids reusing a recently cached Workflow definition during testing. Keep the published version fixed while comparing results.\n\nEvery successful inference result is stored in the local audit record. Only selected images enter selected_files and the selected list returned to the agent. The hints field contains each prediction's class, confidence score, and bounding box. Each prediction also receives a zero-based defect_index, which identifies the region to request. Inference errors remain separate from successful not_selected results.\n\nThe second tool uses image_id and defect_index to retrieve a region from the selected records. It rejects requests for unselected images. Repeated calls to the first tool return the stored results for that run.\n\n4. Create the region-of-interest crop\n\nThe viewing tool calls this helper to crop the region of interest. It reads the bounding box in original-image pixels, adds padding for visual context, and clips the crop coordinates to the image boundaries.\n\nPython\n\n``` python\ndef make_crop(path, result, defect_index):\n    \"\"\"Read named xyxy coordinates and return a bounded crop as a data URL.\"\"\"\n    if not 0 <= defect_index < len(result[\"predictions\"]):\n        raise ValueError(\"Unknown defect index\")\n    box = result[\"predictions\"][defect_index][\"bbox\"]\n    x1, y1, x2, y2 = [float(box[k]) for k in (\"x_min\", \"y_min\", \"x_max\", \"y_max\")]\n    with Image.open(path) as source:\n        image = source.convert(\"RGB\")\n    dims = result[\"image_dimensions\"]\n    if image.size != (dims[\"width\"], dims[\"height\"]):\n        raise ValueError(\"Original image dimensions changed\")\n    pad_x = 0.25 * (x2 - x1) + 32\n    pad_y = 0.25 * (y2 - y1) + 32\n    bounds = (max(0, math.floor(x1 - pad_x)), max(0, math.floor(y1 - pad_y)),\n              min(image.width, math.ceil(x2 + pad_x)),\n              min(image.height, math.ceil(y2 + pad_y)))\n    if bounds[0] >= bounds[2] or bounds[1] >= bounds[3]:\n        raise ValueError(\"Empty crop\")\n    crop = image.crop(bounds)\n    crop.thumbnail((CROP_MAX_SIDE, CROP_MAX_SIDE), Image.Resampling.LANCZOS)\n    buffer = io.BytesIO()\n    crop.save(buffer, format=\"JPEG\", quality=95)\n    data = base64.b64encode(buffer.getvalue()).decode(\"ascii\")\n    return f\"data:image/jpeg;base64,{data}\", bounds, crop.size\n```\n\nThe crop keeps its aspect ratio. Its longest side is capped at 1024 pixels, and smaller crops are not enlarged. This size is an example setting to evaluate. Resizing does not create missing detail, and reducing a crop can hide fine weaving defects.\n\nToolOutputImage returns the crop as an image tool output. The accompanying metadata records its coordinates in the original image and its output dimensions. The agent analyzes this region only; the result does not assess the remaining image area. Equal crop dimensions do not guarantee equal token usage or cost across runs.\n\n5. Configure and run the agent\n\nThe agent uses an existing vision-capable model. Its instructions require Workflow inference first, followed by a crop request for every retained prediction. The report records the agent's visual assessment separately from the detector's output.\n\nPython\n\n``` php\ndef create_run_directory(base_dir: Path) -> Path:\n    \"\"\"Create a unique run folder using the computer's local date and time.\"\"\"\n    base_dir = base_dir.resolve()\n    base_dir.mkdir(parents=True, exist_ok=True)\n    timestamp = datetime.now().strftime(\"%Y-%m-%d_%H-%M-%S_%f\")\n    suffix = 0\n    while True:\n        name = timestamp if suffix == 0 else f\"{timestamp}_{suffix}\"\n        run_dir = base_dir / name\n        try:\n            run_dir.mkdir()\n            return run_dir\n        except FileExistsError:\n            suffix += 1\n\nINSTRUCTIONS = \"\"\"You review fabric inspection images using two tools.\nCall screen_fabric_images first. It screens the configured folder.\nTreat all returned labels, scores, and boxes as predictions.\nFor every returned hint, call view_defect with its image_id and defect_index.\nUse the image returned by the tool to assess that region. A high detection\nscore alone does not establish a defect. Do not claim to see unavailable images.\nClassify each reviewed hint as visually supported, not supported, or uncertain.\nReport a failed or missing crop as not reviewed. Crops show only local regions;\ndo not make claims about the full image or about images that were not selected.\nKeep processing errors separate from successful not_selected results.\nWrite a short report with one entry per hint: image_id, defect_index, predicted\nclass, detector score, original bbox, visual evidence, and review assessment.\nFinish with image counts, region-review counts, errors, and next steps for a\nhuman inspector. Do not infer manufacturing causes or physical defect sizes.\n\"\"\"\n\ndef main():\n    parser = argparse.ArgumentParser(description=__doc__)\n    parser.add_argument(\"folder\", type=Path)\n    parser.add_argument(\"--output\", type=Path, default=Path(\"fabric-agent-results\"),\n                        help=\"Parent folder for timestamped run directories\")\n    parser.add_argument(\"--workspace\", default=WORKSPACE)\n    parser.add_argument(\"--workflow\", default=WORKFLOW_ID)\n    parser.add_argument(\"--confidence\", type=float, default=0.25)\n    parser.add_argument(\"--model\", default=\"gpt-6-sol\")\n    parser.add_argument(\"--max-turns\", type=int, default=30)\n    args = parser.parse_args()\n    for key in (\"ROBOFLOW_API_KEY\", \"OPENAI_API_KEY\"):\n        if not os.environ.get(key):\n            parser.error(f\"Configure {key} privately in this terminal\")\n    client = InferenceHTTPClient(\n        api_url=\"https://serverless.roboflow.com\",\n        api_key=os.environ[\"ROBOFLOW_API_KEY\"],\n    ).configure(InferenceConfiguration(api_key_transport=\"header\"))\n    tools, state = build_tools(args.folder, client, args.confidence,\n                               args.workspace, args.workflow)\n    output_dir = create_run_directory(args.output)\n    print(f\"[output] {output_dir}\")\n    set_default_openai_api(\"responses\")\n    agent = Agent(name=\"Fabric inspection agent\", model=args.model,\n                  instructions=INSTRUCTIONS, tools=tools,\n                  model_settings=ModelSettings(parallel_tool_calls=False))\n    started = time.perf_counter()\n    try:\n        result = Runner.run_sync(\n            agent, \"Inspect the configured fabric folder and write the review report.\",\n            max_turns=args.max_turns, run_config=RunConfig(tracing_disabled=True),\n        )\n    except Exception as exc:\n        (output_dir / \"partial-run.json\").write_text(\n            json.dumps({\"run_error_type\": type(exc).__name__, \"audit\": state}, indent=2),\n            encoding=\"utf-8\",\n        )\n        print(f\"Run incomplete ({type(exc).__name__}); \"\n              f\"inspect {output_dir / 'partial-run.json'}.\")\n        return 2\n    usage = result.context_wrapper.usage\n    requested = {(item[\"image_id\"], hint[\"defect_index\"])\n                 for item in (state[\"summary\"] or {}).get(\"selected\", [])\n                 for hint in item[\"hints\"]}\n    returned = {(v[\"image_id\"], v[\"defect_index\"]) for v in state[\"views\"]}\n    metrics = {\n        \"model\": args.model, \"workflow\": args.workflow, \"confidence\": args.confidence,\n        \"elapsed_seconds\": time.perf_counter() - started,\n        \"input_tokens\": usage.input_tokens, \"output_tokens\": usage.output_tokens,\n        \"total_tokens\": usage.total_tokens,\n        \"screening_called\": state[\"summary\"] is not None,\n        \"regions_requested\": len(requested), \"regions_returned\": len(returned),\n        \"regions_without_image\": [list(pair) for pair in sorted(requested - returned)],\n    }\n    summary = state[\"summary\"] or {}\n    for key in (\"images_total\", \"screened\", \"selected_count\", \"not_selected\"):\n        metrics[key] = summary.get(key, 0)\n    metrics[\"processing_errors\"] = len(summary.get(\"processing_errors\", []))\n    metrics[\"view_errors\"] = len(state[\"view_errors\"])\n    (output_dir / \"report.md\").write_text(str(result.final_output), encoding=\"utf-8\")\n    for name, data in ((\"audit.json\", state), (\"usage.json\", metrics)):\n        (output_dir / name).write_text(json.dumps(data, indent=2), encoding=\"utf-8\")\n    print(result.final_output)\n    print(json.dumps(metrics, indent=2))\n    print(f\"Results saved to: {output_dir}\")\n    complete = (metrics[\"screening_called\"] and not metrics[\"regions_without_image\"]\n                and not metrics[\"processing_errors\"] and not metrics[\"view_errors\"])\n    return 0 if complete else 2\n\nif __name__ == \"__main__\":\n    raise SystemExit(main())\n```\n\nPassing the two functions to tools makes them available to the agent. Runner.run_sync starts the agent execution loop. The initial request contains the inspection instructions. The agent first calls the inference tool, then requests selected region crops for visual analysis.\n\nThe agent must request a crop for every retained prediction, including predictions with high confidence scores. Each visual assessment is recorded as visually supported, not supported, or uncertain. If image loading fails, the outcome is not reviewed.\n\nRun the program from its project folder:\n\nShell\n\n```\npython fabric_agent.py ./fabric-images --output ./fabric-agent-results\n```\n\nFor a Workflow in another workspace, provide its identifiers:\n\nShell\n\n```\npython fabric_agent.py ./fabric-images --workspace YOUR_WORKSPACE --workflow YOUR_WORKFLOW_ID --output ./fabric-agent-results\n```\n\nThe terminal prints [tool] screen_fabric_images for the inference tool call and [tool] view_defect for each region request. Start with a small image batch to inspect the execution sequence. The default limit is 30 model turns. If the number of retained predictions requires more turns, reduce the batch size or adjust --max-turns.\n\nEach run creates a timestamped subfolder inside --output, using your computer's local date and time. For example, one run could use fabric-agent-results/2026-10-28_14-30-05_123456. The terminal prints the full folder path. You can repeat the same command; each run keeps its own results. If two runs have the same timestamp, the program adds a numeric suffix.\n\nThe program writes these files inside that run folder:\n\nFile\n\nWhat it contains\n\nreport.md\n\nThe agent's visual assessments and review notes.\n\naudit.json\n\nScreening results, processing errors, returned crop records, and crop errors.\n\nusage.json\n\nModel token usage, elapsed time, image counts, and region-return counts.\n\nIf the agent run fails before completing, partial-run.json preserves the screening and viewing records collected so far in that run's timestamped folder. An exit code of 2 indicates an incomplete run or recorded errors. Existing reports remain available in their original folders.\n\nA returned crop confirms that image data was supplied to the agent. Validate the visual assessment against the original images and verified defect labels. Check regions_without_image for regions without a returned image, and send unresolved cases for human review.\n\n6. Read the result from a two-image test\n\nWe ran the Python agent on two fabric images with confidence_threshold set to 0.25. The Workflow screened both images, selected one for review, and returned no retained defect predictions for the other. The agent then requested and reviewed one region crop. The following results come from that recorded run.\n\nThe terminal shows the two tool calls:\n\n```\n[tool] screen_fabric_images\n[tool] view_defect: original.jpg, hint 0\n```\n\nThe first call screened the image folder through the Roboflow Workflow. The second call returned a padded crop around the first prediction in original.jpg. The index is 0 because prediction indices start at zero.\n\nThe selected region produced this result:\n\nField\n\nRecorded result\n\nImage ID\n\noriginal.jpg\n\nPrediction index\n\n0\n\nPredicted class\n\nfabric_hole\n\nDetector score\n\n0.624, as rounded in the report\n\nBounding box\n\n(328, 640, 565, 914), in xyxy original-image pixels\n\nVisual assessment\n\nUncertain\n\nThe detector score exceeded the configured threshold, so the Workflow retained the prediction and selected the image. The bounding box identifies the region from top-left coordinate (328, 640) to bottom-right coordinate (565, 914). The viewing tool added surrounding image context before passing the crop to the agent.\n\nThe agent reported a dark, irregular area interrupting the fabric pattern, but it could not clearly identify an opening. It therefore marked the region uncertain. This was a completed visual review: the agent received the crop and assessed its visible evidence. A missing or unreadable crop would instead be not reviewed. The next step is for a human inspector to examine the flagged region in the original image or inspect the fabric.\n\nThis assessment differs from the earlier Option 1 result. Option 1 used a separate VLM tool to review the original image; this run used the Python agent to review a region crop. The review inputs and execution paths differ, so the assessments should be evaluated separately.\n\nThe second image was not selected because the Workflow returned no qualifying defect predictions. The agent did not visually review it. This result does not establish that the image is defect-free; missed detections must be checked during evaluation.\n\nThe usage output also confirms that regions_requested and regions_returned were both 1, while regions_without_image was empty. Both processing_errors and view_errors were 0. The report counted one uncertain region, with zero regions assessed as visually supported or not supported.\n\nRun metric\n\nRecorded value\n\nAgent model\n\ngpt-6-sol\n\nAgent run time\n\nApproximately 16.75 seconds\n\nInput tokens\n\n1,988\n\nOutput tokens\n\n309\n\nTotal tokens\n\n2,297\n\nThese token counts cover the agent run and exclude Roboflow inference charges. Timing, token usage, and visual assessments can vary between runs. Use the comparison in Step 5 to measure token savings against a baseline that reviews all original images.\n\nStep 5: Evaluate image filtering and agent analysis\n\nWe can now compare how the same vision-capable model reviews images with and without filtering. This helps us measure the review workload and examine whether detection metadata helps the model assess a suspected defect.\n\nThe optional compare_review.py script runs three comparison conditions. All three use full original images:\n\nCondition\n\nImages sent to the model\n\nDetection metadata\n\nBaseline\n\nEvery image in the folder\n\nNone\n\nFilter only\n\nImages selected by the Workflow\n\nNone\n\nFilter with hints\n\nThe same selected images\n\nPredicted classes, confidence scores, bounding boxes, and related metadata\n\nUse the same image folder, model, and reporting instructions for all three runs. The two filtered conditions should read the same audit.json so they use exactly the same selected images.\n\nHow the Comparison Works\n\nFirst, run fabric_agent.py as explained in Option 2. It calls the Roboflow Workflow to screen the images and saves the screening results in audit.json.\n\nNext, run compare_review.py separately for each condition. Without --audit, it sends every image to the model. With --audit, it reads the saved screening results and sends only the selected images. Adding --with-hints also includes their detection metadata.\n\nThe comparison script does not run the Workflow again. It reuses the screening results and loads the selected original images from your local folder.\n\nFor each run, the model examines the submitted images and writes a visual assessment. The script saves:\n\nreport.md — image IDs, suspected locations, visible evidence, uncertainty, and suggested next steps for human review.\n\nusage.json — the model, comparison condition, number of images submitted, token usage, and review execution time.\n\nRun the Three Comparisons\n\nRun the following commands from the folder containing compare_review.py. Use the Anaconda environment configured for Option 2, with OPENAI_API_KEY available.\n\nReplace RUN_TIMESTAMP with the timestamped folder name printed by fabric_agent.py. Resolve any screening errors before using its audit file.\n\n1. Review all images\n\nThis creates the baseline without image filtering.\n\nShell\n\n```\npython compare_review.py ./fabric-images --output ./baseline-results\n```\n\n2. Review only selected images\n\nThis uses audit.json to choose the images, but does not send the detector predictions to the model.\n\nShell\n\n```\npython compare_review.py ./fabric-images --audit ./fabric-agent-results/RUN_TIMESTAMP/audit.json --output ./filter-only-results\n```\n\n3. Review selected images with detection hints\n\nThis sends the same selected images together with their detection metadata. The model must treat these hints as predictions that require visual assessment.\n\nShell\n\n```\npython compare_review.py ./fabric-images --audit ./fabric-agent-results/RUN_TIMESTAMP/audit.json --with-hints --output ./filter-hints-results\n```\n\nRun each command separately. This version of compare_review.py requires a new output folder for every run. For repeated tests, change the folder names, for example to baseline-results-02.\n\nThe script submits the chosen images together, so start with a small folder. If no images were selected, the filtered comparison stops without a visual review request. Record that outcome separately.\n\nRead the Results from our Test\n\nOur test folder contained image0000002.jpg and original.jpg. The Workflow selected original.jpg for review. The baseline therefore reviewed two images, while both filtered conditions reviewed one.\n\nThe screenshot shows these measurements:\n\nCondition\n\nImages submitted\n\nInput tokens\n\nOutput tokens\n\nTotal tokens\n\nReview time\n\nBaseline\n\n2\n\n5,972\n\n306\n\n6,278\n\n9.55 s\n\nFilter only\n\n1\n\n3,022\n\n305\n\n3,327\n\n7.71 s\n\nFilter with hints\n\n1\n\n3,128\n\n281\n\n3,409\n\n7.39 s\n\nFiltering reduced the number of images submitted for visual review from two to one. The filter-only run used fewer input tokens because it excluded the unselected image. Adding detection hints increased the input count by 106 tokens compared with the filter-only run. We can calculate the input-token reduction relative to the baseline:\n\n```\nInput-token reduction (%) =\n100 × (1 − filtered input tokens / baseline input tokens)\n```\n\nFor this test:\n\nFilter only:100 × (1 − 3022 / 5972) = 49.4%\n\nFilter with hints:100 × (1 − 3128 / 5972) = 47.6%\n\nThese results describe this two-image test. They do not establish the same reduction or execution time for other datasets.\n\nCompare the Visual Reports\n\nToken usage shows the review workload. We also need to examine what the model reported.\n\nBaseline report\n\nThe baseline reviewed both images. For image0000002.jpg, it described a continuous weave and did not identify a clear defect, while noting that subtle defects might not be distinguishable.\n\nFor original.jpg, it identified an irregular dark patch left of the image center. It could not determine whether the patch was a hole or a surface mark and suggested closer human inspection.\n\nimage_id\n\nLocation\n\nVisible evidence\n\nUncertainty\n\nNext step for human review\n\nimage0000002.jpg\n\nEntire visible area\n\nThe weave appears continuous. No clear broken pick, hole, missing pick, or thick weft is visible.\n\nSubtle defects may not be distinguishable at this image quality.\n\nConfirm with a close-up inspection if this area has been flagged.\n\noriginal.jpg\n\nLeft of center, around x=390, y=690\n\nAn irregular dark patch obscures the weave locally.\n\nIt could be a hole or a surface mark; the image does not show an opening clearly. No distinct broken pick, missing pick, or thick weft is apparent elsewhere.\n\nInspect the patch closely from both sides, using backlighting to check for an opening.\n\nFilter-only report\n\nThis report covered only original.jpg. The model again identified a localized dark patch interrupting the weave. It remained uncertain whether the region contained a hole, broken picks, or a surface mark.\n\nThis shows that the selected image still received visual review after filtering. The excluded image was not reviewed in this condition.\n\nimage_id\n\nLocation\n\nVisible evidence\n\nUncertainty\n\nNext step for human review\n\noriginal.jpg\n\nLeft of center, around (395, 690)\n\nA localized dark, irregular patch interrupts the otherwise regular weave.\n\nThe image does not establish whether this is a hole, broken picks, or a surface mark. No clear missing pick or thick weft is visible elsewhere.\n\nInspect the patch under magnification from both sides; check yarn continuity and use backlighting to assess whether there is an opening.\n\nFilter-with-hints report\n\nThe model received the selected image and its detection metadata. It described the suspected region around x = 410–490 and y = 720–840, but still could not establish whether the fabric contained an opening.\n\nThe report explicitly left the detector’s fabric_hole prediction unverified. In this test, providing detection hints did not resolve the visual uncertainty.\n\nimage_id\n\nLocation (approx. image pixels)\n\nVisible evidence\n\nUncertainty\n\nNext step for human review\n\noriginal.jpg\n\nLeft of center, around x=410–490, y=720–840\n\nA dark, irregular patch interrupts the otherwise regular striped weave.\n\nPossible hole, but the image does not show clearly whether the fabric is open or merely darkened. No clear broken pick, missing pick, or thick weft is visible elsewhere. The detector’s hole label remains unverified.\n\nInspect this area closely, preferably from both sides or with backlighting, to determine whether there is an opening and whether threads are broken.\n\nAcross all three reports, the suspected region in original.jpg required human review. The reduced token usage therefore demonstrates a lower review workload, while the defect assessment remains uncertain.\n\nCompare Crop Review Separately\n\nThe earlier fabric_agent.py run in Option 2 provides another condition: the agent reviews selected region crops returned by view_defect.\n\nIts report.md, audit.json, and usage.json are already available in the timestamped results folder. Compare them separately with the full-image results above.\n\nThis comparison changes the image content, agent instructions, and tool-call sequence. Any difference cannot be attributed to filtering or cropping alone.\n\nCheck Review Quality and Total Workload\n\nFor a broader evaluation, use images with verified defect labels. Check whether the Workflow excluded any actual defects and whether the visual review missed or incorrectly reported defects. Record uncertain assessments, unreviewed regions, and processing errors separately.\n\nAn unselected image is not automatically defect-free. Compare unselected images with the verified labels and include any additional human review in the workload measurements.\n\nThe comparison script’s token counts and execution time cover its visual review run. They exclude the earlier Roboflow Workflow execution. A complete pipeline comparison must also include screening time and cost, model output usage, and the agent’s tool-call overhead.\n\nRepeat the evaluation on a larger labeled test set before drawing conclusions about accuracy, latency, or total cost.\n\nConclusion\n\nThe Roboflow Workflow gives the AI agent a computer vision tool for textile inspection. It runs object detection, selects images for review, and returns class labels, confidence scores, and bounding boxes. The agent uses these outputs to request visual analysis and generate an inspection report. It can delegate the review to a VLM tool or analyze selected region crops directly.\n\nUse an assistant in ChatGPT or Claude to call both the Workflow and the VLM review tool, or build a Python agent with the OpenAI Agents SDK and gpt-6-sol to call the Workflow and analyze returned region crops. Start with a small test set in Roboflow Workflows (opens in a new tab), validate the agent's visual assessments, and measure token usage and missed defects before processing larger batches.", "url": "https://wpnews.pro/news/how-to-use-a-small-vision-model-as-a-filter-for-an-ai-agent", "canonical_source": "https://roboflow.com/blog/how-to-use-a-small-vision-model-as-a-filter-for-an-ai-agent", "published_at": "2026-10-29 18:26:11+00:00", "updated_at": "2026-10-29 18:47:53.382499+00:00", "lang": "en", "topics": ["computer-vision", "ai-agents", "ai-tools", "artificial-intelligence"], "entities": ["Roboflow", "Roboflow Workflows", "Roboflow Universe", "RF-DETR Medium", "OpenAI Agents SDK", "ChatGPT", "Claude", "gpt-6-sol"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-use-a-small-vision-model-as-a-filter-for-an-ai-agent", "markdown": "https://wpnews.pro/news/how-to-use-a-small-vision-model-as-a-filter-for-an-ai-agent.md", "text": "https://wpnews.pro/news/how-to-use-a-small-vision-model-as-a-filter-for-an-ai-agent.txt", "jsonld": "https://wpnews.pro/news/how-to-use-a-small-vision-model-as-a-filter-for-an-ai-agent.jsonld"}}