{"slug": "how-to-run-inference-in-codex-with-the-roboflow-plugin", "title": "How to Run Inference in Codex with the Roboflow Plugin", "summary": "Roboflow published a tutorial on running inference inside OpenAI's Codex coding agent using the Roboflow plugin, which lets Codex locate a trained model in a Roboflow workspace or on Roboflow Universe, run detection or segmentation on images, and visualize results from natural-language prompts. The walkthrough covers installing the plugin through the Codex app or the Codex CLI's /plugins browser, signing in to Roboflow without pasting an API key, and confirming the connection with a test prompt, then runs three examples: ceramic tile defect inspection with RF-DETR Small, steel sheet surface defect detection with RF-DETR Object Detection, and pallet segmentation with RF-DETR Instance Segmentation.", "body_md": "Run inference in Codex with the Roboflow plugin: find a trained model, run detection or segmentation on your images, and visualize the results.\n\nRunning inference means applying a trained model to an image to generate predictions. Codex is OpenAI’s coding agent: you describe a task in plain language, and it writes and runs the code for you. With the Roboflow plugin installed, Codex can find a trained model in your Roboflow workspace or on Roboflow Universe, run inference, and visualize the results, all from natural-language prompts.\n\nWith the Roboflow plugin in Codex, you can select a trained model, run inference, and visualize the results using natural-language prompts.\n\nIn this tutorial, we will start with a simple example using an existing object detection model from a Roboflow workspace. We will then explore two RF-DETR examples with more detailed visualizations and analytics.\n\nTutorial and model source\n\nApplication\n\nModel\n\nOutput\n\n1. Manufacturing inspection\nOwn Roboflow workspace\n\nCeramic tile defect inspection\n\nRF-DETR Small\n\nPrediction table and annotated image\n\n2. Manufacturing inspection\nRoboflow Universe\n\nSteel sheet surface defect detection\n\nRF-DETR Object Detection\n\nBounding boxes and defect analytics\n\n3. Warehouse inspection\nRoboflow Universe\n\nPallet segmentation\n\nRF-DETR Instance Segmentation\n\nInstance masks and area analytics\n\nBefore starting, open Codex with the Roboflow plugin installed and connected. Keep an original test image ready for each example.\n\nHow to Install and Connect the Roboflow Plugin in Codex\n\nThe Roboflow plugin connects Codex to your Roboflow account, so Codex can access your projects, trained models, and public models on Roboflow Universe. Setup takes a few minutes.\n\nStep 1: Add the Roboflow plugin\n\nThere are two ways to install the Roboflow plugin:\n\nCodex App\n\nCodex CLI\n\nOpen the codex app and on the left pane click the Plugins icon then in search plugins text field type \"Roboflow\". You will see a Roboflow plugin as a result. Click on it.\n\nAlternatively to install it via Codex CLI, open your terminal and run codex. Inside Codex, enter /plugins to open the plugin browser. Find the Roboflow plugin by typing \"Roboflow\" and hit enter to install it. If it is installed but disabled, highlight it and press Space to enable it. Complete any connection prompts, then start a new Codex session before using the plugin.\n\nStep 2: Sign in to Roboflow\n\nThe first time the plugin calls Roboflow, Codex asks you to sign in with your Roboflow account. Approve the requested permissions, and you’re connected. You don’t need to create or paste an API key.\n\nIf you’re running Codex somewhere a browser sign-in isn’t possible, such as a headless server, you can connect with a Roboflow API key instead. The plugin’s GitHub repository explains how to set this up. Never paste a private API key into the chat.\n\nStep 3: Confirm the connection\n\nRun a quick test prompt:\n\n```\n@Roboflow List the projects in my workspace.\n```\n\nIf Codex returns your projects, the plugin is ready. Keep an original test image ready for each example below.\n\nHow to Run Inference from Codex with Roboflow\n\nWe will begin with a ceramic tile defect inspection model already available in my Roboflow workspace.\n\nTutorial 1: Run Inference Using an Existing Object Detection Project\n\nThis example follows three steps:\n\nfind a project with a trained model,\n\nrun inference, and\n\ndisplay the annotated image.\n\nStep 1: Find your projects and trained models\n\nAsk the plugin to list object detection projects that already have trained models. Organizing the results by project makes it easier to compare the available models and select an exact model ID.\n\n```\n@Roboflow List the object detection projects in my workspace that have at least one trained model.\n\nPresent the results directly in this chat using clear Markdown formatting.\n\nStart with a short summary showing:\n- Workspace name\n- Number of matching projects\n- Total number of trained models found\n\nThen organize the results into a separate section for each project:\n\n### [Project name — clickable project link]\n- Project ID: `exact project ID`\n- Project type: Object Detection\n- Project image count: number of images\n\nUnder each project, show its trained models in a compact table:\n\n| Model name | Exact model ID | Model architecture |\n|------------|----------------|--------------------|\n\nFor model architecture, include the model family and size where available, such as RF-DETR Small or YOLOv8s.\n\nInclude all trained models under their corresponding project. Distinguish the project image count from dataset-version image counts.\n\nUse exact IDs returned by Roboflow. Mark unavailable information as “Not available.”\n\nKeep everything readable within the chat; do not create a separate file or dashboard.\n```\n\nThe response groups trained models under their corresponding projects, with project details and clickable links. You should see something like this:\n\nSelect a model suitable for your test image. For this example, we will use the RF-DETR small model ceramic-tile-defects-v7pui-1-rfdetr-small-t1 from the ceramic-tile-defects project. When using your own project, copy the exact model ID returned by the plugin.\n\nStep 2: Attach an image and run inference\n\nAttach the original image you want to inspect and then specify the selected model ID in your prompt as follows.\n\n```\n@Roboflow Use ceramic-tile-defects-v7pui-1-rfdetr-small-t1 to run inference on the attached image with a confidence threshold of 0.40.\n\nShow the detected classes and total detection count. Include a table with each detection's number, confidence score, and bounding box center coordinates in pixels.\n\nAlso report the confidence threshold and model inference time, if available.\n```\n\nI have attached the following input image with the prompt,\n\nand received the following response:\n\nThe response reported a 40% confidence threshold and 0.760 seconds model inference time with total 19 detections for different defect classes.\n\nThese values describe this particular run. Predictions and inference time can differ for another image or model.\n\nStep 3: Display the annotated image\n\nThe prediction table tells you what the model detected. To inspect where those detections appear, ask Codex to draw them on the original image.\n\n```\n@Roboflow Show the annotated image here using the predictions from the previous inference run.\n\nDraw the returned bounding boxes on the original image. Use a different color for each detection and number the boxes to match the prediction table.\n\nAdd a compact dark legend in the top-right corner showing each detection's number, class name, and confidence score. Include the model ID below the legend.\n\nRender and display the finished image directly in the conversation using the existing predictions.\n```\n\nYou will see the following output image:\n\nYou have now selected an existing trained model, run inference, and inspected its predictions visually using prompts.\n\nTutorial 2: Run RF-DETR Object Detection for Manufacturing\n\nNext, we will inspect manufactured steel sheet surfaces for defects such as scratches and stains. We will use a trained RF-DETR object detection model and ask Codex to create an annotated image with a visual analytics report.\n\nStep 1: Find a trained RF-DETR detection model\n\nAsk the plugin to find models that match both the manufacturing application and the required architecture.\n\n```\n@Roboflow Find up to 3 public trained RF-DETR object detection models on Roboflow Universe for manufacturing surface-defect inspection, preferably including scratches or stains.\n\nFor each candidate, show:\n- Project name and Universe link\n- Verified model architecture and task\n- Exact inference model ID\n- Classes supported by that trained model\n\nRecommend a suitable model for a demonstration. Verify the selected model's architecture from its model or deployment information, rather than relying only on project tags.\n\nIf you cannot verify a detail, mark it as unverified.\n```\n\nI received the following response:\n\nReview the classes and select a model that matches your inspection task. If you already have a trained RF-DETR manufacturing model, provide its project link and ask the plugin to check the same details.\n\nKeep the exact model ID for Step 3. A project can contain multiple architectures, so verify the selected trained model before continuing.\n\nStep 2: Provide a manufacturing test image\n\nAttach an original image of the surface or component you want to inspect. Choose an image with a similar material, camera distance, and defect appearance to the selected model’s examples. Use an image without existing annotation overlays so the new predictions remain easy to review. You can also provide the full path to an image in a folder Codex can access. I'll be using the following image:\n\nStep 3: Run inference through the plugin\n\nUse the following prompt. Attach the image and specify the verified model ID from Step 1. I am using metal-defect-detection-d3wl6/11.\n\n```\n@Roboflow Run object detection on the attached image using the verified RF-DETR model metal-defect-detection-d3wl6/11.\n\nUse the Roboflow plugin's direct model inference capability without creating a Workflow.\n\nUse a confidence threshold of 0.40 as an initial setting.\n\nFirst confirm that you can read the image. Then run inference and:\n- Save the complete returned predictions as manufacturing-predictions.json.\n- Preserve the model's original class names and bounding box coordinates.\n- Show the model ID, image dimensions, threshold, total detections, and counts by class.\n\nIf the image cannot be accessed or inference fails, report the problem instead of generating example results.\n```\n\nThe response should summarize the predictions returned by the model. The confidence threshold determines which predictions are retained. Here, 0.40 is an initial setting for exploration; the appropriate threshold depends on your model and inspection requirements.\n\nStep 4: Visualize the predictions and defect analytics\n\nAsk Codex to draw the returned boxes and create a report from the same predictions.\n\n```\n@Roboflow Use the same image and manufacturing-predictions.json to create a visual inspection report.\n\nUse the Roboflow drawing skill and Python Supervision to render the actual predictions. Write and execute the necessary code, then show the finished visualization.\n\nInclude:\n1. The original and annotated images side by side.\n2. Bounding boxes with class names and confidence scores.\n3. A consistent color for each defect class and a readable legend.\n4. Summary cards for total detections, detected classes, and the confidence threshold.\n5. A bar chart showing detection counts by class.\n6. A table listing each detection's ID, class, confidence, and bounding box coordinates.\n\nUse a clean gray layout with clear typography and enough space around the images and charts.\n\nAlign the boxes correctly with the original image dimensions. Derive every count and chart from the same predictions shown on the image.\n\nSave the annotated image as manufacturing-annotated.png, the full report as manufacturing-report.png, and the detection table as manufacturing-detections.csv.\n\nDisplay the annotated image and report, and provide download links. If there are no detections, show that clearly. Do not describe confidence scores as model accuracy.\n```\n\nYou will receive the following response.\n\nClick an image to enlarge it. The output of the report will be similar to this:\n\nReview the annotated image for misplaced boxes, duplicate predictions, missed defects, and incorrect labels. The chart summarizes the predicted defect instances in this image. It does not establish a production defect rate. You now have an annotated manufacturing image, a report, and prediction files that you can inspect or compare with later runs.\n\nTutorial 3: Run RF-DETR Instance Segmentation for a Warehouse\n\nFor the warehouse example, we will identify individual pallets and visualize their predicted shapes. Instance segmentation produces a separate mask for each detected object. This lets us inspect pallet boundaries and distinguish neighboring instances more precisely than a bounding box alone.\n\nStep 1: Find a trained RF-DETR segmentation model\n\nAsk the plugin specifically for an RF-DETR Instance Segmentation model trained on pallets.\n\n```\n@Roboflow Find up to 3 public trained RF-DETR instance segmentation models on Roboflow Universe for warehouse pallets.\n\nFor each candidate, show:\n- Project name and Universe link\n- Verified model architecture and task\n- Exact inference model ID\n- Classes supported by that trained model\n\nRecommend a suitable model for a demonstration.\n\nVerify that the selected model is an RF-DETR segmentation model that returns instance masks or polygons. Check the selected model's deployment information, because a project may contain several model architectures.\n\nIf you cannot verify a detail, mark it as unverified.\n```\n\nI received the following response:\n\nChoose a model whose examples resemble your warehouse images. If you already have a trained RF-DETR segmentation project, provide its link and ask the plugin to identify the available model.\n\nContinue once the architecture, task, and exact inference ID are confirmed. The model uses its trained classes; mentioning additional objects in a prompt does not add new classes to it.\n\nStep 2: Provide a warehouse test image\n\nAttach an original warehouse image containing pallets. Choose a view where pallet boundaries are reasonably visible. An image containing several pallets can help you inspect whether the model separates neighboring instances. You can also provide the full path to an image Codex can access. I'm using the following image.\n\nStep 3: Run instance segmentation through the plugin\n\nNow use the model ID from Step 1, attach the image from step 2, and run the following prompt.\n\n```\n@Roboflow Run instance segmentation on the attached image using the verified RF-DETR segmentation model pallet-nwpgy/1.\n\nUse the Roboflow plugin's direct model inference capability without creating a Workflow.\n\nUse a confidence threshold of 0.40 as an initial setting.\n\nFirst confirm that you can read the image. Then run inference and:\n- Save the complete returned predictions as warehouse-predictions.json.\n- Preserve all returned mask or polygon data, class names, and confidence scores.\n- Show the model ID, image dimensions, threshold, and instance counts by class.\n- Confirm whether usable segmentation masks or polygons were returned.\n\nKeep the full segmentation data in the saved file rather than printing every coordinate in the chat. If masks or polygons are missing, report that clearly.\n```\n\nYou will see output similar to the following.\n\nThe response should identify the detected instances and confirm that segmentation data is available. Keep the complete prediction file. Codex will use the returned masks or polygons to draw the overlay and calculate image-based measurements.\n\nStep 4: Visualize the masks and warehouse analytics\n\nAsk Codex to create a report that connects each colored mask to its corresponding measurements. To visualize the results, use the following prompt:\n\n```\n@Roboflow Use the same image and warehouse-predictions.json to create a warehouse segmentation report.\n\nUse the Roboflow drawing skill and Python Supervision to render the returned instance masks or polygons. Write and execute the necessary code, then show the finished visualization.\n\nInclude:\n1. The original and segmented images side by side.\n2. A distinct color for each instance, with a transparent mask fill and a clear outline.\n3. Labels containing an instance ID, the original class name, and confidence.\n4. Summary cards for total instances, confidence threshold, and combined predicted mask coverage.\n5. A bar chart showing mask area in pixels for each instance.\n6. A table showing instance ID, class, confidence, mask area in pixels, and mask area as a percentage of the image.\n\nUse the same instance IDs and colors across the image, chart, and table. Match the clean gray layout of the manufacturing report.\n\nCalculate areas from the actual masks at the original image dimensions. Calculate combined coverage from the union of the masks so overlapping pixels are counted once.\n\nLabel coverage as \"percentage of image pixels covered by predicted masks.\" Do not describe it as warehouse floor occupancy or a physical area measurement.\n\nSave the segmented image as warehouse-segmented.png, the full report as warehouse-report.png, and the instance table as warehouse-instances.csv.\n\nDisplay the segmented image and report, and provide download links. If segmentation data is missing, explain the issue; do not substitute filled bounding boxes for masks.\n```\n\nYou will get the following output.\n\nClick on the report image to enlarge and view.\n\nReview whether each mask follows the visible pallet boundary and whether adjacent pallets remain separate. Look for masks that extend into the floor, include surrounding objects, or miss part of a pallet.\n\nThe area chart compares predicted masks within this image. Camera perspective affects their apparent size, so a larger mask does not necessarily represent a physically larger pallet.\n\nYou now have a warehouse segmentation overlay and an analytics report built from the same predictions, with matching instance IDs for easy review.\n\nDeploy Your Model After Testing in Codex\n\nRunning inference in Codex is a fast way to check how a model performs on your images. Once you’re happy with the results, the next step is to run the model in production, on live images or video from your application or cameras. Roboflow gives you two ways to get there.\n\nOption 1: Call the model directly from your application\n\nAsk the plugin to write the integration code for the model you tested:\n\n```\n@Roboflow Write Python code that sends an image to metal-defect-detection-d3wl6/11 using Roboflow's hosted inference API and prints the predicted classes and confidence scores. Read the API key from an environment variable.\n```\n\nCodex returns code you can add to your application, using the same model ID from your tests.\n\nOption 2: Build and deploy a Workflow\n\nIf you need more than raw predictions, such as filtering by class, counting objects, drawing results, or sending alerts, build a Roboflow Workflow. A Workflow chains your model with other processing steps in a single pipeline. You can describe the Workflow you need to Codex, or build it in the Workflow Editor. When it’s ready, publish it, then click </> Use in the Workflow Editor to get deployment code for images, video files, webcams, and RTSP camera streams.\n\nChoose where your model runs:\n\nIn the cloud: The Serverless Hosted API runs your model or Workflow on Roboflow’s infrastructure, so you don’t need to provision or manage a GPU.\n\nOn your own hardware: Run the same model or Workflow on your own machines with Roboflow Inference, including edge devices like NVIDIA Jetson and Raspberry Pi. This works well when images need to stay on-site or internet access is limited. Self-hosted inference is free on every Roboflow plan, including the free tier.\n\nFor deployments with many cameras or devices, Roboflow’s Deployment Manager lets you monitor devices, streams, and logs from one place.\n\nRun Inference in Codex with the Roboflow Plugin\n\nThe Roboflow plugin provides a straightforward way to run inference from Codex using natural-language prompts. You can begin with an existing project, select a trained model, and display its predictions directly on an image.\n\nThe RF-DETR examples extend this process with manufacturing defect analytics and warehouse segmentation reports. Codex handles the visualization code, while you specify the model, image, and output you need.\n\nStart by reviewing individual predictions visually, then evaluate the model on representative labeled images before relying on it for an operational task.\n\nTry inference with Codex on your own project today.", "url": "https://wpnews.pro/news/how-to-run-inference-in-codex-with-the-roboflow-plugin", "canonical_source": "https://roboflow.com/blog/run-inference-in-codex-with-roboflow", "published_at": "2026-10-19 19:16:20+00:00", "updated_at": "2026-10-19 19:45:14.062055+00:00", "lang": "en", "topics": ["computer-vision", "ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Roboflow", "OpenAI", "Codex", "Roboflow Universe", "RF-DETR", "RF-DETR Small", "RF-DETR Object Detection", "RF-DETR Instance Segmentation"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-run-inference-in-codex-with-the-roboflow-plugin", "markdown": "https://wpnews.pro/news/how-to-run-inference-in-codex-with-the-roboflow-plugin.md", "text": "https://wpnews.pro/news/how-to-run-inference-in-codex-with-the-roboflow-plugin.txt", "jsonld": "https://wpnews.pro/news/how-to-run-inference-in-codex-with-the-roboflow-plugin.jsonld"}}