{"slug": "multi-model-auto-labeling-for-segmentation-with-roboflow-workflows", "title": "Multi-Model Auto Labeling for Segmentation with Roboflow Workflows", "summary": "Roboflow released a multi-model auto-labeling workflow for segmentation that combines SAM 3, Google Gemini, and OpenAI GPT to produce pixel-perfect masks via a 2-of-3 consensus pipeline, reducing annotation time and errors for tasks like defect analysis and medical imaging.", "body_md": "*Roboflow's Workflows integration with Auto Label isn't just for bounding boxes. In this article, we build a multi-model consensus pipeline that produces pixel-perfect segmentation masks using SAM 3 prompted directly with text, plus two frontier VLMs (Google Gemini and OpenAI's GPT) whose detections are refined into masks by SAM 2. A rules-based consensus block only keeps masks that at least two of the three branches independently agree on, and we launch the whole thing serverlessly on a batch of unannotated images with just a few clicks.*\n\nIn [this post](https://blog.roboflow.com/multi-model-auto-labeling/), we showed how to combine multiple VLMs into a 2-of-3 consensus workflow for auto-labeling object detection datasets. However, bounding boxes only apply to object detection. For many production use cases (defect analysis, medical imaging, [robotics](https://roboflow.com/industries/robotics?ref=blog.roboflow.com), anything where the exact shape of an object matters), you need segmentation masks. Hand-drawing these masks is by far the slowest, most expensive form of annotation.\n\nThis is exactly where multi-model auto labeling is most efficient. If consensus labeling saves you time on boxes, it saves you an order of magnitude more on masks.\n\n## Multi-Model Segmentation Benefits\n\nThe macro benefit here is that you can have the same freedom as before. You're not locked into a single segmentation model or a single provider. But segmentation adds a twist, because the best mask generators and the best object finders are often different models entirely.\n\n[SAM-family models](https://blog.roboflow.com/segment-anything/) produce incredibly crisp masks, but they need to be told where to look. VLMs like [Gemini](https://blog.roboflow.com/gemini-2-5-object-detection-segmentation/) and [GPT](https://blog.roboflow.com/gpt-5-vision-multimodal-evaluation/) are excellent at finding and classifying objects, but their raw outputs are boxes, not pixels. With Workflows, you don't have to choose. You can chain them: let the VLMs find the objects, let SAM carve out the pixels, and let a text-promptable segmentation model like [SAM 3](https://blog.roboflow.com/what-is-sam3/) run as an independent third opinion.\n\nThe result is a pipeline where every mask in your dataset has been found by one model family, refined by another, and verified by a third, before a human ever sees it.\n\n## The Concept: Consensus on Masks\n\nWhen a single model segments your data, it has no safety net. If it hallucinates an object, or bleeds a mask into the background, that error goes straight into your training set.\n\nBy running three independent branches (SAM 3, Gemini + SAM 2, and GPT + SAM 2) we eliminate uncorrelated errors. SAM 3 might occasionally grab a background object that vaguely matches your text prompt, but it's highly unlikely that Gemini and GPT will independently box that same false positive at the same coordinates.\n\nSegmentation also gives the consensus block a new mask aggregation strength. When two or more branches agree on an object, we take the intersection of the agreeing masks, keeping only the pixels that multiple models independently claimed belong to the object. Hallucinated mask spillover gets trimmed away automatically, and what survives is the most conservative, highest-confidence mask possible.\n\n## Building the Multi-Model Segmentation Auto-Labeler\n\nLet's build it from scratch. If you already have a segmentation workflow in mind, you can skip ahead to the auto-labeling section. We'll build a live 2-of-3 mask consensus [workflow](https://app.roboflow.com/workflows/embed/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3JrZmxvd0lkIjoib3ZiSUpJbG1meHV4UzM4TEdObnEiLCJ3b3Jrc3BhY2VJZCI6ImVHM1R4bXRjTUlOSFNiTXhOQVgwNUxKTEtreDEiLCJ1c2VySWQiOiJlRzNUeG10Y01JTkhTYk14TkFYMDVMSkxLa3gxIiwiaWF0IjoxNzg0MjMxNzE5fQ.nQzA1DYBJDp0GbMyyZUL5lJqnWTl4EhxfeymyUAi-9I?ref=blog.roboflow.com) using safety helmets on construction sites as our target class without writing a single line of code.\n\n### Step 1: Access the Roboflow Platform\n\nInitialize the workspace by logging into your Roboflow account. If you are a new user, you can set up a free tier to begin building and organizing your custom vision projects.\n\n### Step 2: Create a New Workflow\n\nNavigate to the Workflows tab in your dashboard sidebar and click Create Workflow. Choose a blank workflow to open up the interactive node graph canvas.\n\n### Step 3: Configure Inputs & Class Setup\n\nJust like the detection version, add a centralized Detection Classes block (`set_detection_classes`\n\n) to your workflow. This is the single control point for your entire pipeline, and every branch downstream reads its target classes from here.\n\nFor this build, set the class to `helmet`\n\n. Since every model in this pipeline is open-vocabulary or promptable, pivoting the entire workflow from helmets to cracks on a weld seam is very simple. All three branches adapt instantly.\n\n### Step 4: Add SAM 3 as Your First Segmentation Branch\n\nDrop a [SAM 3 block](https://inference.roboflow.com/workflows/blocks/sam3/?ref=blog.roboflow.com) (`sam3_masks`\n\n) onto the canvas and wire it to your image input and your detection classes block. SAM 3 is text-promptable, so it takes your class names directly and returns polygon masks natively. Set the output format to polygons.\n\nThis branch is your segmentation expert since it goes straight from text to pixels with no intermediate steps.\n\n### Step 5: Add the VLM Branches\n\nNow add your two verifier branches. Drop in a [Google Gemini block](https://inference.roboflow.com/workflows/blocks/google_gemini/?ref=blog.roboflow.com) (`vlm_detector_a`\n\n, running [gemini-2.5-pro](https://playground.roboflow.com/models/google/gemini-2-5-pro?ref=blog.roboflow.com))\n\nAlso add an OpenAI block (`vlm_detector_b`\n\n, running [gpt-5.5](https://blog.roboflow.com/gpt-5-5-vision-benchmarks-use-cases/)).\n\nSet both to the object-detection task type and wire both to the same image input and detection classes block. All three foundation branches are now evaluating the exact same visual data against your exact target labels at the exact same time.\n\n### Step 6: Convert VLM Outputs and Upgrade Them to Masks\n\nRaw text and coordinate responses from multimodal models need to be structured into format-compliant computer vision data. Drop a [VLM As Detector block](https://inference.roboflow.com/workflows/blocks/vlm_as_detector/?ref=blog.roboflow.com) underneath each VLM to handle that natively.\n\nNext, since the VLMs only give you boxes, so we feed each branch's boxes into its own [Segment Anything 2 block](https://inference.roboflow.com/workflows/blocks/segment_anything2_model/?ref=blog.roboflow.com) (`vlm_a_sam2_masks`\n\nand `vlm_b_sam2_masks`\n\n, using the hiera_large checkpoint). SAM 2 takes each box as a prompt and carves out a pixel-accurate mask inside it.\n\n### Step 7: Route into Rules-Based Mask Consensus\n\nConnect all three mask outputs (SAM 3, Gemini + SAM 2, and GPT + SAM 2) into a single [Detections Consensus block](https://inference.roboflow.com/workflows/blocks/detections_consensus/?ref=blog.roboflow.com) (`verified_consensus`\n\n). Configure it like so:\n\n`required_votes`\n\n: 2, so a mask only survives if at least two of the three branches independently found the same object`class_aware`\n\n: enabled, so branches must agree on both location and class`iou_threshold`\n\n: 0.3, which controls how much overlap counts as agreement`detections_merge_mask_aggregation`\n\n: intersection, so the final mask keeps only the pixels the agreeing models both claimed\n\nIf a branch experiences a random hallucination, it won't get a second vote and gets automatically filtered out. And even for objects that pass the vote, the intersection aggregation trims any pixels only one model believed in.\n\n### Step 8: Add Visualization and Production Telemetry (Optional)\n\nFor manual debugging, you can chain a Mask Visualization block (set the opacity to around 0.5) into Bounding Box and Label Visualization blocks, plus a count block to track how many verified objects appear per image.\n\nI also wired in a Vision Events block, which logs every verified segmentation as a structured production event, complete with custom metadata like the workflow goal and target class. This gives you a queryable audit trail of everything your auto-labeler has done, which is useful once this pipeline is churning through thousands of images.\n\n### Step 9: Direct the Workflow Outputs\n\nJust like with detection, you can choose to have all of the outputs of just a select few.\n\n## How to Use Your Workflow in Auto Label\n\nOnce you've built and saved your workflow, launching it on an unannotated batch of images takes just a few clicks.\n\n### Step 1: Create your project and add images\n\nSign into Roboflow, create a new instance segmentation project or go to an existing one, and add the images you want to annotate.\n\n### Step 2: Choose Your Classes\n\nBecause we have a highly customizable workflow, simply go into the detection classes block and change it to your preferred target class. For this project, we'll keep it as `helmet`\n\n.\n\n### Step 3: Use your Workflow to Annotate your Images\n\nGo back to the [annotate](https://roboflow.com/annotate?ref=blog.roboflow.com) tab, select \"auto-label entire batch\", and select your workflow.\n\nEvery image in your batch comes back with verified, intersection-refined polygon masks, ready for human review or straight into a dataset version.\n\n**Cite this Post**\n\nUse the following entry to cite this post in your research:\n\n[Aarnav Shah](/author/aarnavshah/). (Jul 16, 2026).\nMulti-Model Auto Labeling for Segmentation with Roboflow Workflows. Roboflow Blog: https://blog.roboflow.com/multi-model-auto-labeling-for-segmentation/", "url": "https://wpnews.pro/news/multi-model-auto-labeling-for-segmentation-with-roboflow-workflows", "canonical_source": "https://blog.roboflow.com/multi-model-auto-labeling-for-segmentation/", "published_at": "2026-07-16 23:05:08+00:00", "updated_at": "2026-07-16 23:31:23.177608+00:00", "lang": "en", "topics": ["computer-vision", "ai-tools", "ai-products", "machine-learning"], "entities": ["Roboflow", "SAM 3", "Google Gemini", "OpenAI GPT", "SAM 2"], "alternates": {"html": "https://wpnews.pro/news/multi-model-auto-labeling-for-segmentation-with-roboflow-workflows", "markdown": "https://wpnews.pro/news/multi-model-auto-labeling-for-segmentation-with-roboflow-workflows.md", "text": "https://wpnews.pro/news/multi-model-auto-labeling-for-segmentation-with-roboflow-workflows.txt", "jsonld": "https://wpnews.pro/news/multi-model-auto-labeling-for-segmentation-with-roboflow-workflows.jsonld"}}