GPT-6 Astra for Segmentation

Roboflow reported that OpenAI's GPT-6 Astra, released in early September 2026, ranks #1 on its Vision Evals overall and on object detection as of September 17, 2026, and can output per-instance polygons for segmentation when prompted with a strict JSON schema. Roboflow said Astra's polygon output is not pixel-perfect and recommends pairing it with SAM 3, sending Astra's boxes as box prompts so SAM 3 returns one mask per box, to improve accuracy. OpenAI does not document the segmentation capability; its model page lists only image input and structured outputs.

GPT-6 Astra is the best vision model we have tested. As of September 17, 2026 it's 1 our Vision Evals https://playground.roboflow.com/models/openai/gpt-6-astra?ref=blog.roboflow.com overall and on object detection as well. It turns out it can also outline objects. You can prompt Astra to output polygons and it returns one per object, with no expert segmentation model involved. This post shows what those polygons look like, where they fall short of a real segmentation model, and how to pair GPT-6 Astra with SAM 3 https://blog.roboflow.com/what-is-sam3/ for best accuracy. GPT-6 Astra for Vision OpenAI released GPT-6 Astra https://openai.com/index/gpt-6-astra/?ref=blog.roboflow.com in early September. The launch post is about computer use and coding, but the vision side is where it surprised us. We ran it through our vision evals and it is the best vision model we have seen. It notices tiny details, reads text in context and also draws precise boxes. See this detection deep dive https://x.com/skalskip92/status/2096317264463012261?ref=blog.roboflow.com on X. Two days later we also published a segmentation deep dive https://x.com/skalskip92/status/2096994940530094145?ref=blog.roboflow.com about how Astra's polygon output performs in real world. TL;DR: it's impressive. Note that OpenAI does not document this anywhere. The model page https://developers.openai.com/api/docs/models/gpt-6-astra?ref=blog.roboflow.com lists image input and structured outputs, but that's all you need: a strict JSON schema and a prompt asking it to output boxes/polygons in specific format. GPT-6 Astra for Segmentation We send the image at up to 2048 px on the long edge, the same upload rule as our Vision Evals, and ask for a flat list of pixel vertices per instance. The request looks something like this: { "model": "gpt-6-astra", "reasoning": {"effort": "high"}, "input": {"role": "user", "content": {"type": "input text", "text": "Segment every instance of \"sweet potato\" in this image. The image is 2048 px wide and 1365 px tall. Coordinates are pixels with 0, 0 at the top-left corner. For each distinct instance, return a polygon tracing its outline as tightly as you can, as a flat list x1, y1, x2, y2, ... of at least 3 vertices, together with a confidence in 0, 1 . Use as many vertices as the shape needs. Return one entry per instance; do not merge separate instances."}, {"type": "input image", "image url": "data:image/jpeg;base64,..."} } , "text": {"format": {"type": "json schema", "strict": true, "schema": { "instances": {"polygon": "number" , "confidence": "number"} }}} } Astra returns something like {"polygon": 1339, 1385, 1362, 1391, ... , "confidence": 0.92} per instance. Scale the vertices back to the original resolution and rasterize, and you have an instance mask. If you zoom in, you can see that segmentation results aren't pixel-perfect, as Astra is outputting polygons, not dense segmentation masks comparison below . GPT-6 Astra + Segment Anything 3 SAM3 To get better accuracy, we can use two models and let each model do what they're best at. Astra finds the boxes and SAM3 draws the mask: And here's the difference between Astra's polygon output and Astra+SAM3 output: Segment multiple classes with Astra Compared to SAM3, Astra handles look-alike classes well. Below we asked for "cashew" and "hazelnut" in a single request on a tray of mixed nuts. Astra recognizes them correctly: Astra recognizes all nuts correctly, but the polygons aren't pixel-perfect. We send SAM3 the full image plus every Astra box as a box prompt . Each box means "segment the object inside this box", so SAM 3 returns exactly one mask per box and never has to decide what a cashew is: the class comes with the box. One Astra call, one SAM 3 call, nothing to filter afterwards. curl -X POST "https://serverless.roboflow.com/sam3/visual segment?api key=$ROBOFLOW API KEY" \ -H "Content-Type: application/json" \ -d '{ "image": {"type": "base64", "value": "<base64 jpeg "}, "prompts": {"prompts": {"box": {"x": 412, "y": 265, "width": 180, "height": 96}}, {"box": {"x": 640, "y": 310, "width": 110, "height": 104}} }, "multimask output": false, "format": "rle" }' one prompt per Astra box; x, y is the box CENTRE in pixels response: "predictions" in the same order as the prompts, one mask each {"predictions": {"masks": {"counts": "...", "size": 4480, 6720 }, "confidence": 0.98}, ... } Masks come back in the same order as the boxes, so box i has same class as mask i . SAM 3 can also take boxes as exemplars "find everything that looks like these" . We tried that too, and on look-alike classes it bleeds: three hazelnut boxes as exemplars returned 97 masks, so it segmented all nuts as "hazelnuts". Workaround was to use an overlap filter against Astra's boxes to get back to the right count. Box prompts give that answer directly, so that is the route we use. In Roboflow Workflows https://roboflow.com/workflows/build?ref=blog.roboflow.com it is two blocks: the OpenAI block running object detection, and passing predictions to SAM3 interactive block. Astra for segmentation costs All numbers below are for the mixed nut image above 1920 × 1280 . Astra is billed per tokens $10/MTok input, $50/MTok output . SAM 3 on Roboflow's Serverless API https://docs.roboflow.com/deployment/roboflow-cloud/serverless-api?ref=blog.roboflow.com is billed by compute time, 500 seconds per credit, and a credit is $4 prepaid, so roughly $0.008 per second: | Approach | Astra tokens in / out | SAM 3 compute/cost | Cost per image | Time | |---|---|---|---|---| | Astra polygons low | 3.0K / 9.0K | – | $0.48 | 202 s | | Astra polygons medium | 3.0K / 11.5K | – | $0.61 | 261 s | | SAM 3 only "cashew" | – | 1.0 s / $0.01 | $0.01 | 2 s | | Astra boxes low + SAM3 | 3.0K / 1.9K | 1.2 s / $0.01 | $0.14 | 37 s | | Astra boxes high + SAM3 | 3.0K / 10.5K | 1.4 s / $0.01 | $0.56 | 237 s | Three things stand out: - Polygons are expensive because every vertex is an output token - SAM3 alone is the cheapest row but the wrong one for this image: prompting with "cashew" returns masks covering both kinds of nut and "hazelnut" returns nothing. - Using Astra with SAM3 offers a more accurate and lower cost solution When to Use GPT-6 Astra for Segmentation Compared to SAM3 alone, Astra performs much better on more complex objects, especially ones that require multiple words, and look-alike objects. The polygon outlines are usually good enough for counting, cropping, and labeling. For pixel-accurate masks, use Astra with SAM3. Astra proposes boxes, SAM3 draws the masks, and you get Astra's understanding with SAM3's edges at lower cost per image, because boxes are less complex than polygon lines, less Astra's output tokens. For labeling, Roboflow Auto Label https://blog.roboflow.com/auto-label-with-gpt-6-astra/ supports all options, SAM3, Astra, and Astra + SAM3. After Auto Label finishes, you can train an RF-DETR https://rfdetr.roboflow.com/latest/?ref=blog.roboflow.com real-time segmentation model on the dataset. Compare GPT-6 Astra against every model we have benchmarked, and test it on your own images, on Roboflow Playground https://playground.roboflow.com/?ref=blog.roboflow.com . Cite this Post Use the following entry to cite this post in your research: Erik Kokalj https://blog.roboflow.com/author/erik/ . Sep 17, 2026 . GPT-6 Astra for Segmentation. Roboflow Blog: https://blog.roboflow.com/gpt-6-astra-for-segmentation/