# GPT-6 Astra for Segmentation

> Source: <https://blog.roboflow.com/gpt-6-astra-for-segmentation/>
> Published: 2026-09-17 18:45:11+00:00

GPT-6 Astra is the best vision model we have tested. As of September 17, 2026 it's #1 our [Vision Evals](https://playground.roboflow.com/models/openai/gpt-6-astra?ref=blog.roboflow.com) overall and on object detection as well. It turns out it can also outline objects. You can prompt Astra to output polygons and it returns one per object, with no expert segmentation model involved. 

This post shows what those polygons look like, where they fall short of a real segmentation model, and how to pair GPT-6 Astra with [SAM 3](https://blog.roboflow.com/what-is-sam3/) for best accuracy.

## GPT-6 Astra for Vision

OpenAI released [GPT-6 Astra](https://openai.com/index/gpt-6-astra/?ref=blog.roboflow.com) in early September. The launch post is about computer use and coding, but the vision side is where it surprised us. We ran it through our vision evals and it is the best vision model we have seen. It notices tiny details, reads text in context and also draws precise boxes. See this [detection deep dive](https://x.com/skalskip92/status/2096317264463012261?ref=blog.roboflow.com) on X.

Two days later we also published a [segmentation deep dive](https://x.com/skalskip92/status/2096994940530094145?ref=blog.roboflow.com) about how Astra's polygon output performs in real world. TL;DR: it's impressive.

Note that OpenAI does not document this anywhere. The [model page](https://developers.openai.com/api/docs/models/gpt-6-astra?ref=blog.roboflow.com) lists image input and structured outputs, but that's all you need: a strict JSON schema and a prompt asking it to output boxes/polygons in specific format.

## GPT-6 Astra for Segmentation

We send the image at up to 2048 px on the long edge, the same upload rule as our Vision Evals, and ask for a flat list of pixel vertices per instance. The request looks something like this:

```
{
  "model": "gpt-6-astra",
  "reasoning": {"effort": "high"},
  "input": [{"role": "user", "content": [
    {"type": "input_text", "text": "Segment every instance of \"sweet potato\" in this image. The image is 2048 px wide and 1365 px tall. Coordinates are pixels with (0, 0) at the top-left corner. For each distinct instance, return a polygon tracing its outline as tightly as you can, as a flat list [x1, y1, x2, y2, ...] of at least 3 vertices, together with a confidence in [0, 1]. Use as many vertices as the shape needs. Return one entry per instance; do not merge separate instances."},
    {"type": "input_image", "image_url": "data:image/jpeg;base64,..."}
  ]}],
  "text": {"format": {"type": "json_schema", "strict": true, "schema": {
    "instances": [{"polygon": ["number"], "confidence": "number"}]
  }}}
}
```

Astra returns something like `{"polygon": [1339, 1385, 1362, 1391, ...], "confidence": 0.92}` per instance. Scale the vertices back to the original resolution and rasterize, and you have an instance mask.

If you zoom in, you can see that segmentation results aren't pixel-perfect, as Astra is outputting polygons, not dense segmentation masks (comparison below).

## GPT-6 Astra + Segment Anything 3 (SAM3)

To get better accuracy, we can use two models and let each model do what they're best at. Astra finds the boxes and SAM3 draws the mask:

And here's the difference between Astra's polygon output and Astra+SAM3 output:

### Segment multiple classes with Astra

Compared to SAM3, Astra handles look-alike classes well. Below we asked for "cashew" and "hazelnut" in a single request on a tray of mixed nuts. Astra recognizes them correctly:

Astra recognizes all nuts correctly, but the polygons aren't pixel-perfect.

We send SAM3 the full image plus every Astra box as a *box prompt*. Each box means "segment the object inside this box", so SAM 3 returns exactly one mask per box and never has to decide what a cashew is: the class comes with the box. One Astra call, one SAM 3 call, nothing to filter afterwards.

```
curl -X POST "https://serverless.roboflow.com/sam3/visual_segment?api_key=$ROBOFLOW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "image": {"type": "base64", "value": "<base64 jpeg>"},
    "prompts": {"prompts": [
      {"box": {"x": 412, "y": 265, "width": 180, "height": 96}},
      {"box": {"x": 640, "y": 310, "width": 110, "height": 104}}
    ]},
    "multimask_output": false,
    "format": "rle"
  }'

# one prompt per Astra box; x, y is the box CENTRE in pixels
# response: "predictions" in the same order as the prompts, one mask each
# {"predictions": [{"masks": {"counts": "...", "size": [4480, 6720]}, "confidence": 0.98}, ...]}
```

Masks come back in the same order as the boxes, so box *i* has same class as mask *i*.

SAM 3 can also take boxes as *exemplars* ("find everything that looks like these"). We tried that too, and on look-alike classes it bleeds: three hazelnut boxes as exemplars returned 97 masks, so it segmented all nuts as "hazelnuts". Workaround was to use an overlap filter against Astra's boxes to get back to the right count.

Box prompts give that answer directly, so that is the route we use. In [Roboflow Workflows](https://roboflow.com/workflows/build?ref=blog.roboflow.com) it is two blocks: the OpenAI block running object detection, and passing predictions to SAM3 interactive block.

## Astra for segmentation costs

All numbers below are for the mixed nut image above (1920 × 1280). Astra is billed per tokens ($10/MTok input, $50/MTok output). SAM 3 on Roboflow's [Serverless API](https://docs.roboflow.com/deployment/roboflow-cloud/serverless-api?ref=blog.roboflow.com) is billed by compute time, 500 seconds per credit, and a credit is $4 prepaid, so roughly $0.008 per second:

| Approach | Astra tokens in / out | SAM 3 compute/cost | Cost per image | Time | 
|---|---|---|---|---|
| Astra polygons (low) | 3.0K / 9.0K | – | **$0.48** | 202 s | 
| Astra polygons (medium)* | 3.0K / 11.5K | – | **$0.61** | 261 s | 
| SAM 3 only ("cashew") | – | 1.0 s / $0.01 | **$0.01** | 2 s | 
| Astra boxes (low) + SAM3 | 3.0K / 1.9K | 1.2 s / $0.01 | **$0.14** | 37 s | 
| Astra boxes (high) + SAM3 | 3.0K / 10.5K | 1.4 s / $0.01 | **$0.56** | 237 s | 

Three things stand out:

- Polygons are expensive because every vertex is an output token
- SAM3 alone is the cheapest row but the wrong one for this image: prompting with "cashew" returns masks covering both kinds of nut and "hazelnut" returns nothing.
- Using Astra with SAM3 offers a more accurate and lower cost solution

## When to Use GPT-6 Astra for Segmentation

Compared to SAM3 alone, Astra performs much better on more complex objects, especially ones that require multiple words, and look-alike objects. The polygon outlines are usually good enough for counting, cropping, and labeling.

For pixel-accurate masks, use Astra with SAM3. Astra proposes boxes, SAM3 draws the masks, and you get Astra's understanding with SAM3's edges at lower cost per image, because boxes are less complex than polygon lines, less Astra's output tokens.

For labeling, Roboflow [Auto Label](https://blog.roboflow.com/auto-label-with-gpt-6-astra/) supports all options, SAM3, Astra, and Astra + SAM3. After Auto Label finishes, you can train an [RF-DETR](https://rfdetr.roboflow.com/latest/?ref=blog.roboflow.com) real-time segmentation model on the dataset.

Compare GPT-6 Astra against every model we have benchmarked, and test it on your own images, on [Roboflow Playground](https://playground.roboflow.com/?ref=blog.roboflow.com).

### **Cite this Post**

Use the following entry to cite this post in your research:

[Erik Kokalj](https://blog.roboflow.com/author/erik/). (Sep 17, 2026).
      GPT-6 Astra for Segmentation. Roboflow Blog: https://blog.roboflow.com/gpt-6-astra-for-segmentation/