Understanding Video with GPT-6 Astra

GPT-6 Astra accepts text and images but rejects mp4 video input through its API, so developers are working around the limitation by sending sequences of timestamped frames as input_image parts alongside timestamps in the prompt. Roboflow's testing called GPT-6 Astra the best vision model it has seen on detection benchmarks, and a 100-frame request at 768x432 costs roughly $1 and takes two to three minutes to process at reasoning effort high. In a tennis analytics demo at 5 frames per second, the model found all 7 serves and 17 hits at every rate from 1 to 5 fps, but missed a point below 4 fps because it lacked a frame to judge whether the ball landed in the court.

GPT-6 Astra is really good at vision, but doesn't support video inputs. Send it a sequence of timestamped frames in a request, ask for a strict JSON output, and it can accurately do action recognition by providing timestamps of events in the clip. We tried it on sports analytics, process monitoring and logistics turnaround time. Every video below is Astra's output drawn back onto the original video. GPT-6 Astra for Vision GPT-6 Astra takes text and images. The API rejects an mp4, whether sent as a video part or as a file. So the question is how much of a video you can push through the image path. The vision capabilities itself are very strong. Roboflow tested it on detection benchmarks and calls it the best vision model he has seen. See thread here https://x.com/skalskip92/status/2096317264463012261?s=20&ref=blog.roboflow.com . Sending Video to a Model that only sees Images The workaround is to send a sequence of frames at a fixed rate via input image along with their timestamps in the prompt input text itself. For the examples below we forced the output to be JSON with event type enums and timestamps, then we overlayed the status on original video. Reasoning effort high helped on every clip. A 100 frame request takes two to three minutes and costs about $1 at $12.50/MTok input, $50/MTok output . Demo 1: Sports Analytics A fixed camera behind the baseline with two players and scoreboard blacked out so Astra has to infer the score from play. We used 5FPS, and asked the model to return event type, timestamp and player id shirt color . Request prompt looked something like this: { "model": "gpt-6-astra", "reasoning": {"effort": "high"}, "input": {"role": "user", "content": {"type": "input text", "text": "Fixed camera behind the baseline. RED shirt = near player, GRAY shirt = far player. One frame every 0.2 s, timestamp burned in bottom-left. Report every SERVE, HIT, POINT and FAULT with the player, the time and a short note stroke, or why the point ended ."}, {"type": "input text", "text": "Frame at t=0.0s:"}, {"type": "input image", "image url": "data:image/jpeg;base64,..."}, {"type": "input text", "text": "Frame at t=0.2s:"}, {"type": "input image", "image url": "data:image/jpeg;base64,..."} // ... 100 frames per 20 s chunk } , "text": {"format": {"type": "json schema", "strict": true, "schema": { "events": {"time": "number", "player": "RED | GRAY", "type": "SERVE | HIT | POINT | FAULT", "note": "string"} }}} } Astra returns events like {"time": 3.4, "player": "RED", "type": "SERVE"} . Tennis scoring over the POINT events gives the scoreboard. Cost: 100 frames at 768x432 are about 425 tokens each 85 plus 170 per 512 px tile , so ~45k input tokens per chunk, plus ~8k output tokens of reasoning. This takes ~3 minutes to process and costs about $1. FPS vs Accuracy? We re-ran the clip at 1 to 5 fps. Strokes are robust: all 7 serves and 17 hits were found at every rate, even at 1 fps. Point outcomes are not. Below 4 fps the model misses a point, as model doesn't have a frame to determine whether the ball went into the court. At other 2 demos FPS doesn't need to be nearly as high as for sports analytics. Demo 2: Process Monitoring An overhead camera above a taco line. Astra is checking whether chef has put all the right ingredients on the order. Same prompt shape: frames at 3 fps, the dishes on the plate, an event list. The one addition is an enum of the ingredients the chef has at their disposal from the buffet . The prompt structure is the same as for the tennis analytics, just the text, enums, and output format are specific to this environment: { "model": "gpt-6-astra", "reasoning": {"effort": "high"}, "input": {"role": "user", "content": {"type": "input text", "text": "Overhead camera above a taco station. Focus on the purple plate: one large TORTILLA and two small GORDITAS, minced meat already on them. Ingredients on the table: SHREDDED CABBAGE, GREEN SALSA, RED SALSA, GUACAMOLE, RADISH SLICES, LIME WEDGES, GRILLED SPRING ONIONS, ... Report every moment an ingredient lands on one of the items."}, {"type": "input text", "text": "Frame at t=0.00s:"}, {"type": "input image", "image url": "data:image/jpeg;base64,..."} // ... 82 frames, one every 0.33 s } , "text": {"format": {"type": "json schema", "strict": true, "schema": { "events": {"time": "number", "dish": "TORTILLA | GORDITA 1 | GORDITA 2", "ingredient": "SHREDDED CABBAGE | GREEN SALSA | RED SALSA | ...", "confidence": "high | medium | low"} }}} } Astra reported 13 additions in 27 seconds, all within 0.3 s of what we could see in the frames. There was single request with 82 frames, with 43k input tokens, took 160 seconds, and costed <$1. Demo 3: Logistics Turnaround Time A loading dock with three gates and an 11 minute CCTV clip. For each gate we want a timeline of gate state - either closed, open with no truck, truck docked and idle, loading, or unloading. That provides turnaround time and truck dwell time: One frame every 6 seconds is 112 frames for the whole 11min clip in a single request. The schema asks for {state, start, end} intervals per gate with no gaps. "schema": { "GATE 1": {"state": "CLOSED | OPEN NO TRUCK | TRUCK DOCKED IDLE | TRUCK LOADING | TRUCK UNLOADING", "start": "number", "end": "number"} , "GATE 2": ... , "GATE 3": ... } Cost: 34k input tokens, about $0.75 cents for 11 minutes footage. Cost Optimizations when using Astra for video In the example above, streaming video to Astra for 24h a day to understand the turnaround time might not be feasible, as you'd burn ~$100/day in tokens. Most of that footage does not need a frontier model. Whether a gate is open or closed, and whether a truck is docked, can be read by an object detector: train RF-DETR https://github.com/roboflow/rf-detr?ref=blog.roboflow.com on a few hundred labelled frames with classes like "open gate", "closed gate" and "truck" on Roboflow platform, and the state of each gate falls out of what is detected where. A small classifier on a per-gate crop works too. The hard part is telling cargo loading from unloading. That is action recognition: a few seconds of context about people, forklifts and where the boxes are going, which is what Astra is good at. It also does not need the 6 second sampling we used here. In production, the small detector models says when a truck docks and when it leaves, and in between there is one question to answer per truck, so a handful of frames, cropped to that gate one tile, about 255 tokens each , is enough. That turns about $100 a day per camera into a few cents per truck. When to Use GPT-6 Astra on Video Use Astra when you need to know what happened in a clip and can accept event times to within a sampling step. Frame sampling with burned-in timestamps and a strict schema is enough, at about a dollar per hundred frames. Astra is not a per-frame tracker. For keypoints, boxes or masks on every frame, run a fine-tuned RF-DETR in Roboflow https://roboflow.com/?ref=blog.roboflow.com and let Astra reason on top. Compare GPT-6 Astra against every model we have benchmarked, and test it on your own images, on Roboflow Playground https://playground.roboflow.com/?ref=blog.roboflow.com . Cite this Post Use the following entry to cite this post in your research: Erik Kokalj /author/erik/ . Sep 11, 2026 . Understanding Video with GPT-6 Astra. Roboflow Blog: https://blog.roboflow.com/gpt6-astra-video-understanding/