GPT-6 Astra is really good at vision, but doesn't support video inputs. Send it a sequence of timestamped frames in a request, ask for a strict JSON output, and it can accurately do action recognition by providing timestamps of events in the clip. We tried it on sports analytics, process monitoring and logistics turnaround time. Every video below is Astra's output drawn back onto the original video.
Score and event banners from GPT-6 Astra; skeletons from a Roboflow pose model.
GPT-6 Astra for Vision
GPT-6 Astra takes text and images. The API rejects an mp4, whether sent as a video part or as a file. So the question is how much of a video you can push through the image path.
The vision capabilities itself are very strong. Roboflow tested it on detection benchmarks and calls it the best vision model he has seen. See thread here.
Sending Video to a Model that only sees Images
The workaround is to send a sequence of frames (at a fixed rate) via input_image along with their timestamps in the prompt (input_text) itself. For the examples below we forced the output to be JSON with event type enums and timestamps, then we overlayed the status on original video.
Reasoning effort high helped on every clip. A 100 frame request takes two to three minutes and costs about $1 (at $12.50/MTok input, $50/MTok output).
Demo 1: Sports Analytics
A fixed camera behind the baseline with two players and scoreboard blacked out so Astra has to infer the score from play. We used 5FPS, and asked the model to return event type, timestamp and player id (shirt color).
Astra's SERVE, HIT, POINT and FAULT events as they happen.
Request (prompt) looked something like this:
{
"model": "gpt-6-astra",
"reasoning": {"effort": "high"},
"input": [{"role": "user", "content": [
{"type": "input_text", "text": "Fixed camera behind the baseline. RED shirt = near player, GRAY shirt = far player. One frame every 0.2 s, timestamp burned in bottom-left. Report every SERVE, HIT, POINT and FAULT with the player, the time and a short note (stroke, or why the point ended)."},
{"type": "input_text", "text": "Frame at t=0.0s:"},
{"type": "input_image", "image_url": "data:image/jpeg;base64,..."},
{"type": "input_text", "text": "Frame at t=0.2s:"},
{"type": "input_image", "image_url": "data:image/jpeg;base64,..."}
// ... 100 frames per 20 s chunk
]}],
"text": {"format": {"type": "json_schema", "strict": true, "schema": {
"events": [{"time": "number", "player": "RED | GRAY",
"type": "SERVE | HIT | POINT | FAULT", "note": "string"}]
}}}
}Astra returns events like {"time": 3.4, "player": "RED", "type": "SERVE"}. Tennis scoring over the POINT events gives the scoreboard.
Cost: 100 frames at 768x432 are about 425 tokens each (85 plus 170 per 512 px tile), so ~45k input tokens per chunk, plus ~8k output tokens of reasoning. This takes ~3 minutes to process and costs about $1.
FPS vs Accuracy? We re-ran the clip at 1 to 5 fps. Strokes are robust: all 7 serves and 17 hits were found at every rate, even at 1 fps. Point outcomes are not. Below 4 fps the model misses a point, as model doesn't have a frame to determine whether the ball went into the court. At other 2 demos FPS doesn't need to be nearly as high as for sports analytics.
Demo 2: Process Monitoring
An overhead camera above a taco line. Astra is checking whether chef has put all the right ingredients on the order. Same prompt shape: frames at 3 fps, the dishes on the plate, an event list. The one addition is an enum of the ingredients the chef has at their disposal (from the buffet).
Each ingredient ticked when Astra saw it land. Guacamole was ordered and never added. Hand skeletons and the plate box come from separate tracking models, not Astra.
The prompt structure is the same as for the tennis analytics, just the text, enums, and output format are specific to this environment:
{
"model": "gpt-6-astra",
"reasoning": {"effort": "high"},
"input": [{"role": "user", "content": [
{"type": "input_text", "text": "Overhead camera above a taco station. Focus on the purple plate: one large TORTILLA and two small GORDITAS, minced meat already on them. Ingredients on the table: SHREDDED_CABBAGE, GREEN_SALSA, RED_SALSA, GUACAMOLE, RADISH_SLICES, LIME_WEDGES, GRILLED_SPRING_ONIONS, ... Report every moment an ingredient lands on one of the items."},
{"type": "input_text", "text": "Frame at t=0.00s:"},
{"type": "input_image", "image_url": "data:image/jpeg;base64,..."}
// ... 82 frames, one every 0.33 s
]}],
"text": {"format": {"type": "json_schema", "strict": true, "schema": {
"events": [{"time": "number", "dish": "TORTILLA | GORDITA_1 | GORDITA_2",
"ingredient": "SHREDDED_CABBAGE | GREEN_SALSA | RED_SALSA | ...",
"confidence": "high | medium | low"}]
}}}
}Astra reported 13 additions in 27 seconds, all within 0.3 s of what we could see in the frames. There was single request with 82 frames, with 43k input tokens, took 160 seconds, and costed <$1.
Demo 3: Logistics Turnaround Time
A loading dock with three gates and an 11 minute CCTV clip. For each gate we want a timeline of gate state - either closed, open with no truck, truck docked and idle, loading, or unloading. That provides turnaround time and truck dwell time:
Per-gate state timeline from Astra, played back at 15x; skeletons from a Roboflow pose model.
One frame every 6 seconds is 112 frames for the whole 11min clip in a single request. The schema asks for {state, start, end} intervals per gate with no gaps.
"schema": {
"GATE_1": [{"state": "CLOSED | OPEN_NO_TRUCK | TRUCK_DOCKED_IDLE | TRUCK_LOADING | TRUCK_UNLOADING", "start": "number", "end": "number"}],
"GATE_2": [ ... ],
"GATE_3": [ ... ]
}Cost: 34k input tokens, about $0.75 cents for 11 minutes footage.
Cost Optimizations when using Astra for video
In the example above, streaming video to Astra for 24h a day to understand the turnaround time might not be feasible, as you'd burn ~$100/day in tokens.
Most of that footage does not need a frontier model. Whether a gate is open or closed, and whether a truck is docked, can be read by an object detector: train RF-DETR on a few hundred labelled frames with classes like "open_gate", "closed_gate" and "truck" on Roboflow platform, and the state of each gate falls out of what is detected where. A small classifier on a per-gate crop works too.
The hard part is telling cargo loading from unloading. That is action recognition: a few seconds of context about people, forklifts and where the boxes are going, which is what Astra is good at. It also does not need the 6 second sampling we used here. In production, the small detector models says when a truck docks and when it leaves, and in between there is one question to answer per truck, so a handful of frames, cropped to that gate (one tile, about 255 tokens each), is enough. That turns about $100 a day per camera into a few cents per truck.
When to Use GPT-6 Astra on Video
Use Astra when you need to know what happened in a clip and can accept event times to within a sampling step. Frame sampling with burned-in timestamps and a strict schema is enough, at about a dollar per hundred frames.
Astra is not a per-frame tracker. For keypoints, boxes or masks on every frame, run a fine-tuned RF-DETR in Roboflow and let Astra reason on top.
Compare GPT-6 Astra against every model we have benchmarked, and test it on your own images, on Roboflow Playground.
Cite this Post
Use the following entry to cite this post in your research:
Erik Kokalj. (Sep 11, 2026). Understanding Video with GPT-6 Astra. Roboflow Blog: https://blog.roboflow.com/gpt6-astra-video-understanding/