An annotation benchmark that accepts valid alternatives
TL;DR: Shotwell-Bench fixes the core problem with annotation benchmarks. Existing benchmarks are extremely rigid in how they expect annotations, penalizing accurate model outputs which leads to noisy benchmark scores. Shotwell-Bench accepts multiple valid ways to annotate the same video and reflects model performance far more comprehensively. GPT-6 Astra performs the best on this benchmark.
Why another benchmark? #
VLMs are a practical way to produce dense language annotations for the large amounts of egocentric video used in robot-policy pretraining.
Existing benchmarks that assess VLM annotation accuracy are flawed because they penalize annotations that differ from the benchmark's reference set but are still valid annotations. These benchmark limitations hinder research efforts, as they don’t accurately capture nuance. We built Shotwell-Bench, extending WGO-Bench, to assess accuracy in producing annotations without penalizing different valid ways of segmenting the same video.
Below are the F1 scores on segmentation and description accuracy. GPT-6 Astra leads in both.
How we align annotation for better scoring #
An annotation divides an episode into segments representing actions and describes each action with a text description. The tradeoffs between VLM approaches to doing this are:
- Accuracy: are the segment boundaries correct, or did actions get cut off or grouped together? Are the descriptions correct, or do they mention the wrong object, action, or hand?
- Level of detail: do the descriptions miss key information?
- Cost: how much does it cost to produce the annotations?
Accuracy is particularly difficult to benchmark: annotators can choose different boundaries or group actions differently, yet produce equally valid accounts of what happened.
Shotwell-Bench provides a better way to judge accuracy by restructuring WGO-Bench in four ways:
- Multiple acceptable segmentations. Even with a precise guide, there is a range of timestamps where an action boundary can be placed, and some actions can reasonably be grouped or split. The reference annotations include acceptable ranges for each individual boundary, rather than one fixed tolerance for every boundary. They also include optional sub-boundaries where splitting an action is valid. Every valid possible segmentation logic for an episode is captured.
- Multiple acceptable descriptions. For description grading, models receive fixed segments to describe. Each reference description provides required details and optional details for each segment. An LLM judge checks that every required action is present with the right verb, object, and hands. It allows compatible specificity and slight generalizations, but rejects contradictions and independent actions without reference support. The guide and explicit reference details leave less to the LLM judge's discretion.
- An extremely detailed annotation guide. Our guide specifies which actions merit their own segment, exactly where boundaries should fall, and the format and level of detail expected in each description. This makes model outputs very consistent before grading begins. Shotwell-Bench also demands a much higher level of detail in outputs that spotlights the limits of weaker models.
- Curated episodes with clearer actions. We select clips where action boundaries and descriptions can be judged with relatively little ambiguity, limiting cases where equally careful annotators would disagree.
The limits of a single reference timeline #
WGO-Bench and MicroAnnotationBench compare VLM annotations with human reference annotations. Both compare segmentation against just a single reference timeline. This can penalize logical alternative groupings of actions and understate annotation quality. Astra scores around 0.40 F1 score in published evaluations of both benchmarks. Below, we show examples of our run of Astra on the WGO-Bench where valid alternative groupings are marked wrong due to the benchmark's IoU threshold of 0.75.
Multiple acceptable segmentations #
Our approach relies on two important properties in our reference annotations and scoring: boundary ranges and optional boundaries.
Boundary ranges
There is always a range of timestamps where a boundary is valid, but the width of that range will vary per segment. In some cases, the range might be wide. For example, a ‘pick up’ action might end as soon as the object is lifted above the surface, or later in the trajectory, if no other action begins. In other cases, the range might be much narrower, such as when a pour ends and the operator immediately starts placing the container.
Our approach captures acceptable timing more directly than a fixed tolerance or an IoU overlap threshold. We create an acceptability range for every boundary: a model marker inside that range can count as a hit, while one outside all accepted ranges is an error. Matching is one-to-one, so an extra marker inside an already matched range is still an error.
Optional boundaries
No matter how specific an annotation guide is, there will always be cases where different segmentation interpretations are valid. Two actions could be grouped together or split into separate segments, and both choices can be defensible. The scoring should accept each valid interpretation. Shotwell-Bench handles these alternatives by distinguishing between minor and major boundaries: Major boundaries are required, and missing one incurs a penalty. Minor boundaries are optional and can be omitted without penalty. Placing a marker outside every minor and major boundary range also incurs a penalty.
In the example above, Opus splits the glass and sponge placement; Astra groups it. Both match the required boundaries, and the split is optional. Luna, however, misses the last boundary as the last segment shown above extends well past it.
Multiple acceptable descriptions #
There's more than one good way to describe the same action. For each segment, we list the details that matter and mark others as optional. A correct output must list all the required details, and must not fabricate any detail not present in the required or optional detail list. Within a detail, there may be different interpretations of an action or item. These interpretation options are represented as brackets below, and the judge will accept any of the options. Different wording is accepted, as is having a slightly different level of specificity.
Below are three examples of reference segment descriptions, and accepted and rejected annotations.
-
Required details
- Pour water into the glass with the right hand.
-
Optional details
- Hold the glass with the left hand.
-
Accepted
- “Pour water into the glass with the right hand.”
- “Pour water into the glass with the right hand and hold the glass with the left hand.”
-
Rejected
- “Pour water into the glass with the left hand.” (Wrong hand.)
-
Required details
- [Slide left / push] a glass with the left hand.
-
Optional details
-
None.
-
Accepted
- “Slide a glass left with the left hand.”
- “Push a glass with the left hand.”
-
Rejected
- “Pick up the glass with the left hand.” (Wrong action.)
-
Required details
- Squeeze the sponge with the right hand.
- Turn on the faucet with the left hand.
-
Optional details
-
None.
-
Accepted
- “Squeeze the sponge with the right hand and turn on the faucet with the left hand.”
-
Rejected
- “Squeeze the sponge with the right hand.” (Missing the faucet action.)
Disambiguating with a pedantic guide #
In the clips above, Astra's Shotwell annotations closely match the reference segments, which makes grading much clearer and more objective. The segment boundaries are almost identical, and some captions differ by only a few words. Take a look at how Astra captions line up more closely on Shotwell-Bench than WGO-Bench, both in timing and wording.
The similarity is because of how specific the guide is in where it expects segment boundaries, and how it expects descriptions to be formatted. By reducing ambiguity in segmentation and captioning, the guide makes model outputs more consistent with the reference annotations and easier to evaluate reliably.
Where the boundaries go
The guide has a large list of rules for where segment boundaries must be placed. These are the parts most relevant to the segments above; expand a snippet to read the full rule:
Merge actions into one segment only when they overlap in time. #
Set boundaries so that each segment features a single complete or attempted action, fully contained within that segment.
Merge actions into a single segment only when the actions overlap in time.
End a pick-up segment when the object clears its supporting surface. #
Grabbing an object with one or both hands and lifting it. The object must be completely lifted from its supporting surface; otherwise, it is a single Move action rather than separate Pick up and Place actions. Keep multiple lifting attempts of the same object together unless the operator completely releases the object between attempts. If the object is released without being lifted but a clear attempt to lift the object was made, treat the attempt as a Pickup action (completed unsuccessfully).
Start: When hands start reaching or moving toward the object, or the operator starts leaning forward to pick up the items.
End: Once the object is lifted and not in contact with the surface. For an unsuccessful attempt, end when the hands completely release it.
Start a Place segment as the object moves toward its destination; end it at release. #
Placing / fully releasing an object that is held in hand(s), or pouring a liquid or grain, or spraying using a spray nozzle.
Start: When the object begins moving toward its destination.
End: When the hands completely release the object. For pouring, end when no more liquid or grain is being transferred. For spraying, after the spray leaves the nozzle. If it has stopped moving the object entirely but not released when another action begins, mark the end when the object stops moving instead of overlapping. If the hands nudge the placed item for the purposes of securing it in place after placement, include that nudging in the segment and end after.
End a Rub segment when the hand releases the tool or removes it from the object. #
Rubbing or wiping a surface or object using a tool, such as a sponge or cloth, or using the hands directly.
Start: When the hand begins reaching toward the object.
End: When the hand releases the rubbing tool or removes it from the object.
Group repetitions on one object only when the hands and motion stay the same. #
Group short, repetitive motions into one segment when they act on the same object(s), use the same hand(s), and are performed in the same way. They may act on different parts of the same object, for example cleaning the top and sides of furniture is the same and should remain in one segment. There is one exception, which is cleaning different shelves, even if they’re connected. Split the repetitions when the object or hand changes, or when a separate action occurs between them, such as rotating the object. The segment starts with the first occurrence’s start condition and ends with the last occurrence’s end condition.
Group both hands' actions when their active movements overlap. #
Group overlapping actions into one segment when each hand is performing an action and the first action’s end condition occurs after the next action’s start condition. Start at the first action’s start condition and end at the last action’s end condition. Simply holding an object in a hand does not constitute a simultaneous action.
The guide provides far more rules on segmentation than WGO-Bench. This leaves no room for ambiguity, forcing model outputs to align.
What the descriptions say
The guide is just as specific about what each description must say. These snippets show the relevant parts; expand one to read the full rule:
Name the action, object, acting hand, and any tool in each caption. #
Descriptions take the imperative form. They explain what action was performed and which hands (and which tools, if any) were used.
Standard action sentence: "{verb} + {object} + {hand (+ tool) used}."
Navigation sentence: "{verb} {destination}".
No extra detail beyond these structures, other than the exceptions in this guide.
For an IDLE segment, write just "IDLE".
[Overlap](https://github.com/shotwellai/shotwell-bench/blob/main/guide.md#compound-sentences)
Join overlapping actions as independent clauses with “and.” #
When a segment contains overlapping actions, its description is a compound sentence. Each action gets a full, independent description that could stand alone, joined with "and" in the order the actions start. In the compound sentence, the clauses should be fully independent without any reference to each other (no "Pick the cup with the left hand and pick the other cup with the right hand"). For example:
-
One hand performs one action while the other hand performs a different action, overlapping in time ("Pick the cup with the left hand and pick the bowl with the right hand.")
-
The person walks while interacting with an object in their hands ("Walk to the trash can and carry the paper plate in both hands.")
-
An item is picked up and then transferred quickly, such that either the pick or transfer segments individually would be shorter than 0.5s ("Pick up the bottle with the left hand and transfer the bottle from the left hand to the right hand.")
Name the destination surface or container in a placement caption. #
Give brief location information in only two cases:
-
The destination of a navigation segment ("Walk toward the counter.")
-
Placing an item ("Put the bottle on the shelf with the right hand.")
Placed inside a container → name the container ("Put the bottle in the fridge", not "Put the bottle on the top shelf of the fridge").
Placed on a surface → name the surface ("Place the plate on the table").
Placing / stacking on top of other item(s) → name the other items and the surface or container they are on/in ("Place the book on the stack of books on the shelf with the left hand" or “Place the bottle on the coaster on the table with the right hand”).
Do not give extra detail on the location (not "kitchen-side counter" or "top shelf").
For an object moved along a surface without lifting it, provide the direction of movement using one of the following options for the verb:
-
Slide left
-
Slide right
-
Pull
-
Push
("Slide the bottle left with the right hand").
Do not add adjectives merely to distinguish similar objects. #
Do not add adjectives to disambiguate when several items matching a description are present, but use “a/an {object}” in place of “the {object}” when interacting with an object not already in the hands and several objects fit the description during the segment. When several apples are in view, "Pick an apple with the left hand" is correct; "Pick the nearest apple with the left hand" is incorrect.
If unsure about the identity of an item, use a broader category ("garment" instead of "shirt", "fruit" instead of "apple").
[Holding](https://github.com/shotwellai/shotwell-bench/blob/main/guide.md#holding-and-other-passive-actions)
Name an item held in the other hand when that hold is useful. #
If one hand holds an item during a segment created for another, more noticeable activity, mention the held item too ("Wipe the mirror with the cloth with the right hand and hold the spray bottle in the left hand."). This also applies to other passive but useful interactions with objects ("Place the bottle in the fridge with the left hand and hold the fridge door open with the right hand."). Do not name passive, non-useful actions, such as resting a hand on a surface or on an item. For example, do not write "Place item in container with right hand and hold container with left hand" if the left hand is only resting on the container edge.
Do name a hand that is actively steadying an item, perhaps because the other hand's activity may destabilize it ("Scrub the container with the cloth with the right hand and steady the container with the left hand").
“Releasing” an item from grasp usually should not be noted. Do not name the “release” of an item by a hand separately if the hand was moving the item along a surface or steadying the item without lifting it off a surface. For items lifted off a surface, then placing the item should be noted, but there shouldn’t be an additional “release” action as well. (no “Release the cup with the left hand” after “Place the cup on the table with the left hand.”)
Description rules are also far more explicit in Shotwell-Bench than in the original WGO-Bench. The exact required format of sentences and rulings on required details leave no room for ambiguity, which makes checking accuracy far more objective. The guide also has more required details than WGO-Bench, making this a harder benchmark that punishes weaker models.
Results #
The chart compares the mean of each model's segmentation and description F1 with estimated cost per hour of video. The green line marks the Pareto frontier: GPT-6 Luna, GPT-6.1 Terra, GPT-6.1 Sol, and GPT-6 Astra.
Here, the capabilities of stronger models, and the gap between them and weaker models, are shown much more strongly than in the original WGO-Bench. In the original benchmark, smaller models that were clearly weaker (such as FLORA) beat Astra and GPT-5.6 Sol on segmentation F1 due to the noisiness of the scoring approach. Shotwell-Bench results correlate much more strongly with the generally accepted visual abilities of these models.
Browse the benchmark videos and reference annotations here. The annotation guide, runner, and scoring code are in the benchmark repository.
Acknowledgments #
Thanks to the teams behind WGO-Bench and MicroAnnotationBench, whose work we build on and cite throughout this benchmark. Our data was curated from the WGO-Bench dataset.