Autonomous vehicle (AV) development often relies on separate models for trajectory generation, high-level intent prediction, scene understanding, and data labeling. This separation makes it hard to compare related outputs, investigate model behavior, and reuse the same representations across the development workflow.
NVIDIA Alpamayo 2 Super is an open 34-billion-parameter reasoning vision-language-action (VLA) model designed to accelerate autonomous vehicle (AV) development. It combines the 32-billion-parameter NVIDIA Cosmos 3 Super Reasoner with a 2-billion-parameter diffusion-based Action Expert and is post-trained with reinforcement learning. The reasoner interprets multi-camera video, language context, and prior motion history, while the Action Expert converts the model’s resulting internal representation into a future ego-vehicle trajectory.
Alpamayo 2 Super’s perception expands to 360-degree coverage across up to seven cameras and can return multiple complementary outputs: future trajectories, Chain-of-Causation (CoC) reasoning traces, high-level meta-actions, grounded answers to questions about the scene, and reasoning auto-labels.
This multi-task design gives AV developers a common foundation across multiple stages of the development workflow. The same foundation model can be used as an offline policy teacher, an evaluation critic, a data engine, or a starting point for new task customization, instead of maintaining a separate model for each stage of the workflow.
This post provides a hands-on introduction to four Alpamayo 2 Super-enabled workflows:
- Generate trajectories and CoC reasoning traces, evaluating the results in open-loop and closed-loop benchmarks.
- Predict meta-actions such as yield, change lanes, and stop alongside a trajectory.
- Ask natural-language questions about a multi-camera driving scene.
- Generate CoC auto-labels with 2D grounding on your own clips.
The model weights are available on Hugging Face and the inference notebooks on GitHub. The model is released under OpenMDW-1.1, the Linux Foundation permissive license for open model distributions, which covers fine-tuning, derivative models, and commercial redistribution. Distilled models can be deployed commercially without further permission from NVIDIA, and model outputs carry no license conditions.
Planning and reasoning #
Reasoning through new scenarios is a fundamental problem in autonomous driving. Navigating construction zones, partially occluded pedestrians, unusual right-of-way interactions, and objects entering the roadway requires more than matching a common trajectory pattern. A useful driving model must identify scene context that matters, connect it to the appropriate driving decision, and produce an action consistent with that decision.
Trajectories and CoC traces: What and why
Alpamayo 2 Super, like its predecessors, jointly produces output trajectories and CoC reasoning traces. The trajectory expresses what the ego vehicle could do next. The CoC reasoning trace provides insights into why a driving decision was made from observed scene context. Returning both outputs makes it easier to understand the model’s decision-making, curate difficult cases, compare a deployed policy with a larger teacher, and diagnose whether a failure originated in perception, reasoning, or action generation. CoC traces also feed into the NVIDIA Halos safety validation workflows by enabling introspection into the model’s understanding of the scene.
The Alpamayo 2 Super repository’s inference notebook loads a surround-view clip, prepares ego-motion history, and samples a trajectory with its associated CoC trace. The core inference step and outputs are shown below.
from alpamayo2_super import helper
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.models.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.visualization import plot_inference_result
data = load_physical_aiavdataset(
"030c760c-ae38-49aa-9ad8-f5650a545d26",
t0_us=2000000,
)
model = Alpamayo2Super.from_pretrained("nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device_map="cuda:0")
model_inputs = helper.prepare_model_inputs(data, model.config, model.tokenizer)
model_inputs = helper.to_device(model_inputs, "cuda")
torch.cuda.manual_seed_all(42)
with torch.autocast("cuda", dtype=torch.bfloat16):
pred_xyz, pred_rot, logprob, extra = model.sample_trajectories_from_data(
data=model_inputs,
top_p=0.98,
temperature=0.6,
num_traj_samples=1,
diffusion_kwargs={"inference_step": 10},
return_extra=True,
)
fig, metadata = plot_inference_result(
data=data,
pred_xyz=pred_xyz,
extra=extra,
)
Evaluation methods #
To evaluate the model’s output reasoning and trajectory quality, we can use open-loop and closed-loop evaluation methods. Open-loop evaluation measures trajectory and reasoning quality on recorded scenes by comparing them to ground-truth labels.
Alpamayo 2 Super achieves the following results:
Trajectory prediction: Across 1,434 challenging samples from thePhysical AI AV Dataset, it records a 6.4-secondminADE_6
of 0.911 m.AV reasoning: It scores 0.433 on thePhysical AI AV Reasoning Benchmark.LingoQA: Alpamayo 2 Super achieves 79.2 on theLingoQA benchmark, ranking first among 37 evaluated models. With 34 billion parameters, it leads Qwen2.5-VL (72B) by 17.0 points, Qwen3-VL (32B) by 7.0 points, Gemini 2.5 Pro by 15.1 points, and GPT-4o by 23.2 points.
Lower minADE_6
values indicate better trajectory predictions; higher reasoning scores indicate better performance.
The main challenge with open-loop metrics, however, is that they evaluate predictions against a fixed, prerecorded future and therefore don’t capture what happens after the model’s first action, which may affect the rest of the scene.
For example, if the ego vehicle changes lanes, an open-loop replay may continue moving an adjacent vehicle along its recorded trajectory rather than accounting for how it would respond to the ego vehicle.
Closed-loop simulation executes each predicted action within the scene and, when the simulator includes reactive behavior models, captures how surrounding agents may react. NVIDIA AlpaSim enables this by repeatedly rendering observations, querying the policy, and applying its actions so developers can measure these closed-loop effects over time.
To run Alpamayo 2 Super on an AlpaSim evaluation suite, use the corresponding shell command:
uv run alpasim_wizard deploy=local topology=2gpu driver=alpamayo2
wizard.log_dir=$PWD/tutorial eval.video.video_layouts=[REASONING_OVERLAY]
with the following AlpaSim wizard configuration:
defaults:
- alpamayo_configs # Camera and simulation configs for 4-cam 10Hz
- _self_ # YAML values override schema defaults
log_level: ${wizard.log_level}
model:
model_type: alpamayo2 # Entry-point name in alpasim.models registry
checkpoint_path: "nvidia/Alpamayo2-Super"
device: "cuda"
use_classifier_free_guidance_nav: false
host: "0.0.0.0"
port: ???
inference:
use_cameras:
- camera_cross_left_120fov
- camera_front_wide_120fov
- camera_cross_right_120fov
- camera_front_tele_30fov
max_batch_size: 1 # A2Super is memory intensive, start with batch size 1
subsample_factor: 1
context_length: 4 # A2Super uses 4 temporal frames per camera
route:
use_waypoint_commands: false
output_dir: "/mnt/output/driver"
trajectory_optimizer:
enabled: false
plot_debug_images: false
On 913 reconstructed scenes, Alpamayo 2 Super obtains an AlpaSim Score of 1.50 ± 0.13. This closed-loop score complements open-loop results by revealing collisions, road departures, close encounters, and other failures that can emerge only after the policy influences future observations.
| Evaluation | Benchmark (set) | Metric | Alpamayo 2 Super | Reference scores | Better | | Open-loop | |
Physical AI AV Reasoning BenchmarkGPT-5.5: 0.502
LingoQAQwen3-VL 32B: 72.2Qwen2.5-VL 72B: 62.2
Gemini 2.5 Pro: 64.1GPT-4o: 56.0
AlpaSim(913 reconstructed scenes from thePhysical AI AV NuRecdataset)Table 1. Open-loop and closed-loop evaluation of Alpamayo 2 Super on long-tail driving scenarios
Meta-actions #
A trajectory is precise, but it doesn’t always provide a compact description of intent. Meta-actions summarize a plan in terms of high-level decisions such as yield, change lanes, stop, or accelerate. These outputs can help bridge an end-to-end foundation model and a modular AV stack: a downstream planner can consume the decision, an evaluator can check whether trajectory geometry agrees with intent, and teams can search their data corpus for particular maneuvers.
Generate meta-actions
The meta-actions notebook shows how to produce meta-action outputs with an example scene. For a detailed list of supported meta-actions, refer to these lists. The core inference step and outputs are shown below.
from alpamayo2_super import helper
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.models.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs
data = load_physical_aiavdataset(
"030c760c-ae38-49aa-9ad8-f5650a545d26",
t0_us=2000000,
)
model = Alpamayo2Super.from_pretrained("nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device_map="cuda:0")
task_inputs = prepare_text_generation_inputs(
data=data,
model_config=model.config,
tokenizer=model.tokenizer,
task="meta_action",
)
task_inputs = helper.to_device(task_inputs, "cuda")
torch.cuda.manual_seed_all(42)
with torch.autocast("cuda", dtype=torch.bfloat16):
result = generate_text(
model,
task_inputs,
top_p=0.98,
temperature=0.6,
max_new_tokens=512,
)
cot = result["cot"][0]
meta_action = result["meta_action"][0]
print("Chain-of-Causation:\n", cot)
print("\nMeta-action:\n", meta_action)
Evaluate meta-action accuracy
To evaluate meta-action accuracy, we compare Alpamayo 2 Super’s output with ground-truth labels and report the resulting classification accuracy in terms of intersection-over-union (IoU) for the three components of its meta-action taxonomy: lateral, longitudinal, and lane-wise.
On an internal set of 94K clips with ground-truth meta-action data, Alpamayo 2 Super achieves 74.59 lateral IoU, 61.91 longitudinal IoU, and 73.55 lane-wise IoU across its meta-action taxonomy.
Scene understanding #
Planning is only one way to use a driving foundation model. Visual question answering (VQA) exposes the model’s scene understanding directly through natural language. Developers can ask about key elements in the scene, how they affect driving behavior, why the ego vehicle should slow, or about other aspects of scenarios.
ne-change gap.
VQA is valuable for interactive debugging and data operations. It can help a developer inspect why a policy behaved a certain way, build semantic filters over large clip collections, and generate candidate annotations for human review. With surround-view input, questions can reference side and rear context that a front-view-only model would miss.
Querying multi-camera scenes with VQA and 2D grounding
For each clip, Alpamayo 2 Super can generate answers and spatially localize referenced actors by predicting 2D bounding boxes in the relevant camera frames. This grounding makes outputs more useful than free-form text alone. Reviewers can verify which specific object the model is referring to, automated checks can flag missing or inconsistent boxes, and downstream models can leverage (e.g., through distillation) these tighter links between visual evidence, reasoning, and action.
The scene understanding and VQA notebook shows how to perform VQA with the same inputs. The core inference step and outputs are shown below.
from alpamayo2_super import helper
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.models.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs
data = load_physical_aiavdataset(
"ea4a6729-bf33-4997-905b-cd58774a3580",
t0_us=7500000,
)
model = Alpamayo2Super.from_pretrained("nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device_map="cuda:0")
task_inputs = prepare_vqa_inputs(
data=data,
model_config=model.config,
tokenizer=model.tokenizer,
question="Describe the driving scene and identify the key traffic elements that should influence ego behavior.",
)
task_inputs = helper.to_device(task_inputs, "cuda")
with torch.autocast("cuda", dtype=torch.bfloat16):
result = generate_text(
model,
task_inputs,
top_p=1.0,
temperature=0.1,
max_new_tokens=1024,
)
answer = result["answer"][0]
print(answer)
The model can return two types of output for the same input: Figure 4, above, shows a natural-language description of the scene, while Figure 5, below, shows the same model spatially localizing referenced objects by predicting 2D bounding boxes in the relevant camera frame.
Evaluate VQA response and grounding quality
To evaluate VQA performance, we compare Alpamayo 2 Super’s generated responses with ground truth answers across an internal set of 8K question-answer pairs. The model achieves 0.652 answer similarity (higher is better), compared with Qwen3-VL 32B at 0.450. For 2D grounding, we compare the predicted bounding boxes to ground-truth annotations using IoU and obtain 0.71 compared to 0.17 for Qwen3-VL 32B. Together, these measurements evaluate whether the model both answers questions accurately and associates its responses with the correct visual evidence across the surround-view cameras.
Auto-labeling #
Reasoning models need decision-grounded reasoning data, but labeling long-tail driving clips by hand is expensive and slow. Annotators must inspect temporal and multi-camera context, identify the causal actors, describe how they affect the ego vehicle, and keep the label consistent with the intended maneuver. Alpamayo 2 Super can serve as an offline auto-labeler that proposes this structure at scale. This can compress annotation cycles from months to days.
Generate structured CoC auto-labels
The CoC auto-labeling notebook accepts clips in the released schema and writes one structured record per selected keyframe.
from alpamayo2_super import helper
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.models.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs
data = load_physical_aiavdataset(
"b5f3756c-4f0e-4298-a1ff-cc92ed392ae0",
t0_us=11000000,
)
model = Alpamayo2Super.from_pretrained("nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device_map="cuda:0")
task_inputs = prepare_text_generation_inputs(
data=data,
model_config=model.config,
tokenizer=model.tokenizer,
task="auto_labeling",
)
task_inputs = helper.to_device(task_inputs, "cuda")
torch.cuda.manual_seed_all(42)
with torch.autocast("cuda", dtype=torch.bfloat16):
result = generate_text(
model,
task_inputs,
top_p=0.98,
temperature=0.6,
max_new_tokens=1024,
)
auto_labeling_text = result["cot_auto_labeling"][0]
auto_labeling_json = result["cot_auto_labeling_json"][0]
print(json.dumps(auto_labeling_json, indent=2))
By default, Alpamayo 2 Super assumes access to future ego-trajectory information. However, it can also auto-label data that does not contain future trajectory information:
trajectory_inputs = helper.prepare_model_inputs(data, model.config, model.tokenizer)
trajectory_inputs = helper.to_device(trajectory_inputs, "cuda")
torch.cuda.manual_seed_all(42)
with torch.autocast("cuda", dtype=torch.bfloat16):
pred_xyz, pred_rot, _, extra = model.sample_trajectories_from_data(
data=trajectory_inputs,
top_p=0.98,
temperature=0.6,
num_traj_samples=1,
diffusion_kwargs={"inference_step": 10},
return_extra=True,
)
future_xyz = pred_xyz[:, 0, 0].detach().cpu()
future_rot = pred_rot[:, 0, 0].detach().cpu()
task_inputs = prepare_text_generation_inputs(
data=data,
model_config=model.config,
tokenizer=model.tokenizer,
task="auto_labeling",
future_xyz=future_xyz,
future_rot=future_rot,
)
task_inputs = helper.to_device(task_inputs, "cuda")
torch.cuda.manual_seed_all(42)
with torch.autocast("cuda", dtype=torch.bfloat16):
result = generate_text(
model,
task_inputs,
top_p=0.98,
temperature=0.6,
max_new_tokens=1024,
)
auto_labeling_text = result["cot_auto_labeling"][0]
auto_labeling_json = result["cot_auto_labeling_json"][0]
print(json.dumps(auto_labeling_json, indent=2))
Evaluate CoC auto-labeling quality
To evaluate CoC auto-labeling quality, we compare Alpamayo 2 Super’s generated labels with expert annotations on 8k internal clips. We use an internal judge model to assess similarity. Alpamayo 2 Super achieves a 0.652 similarity score, compared with Qwen3-VL 32B at 0.450. These results show its potential to generate structured reasoning labels at scale while maintaining consistency with expert-authored annotations.
Build with Alpamayo 2 Super #
Alpamayo 2 Super brings surround-view perception, reasoning, planning, scene understanding, and data auto-labeling into one open model workflow. These capabilities give developers a practical foundation for building teacher models, curating long-tail data, inspecting policy decisions, and evaluating AV systems beyond a single open-loop trajectory metric. As a teacher model, it can be distilled into compact models that run on NVIDIA DRIVE AGX Thor inside the vehicle.
Explore the model on Hugging Face, run the inference notebooks, and share what you build on the Alpamayo developer forum.
For more details on the broader updates introduced as part of the Alpamayo 2 release, see the associated Hugging Face blog post.