Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super

NVIDIA released Alpamayo 2 Super, an open 34-billion-parameter reasoning vision-language-action model for autonomous vehicle development, combining a 32-billion-parameter Cosmos 3 Super Reasoner with a 2-billion-parameter diffusion-based Action Expert. The model generates future trajectories, Chain-of-Causation reasoning traces, meta-actions, grounded answers, and auto-labels across up to seven cameras, and is available on Hugging Face under the OpenMDW-1.1 license. It aims to unify multiple AV development stages into a single foundation model, supporting offline policy teaching, evaluation, data labeling, and task customization.

Autonomous vehicle AV development often relies on separate models for trajectory generation, high-level intent prediction, scene understanding, and data labeling. This separation makes it hard to compare related outputs, investigate model behavior, and reuse the same representations across the development workflow. NVIDIA Alpamayo 2 Super https://huggingface.co/nvidia/Alpamayo2-Super is an open 34-billion-parameter reasoning vision-language-action VLA model designed to accelerate autonomous vehicle AV development. It combines the 32-billion-parameter NVIDIA Cosmos 3 Super https://huggingface.co/nvidia/Cosmos3-Super Reasoner with a 2-billion-parameter diffusion-based Action Expert and is post-trained with reinforcement learning. The reasoner interprets multi-camera video, language context, and prior motion history, while the Action Expert converts the model’s resulting internal representation into a future ego-vehicle trajectory. Alpamayo 2 Super’s perception expands to 360-degree coverage across up to seven cameras and can return multiple complementary outputs: future trajectories, Chain-of-Causation CoC reasoning traces, high-level meta-actions, grounded answers to questions about the scene, and reasoning auto-labels. This multi-task design gives AV developers a common foundation across multiple stages of the development workflow. The same foundation model can be used as an offline policy teacher, an evaluation critic, a data engine, or a starting point for new task customization, instead of maintaining a separate model for each stage of the workflow. This post provides a hands-on introduction to four Alpamayo 2 Super-enabled workflows: - Generate trajectories and CoC reasoning traces, evaluating the results in open-loop and closed-loop benchmarks. - Predict meta-actions such as yield, change lanes, and stop alongside a trajectory. - Ask natural-language questions about a multi-camera driving scene. - Generate CoC auto-labels with 2D grounding on your own clips. The model weights are available on Hugging Face https://huggingface.co/nvidia/Alpamayo2-Super and the inference notebooks on GitHub https://github.com/NVlabs/alpamayo2 . The model is released under OpenMDW-1.1, the Linux Foundation permissive license for open model distributions, which covers fine-tuning, derivative models, and commercial redistribution. Distilled models can be deployed commercially without further permission from NVIDIA, and model outputs carry no license conditions. Planning and reasoning Reasoning through new scenarios is a fundamental problem in autonomous driving. Navigating construction zones, partially occluded pedestrians, unusual right-of-way interactions, and objects entering the roadway requires more than matching a common trajectory pattern. A useful driving model must identify scene context that matters, connect it to the appropriate driving decision, and produce an action consistent with that decision. Trajectories and CoC traces: What and why Alpamayo 2 Super, like its predecessors, jointly produces output trajectories and CoC reasoning traces. The trajectory expresses what the ego vehicle could do next. The CoC reasoning trace provides insights into why a driving decision was made from observed scene context. Returning both outputs makes it easier to understand the model’s decision-making, curate difficult cases, compare a deployed policy with a larger teacher, and diagnose whether a failure originated in perception, reasoning, or action generation. CoC traces also feed into the NVIDIA Halos safety validation workflows by enabling introspection into the model’s understanding of the scene. The Alpamayo 2 Super repository’s inference notebook https://github.com/NVlabs/alpamayo2/blob/main/notebooks/inference.ipynb loads a surround-view clip, prepares ego-motion history, and samples a trajectory with its associated CoC trace. The core inference step and outputs are shown below. python from alpamayo2 super import helper from alpamayo2 super.load physical aiavdataset import load physical aiavdataset from alpamayo2 super.models.alpamayo2 super import Alpamayo2Super from alpamayo2 super.visualization import plot inference result data = load physical aiavdataset "030c760c-ae38-49aa-9ad8-f5650a545d26", t0 us=2000000, model = Alpamayo2Super.from pretrained "nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device map="cuda:0" model inputs = helper.prepare model inputs data, model.config, model.tokenizer model inputs = helper.to device model inputs, "cuda" torch.cuda.manual seed all 42 with torch.autocast "cuda", dtype=torch.bfloat16 : pred xyz, pred rot, logprob, extra = model.sample trajectories from data data=model inputs, top p=0.98, temperature=0.6, num traj samples=1, diffusion kwargs={"inference step": 10}, return extra=True, fig, metadata = plot inference result data=data, pred xyz=pred xyz, extra=extra, Evaluation methods To evaluate the model’s output reasoning and trajectory quality, we can use open-loop and closed-loop evaluation methods. Open-loop evaluation measures trajectory and reasoning quality on recorded scenes by comparing them to ground-truth labels. Alpamayo 2 Super achieves the following results: Trajectory prediction: Across 1,434 challenging samples from the Physical AI AV Dataset https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles , it records a 6.4-second minADE 6 of 0.911 m. AV reasoning: It scores 0.433 on the Physical AI AV Reasoning Benchmark https://huggingface.co/spaces/nvidia/PhysicalAI-AV-OOD-Reasoning-Challenge-2026 . LingoQA: Alpamayo 2 Super achieves 79.2 on the LingoQA benchmark https://github.com/wayveai/LingoQA/ , ranking first among 37 evaluated models. With 34 billion parameters, it leads Qwen2.5-VL 72B by 17.0 points, Qwen3-VL 32B by 7.0 points, Gemini 2.5 Pro by 15.1 points, and GPT-4o by 23.2 points. Lower minADE 6 values indicate better trajectory predictions; higher reasoning scores indicate better performance. The main challenge with open-loop metrics, however, is that they evaluate predictions against a fixed, prerecorded future and therefore don’t capture what happens after the model’s first action, which may affect the rest of the scene. For example, if the ego vehicle changes lanes, an open-loop replay may continue moving an adjacent vehicle along its recorded trajectory rather than accounting for how it would respond to the ego vehicle. Closed-loop simulation executes each predicted action within the scene and, when the simulator includes reactive behavior models, captures how surrounding agents may react. NVIDIA AlpaSim https://github.com/NVlabs/alpasim enables this by repeatedly rendering observations, querying the policy, and applying its actions so developers can measure these closed-loop effects over time. To run Alpamayo 2 Super on an AlpaSim evaluation suite, use the corresponding shell command: uv run alpasim wizard deploy=local topology=2gpu driver=alpamayo2 wizard.log dir=$PWD/tutorial eval.video.video layouts= REASONING OVERLAY with the following AlpaSim wizard configuration: Should be used in defaults list, e.g. - /driver: alpamayo2 Type validation happens at driver runtime via OmegaConf.structured merge defaults: - alpamayo configs Camera and simulation configs for 4-cam 10Hz - self YAML values override schema defaults Alpamayo 2 Super Driver Configuration for Alpasim Logging level uses wizard's global setting log level: ${wizard.log level} Model configuration model: model type: alpamayo2 Entry-point name in alpasim.models registry HuggingFace model ID requires cached download or hf authentication in the driver container checkpoint path: "nvidia/Alpamayo2-Super" Alternative local path to a pre-downloaded model checkpoint path: "/mnt/drivers/alpamayo2/Alpamayo2-Super" device: "cuda" Enable classifier-free guidance navigation sampling NOTE: this requires 2 GPUs with at least 70 GB VRAM . Set to true only when sufficient GPU memory is available. use classifier free guidance nav: false Server configuration host: "0.0.0.0" port: ??? Inference configuration inference: use cameras: - camera cross left 120fov - camera front wide 120fov - camera cross right 120fov - camera front tele 30fov max batch size: 1 A2Super is memory intensive, start with batch size 1 subsample factor: 1 context length: 4 A2Super uses 4 temporal frames per camera Route configuration — A2Super uses language-only navigation, not waypoint commands route: use waypoint commands: false Output configuration output dir: "/mnt/output/driver" Trajectory optimization disabled by default for A2Super trajectory optimizer: enabled: false plot debug images: false On 913 reconstructed scenes, Alpamayo 2 Super obtains an AlpaSim Score of 1.50 ± 0.13. This closed-loop score complements open-loop results by revealing collisions, road departures, close encounters, and other failures that can emerge only after the policy influences future observations. | Evaluation | Benchmark set | Metric | Alpamayo 2 Super | Reference scores | Better | | Open-loop | | Physical AI AV Reasoning Benchmark https://huggingface.co/spaces/nvidia/PhysicalAI-AV-OOD-Reasoning-Challenge-2026 GPT-5.5: 0.502 LingoQA https://arxiv.org/abs/2312.14115 Qwen3-VL 32B: 72.2Qwen2.5-VL 72B: 62.2 Gemini 2.5 Pro: 64.1GPT-4o: 56.0 AlpaSim https://github.com/NVlabs/alpasim 913 reconstructed scenes from the Physical AI AV NuRec https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec dataset Table 1. Open-loop and closed-loop evaluation of Alpamayo 2 Super on long-tail driving scenarios Meta-actions A trajectory is precise, but it doesn’t always provide a compact description of intent. Meta-actions summarize a plan in terms of high-level decisions such as yield, change lanes, stop , or accelerate . These outputs can help bridge an end-to-end foundation model and a modular AV stack: a downstream planner can consume the decision, an evaluator can check whether trajectory geometry agrees with intent, and teams can search their data corpus for particular maneuvers. Generate meta-actions The meta-actions notebook https://github.com/NVlabs/alpamayo2/blob/main/notebooks/meta actions.ipynb shows how to produce meta-action outputs with an example scene. For a detailed list of supported meta-actions, refer to these lists https://github.com/search?q=repo%3ANVlabs%2Falpamayo-coc-autolabeler%20ordered actions&type=code . The core inference step and outputs are shown below. python from alpamayo2 super import helper from alpamayo2 super.load physical aiavdataset import load physical aiavdataset from alpamayo2 super.models.alpamayo2 super import Alpamayo2Super from alpamayo2 super.text tasks import generate text, prepare text generation inputs data = load physical aiavdataset "030c760c-ae38-49aa-9ad8-f5650a545d26", t0 us=2000000, model = Alpamayo2Super.from pretrained "nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device map="cuda:0" task inputs = prepare text generation inputs data=data, model config=model.config, tokenizer=model.tokenizer, task="meta action", task inputs = helper.to device task inputs, "cuda" torch.cuda.manual seed all 42 with torch.autocast "cuda", dtype=torch.bfloat16 : result = generate text model, task inputs, top p=0.98, temperature=0.6, max new tokens=512, cot = result "cot" 0 meta action = result "meta action" 0 print "Chain-of-Causation:\n", cot print "\nMeta-action:\n", meta action Evaluate meta-action accuracy To evaluate meta-action accuracy, we compare Alpamayo 2 Super’s output with ground-truth labels and report the resulting classification accuracy in terms of intersection-over-union IoU for the three components of its meta-action taxonomy: lateral, longitudinal, and lane-wise. On an internal set of 94K clips with ground-truth meta-action data, Alpamayo 2 Super achieves 74.59 lateral IoU, 61.91 longitudinal IoU, and 73.55 lane-wise IoU across its meta-action taxonomy. Scene understanding Planning is only one way to use a driving foundation model. Visual question answering VQA exposes the model’s scene understanding directly through natural language. Developers can ask about key elements in the scene, how they affect driving behavior, why the ego vehicle should slow, or about other aspects of scenarios. ne-change gap. VQA is valuable for interactive debugging and data operations. It can help a developer inspect why a policy behaved a certain way, build semantic filters over large clip collections, and generate candidate annotations for human review. With surround-view input, questions can reference side and rear context that a front-view-only model would miss. Querying multi-camera scenes with VQA and 2D grounding For each clip, Alpamayo 2 Super can generate answers and spatially localize referenced actors by predicting 2D bounding boxes in the relevant camera frames. This grounding makes outputs more useful than free-form text alone. Reviewers can verify which specific object the model is referring to, automated checks can flag missing or inconsistent boxes, and downstream models can leverage e.g., through distillation these tighter links between visual evidence, reasoning, and action. The scene understanding and VQA notebook https://github.com/NVlabs/alpamayo2/blob/main/notebooks/vqa.ipynb shows how to perform VQA with the same inputs. The core inference step and outputs are shown below. python from alpamayo2 super import helper from alpamayo2 super.load physical aiavdataset import load physical aiavdataset from alpamayo2 super.models.alpamayo2 super import Alpamayo2Super from alpamayo2 super.text tasks import generate text, prepare text generation inputs data = load physical aiavdataset "ea4a6729-bf33-4997-905b-cd58774a3580", t0 us=7500000, model = Alpamayo2Super.from pretrained "nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device map="cuda:0" task inputs = prepare vqa inputs data=data, model config=model.config, tokenizer=model.tokenizer, question="Describe the driving scene and identify the key traffic elements that should influence ego behavior.", task inputs = helper.to device task inputs, "cuda" with torch.autocast "cuda", dtype=torch.bfloat16 : result = generate text model, task inputs, top p=1.0, temperature=0.1, max new tokens=1024, answer = result "answer" 0 print answer The model can return two types of output for the same input: Figure 4, above, shows a natural-language description of the scene, while Figure 5, below, shows the same model spatially localizing referenced objects by predicting 2D bounding boxes in the relevant camera frame. Evaluate VQA response and grounding quality To evaluate VQA performance, we compare Alpamayo 2 Super’s generated responses with ground truth answers across an internal set of 8K question-answer pairs. The model achieves 0.652 answer similarity higher is better , compared with Qwen3-VL 32B at 0.450. For 2D grounding, we compare the predicted bounding boxes to ground-truth annotations using IoU and obtain 0.71 compared to 0.17 for Qwen3-VL 32B. Together, these measurements evaluate whether the model both answers questions accurately and associates its responses with the correct visual evidence across the surround-view cameras. Auto-labeling Reasoning models need decision-grounded reasoning data, but labeling long-tail driving clips by hand is expensive and slow. Annotators must inspect temporal and multi-camera context, identify the causal actors, describe how they affect the ego vehicle, and keep the label consistent with the intended maneuver. Alpamayo 2 Super can serve as an offline auto-labeler that proposes this structure at scale. This can compress annotation cycles from months to days. Generate structured CoC auto-labels The CoC auto-labeling notebook https://github.com/NVlabs/alpamayo2/blob/main/notebooks/autolabeling.ipynb accepts clips in the released schema and writes one structured record per selected keyframe. python from alpamayo2 super import helper from alpamayo2 super.load physical aiavdataset import load physical aiavdataset from alpamayo2 super.models.alpamayo2 super import Alpamayo2Super from alpamayo2 super.text tasks import generate text, prepare text generation inputs data = load physical aiavdataset "b5f3756c-4f0e-4298-a1ff-cc92ed392ae0", t0 us=11000000, model = Alpamayo2Super.from pretrained "nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device map="cuda:0" task inputs = prepare text generation inputs data=data, model config=model.config, tokenizer=model.tokenizer, task="auto labeling", task inputs = helper.to device task inputs, "cuda" torch.cuda.manual seed all 42 with torch.autocast "cuda", dtype=torch.bfloat16 : result = generate text model, task inputs, top p=0.98, temperature=0.6, max new tokens=1024, auto labeling text = result "cot auto labeling" 0 auto labeling json = result "cot auto labeling json" 0 print json.dumps auto labeling json, indent=2 By default, Alpamayo 2 Super assumes access to future ego-trajectory information. However, it can also auto-label data that does not contain future trajectory information: ... same imports and model loading as above ... Get the model to predict a future trajectory of its own, and use that in auto-labeling as the "observed" future motion. trajectory inputs = helper.prepare model inputs data, model.config, model.tokenizer trajectory inputs = helper.to device trajectory inputs, "cuda" torch.cuda.manual seed all 42 with torch.autocast "cuda", dtype=torch.bfloat16 : pred xyz, pred rot, , extra = model.sample trajectories from data data=trajectory inputs, top p=0.98, temperature=0.6, num traj samples=1, diffusion kwargs={"inference step": 10}, return extra=True, future xyz = pred xyz :, 0, 0 .detach .cpu future rot = pred rot :, 0, 0 .detach .cpu In case you want to see the model's reasoning, uncomment these: trajectory cot = str extra "cot" .reshape -1 0 print "trajectory cot:\n", trajectory cot task inputs = prepare text generation inputs data=data, model config=model.config, tokenizer=model.tokenizer, task="auto labeling", This is where the predicted trajectories are passed in: future xyz=future xyz, future rot=future rot, task inputs = helper.to device task inputs, "cuda" torch.cuda.manual seed all 42 with torch.autocast "cuda", dtype=torch.bfloat16 : result = generate text model, task inputs, top p=0.98, temperature=0.6, max new tokens=1024, auto labeling text = result "cot auto labeling" 0 auto labeling json = result "cot auto labeling json" 0 print json.dumps auto labeling json, indent=2 Evaluate CoC auto-labeling quality To evaluate CoC auto-labeling quality, we compare Alpamayo 2 Super’s generated labels with expert annotations on 8k internal clips. We use an internal judge model to assess similarity. Alpamayo 2 Super achieves a 0.652 similarity score, compared with Qwen3-VL 32B at 0.450. These results show its potential to generate structured reasoning labels at scale while maintaining consistency with expert-authored annotations. Build with Alpamayo 2 Super Alpamayo 2 Super brings surround-view perception, reasoning, planning, scene understanding, and data auto-labeling into one open model workflow. These capabilities give developers a practical foundation for building teacher models, curating long-tail data, inspecting policy decisions, and evaluating AV systems beyond a single open-loop trajectory metric. As a teacher model, it can be distilled into compact models that run on NVIDIA DRIVE AGX Thor inside the vehicle. Explore the model on Hugging Face https://huggingface.co/nvidia/Alpamayo2-Super , run the inference notebooks https://github.com/NVlabs/alpamayo2 , and share what you build on the Alpamayo developer forum https://forums.developer.nvidia.com/c/autonomous-vehicles/alpamayo/766 . For more details on the broader updates introduced as part of the Alpamayo 2 release, see the associated Hugging Face blog post https://huggingface.co/blog/drmapavone/nvidia-alpamayo-2 .