{"slug": "generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2", "title": "Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super", "summary": "NVIDIA released Alpamayo 2 Super, an open 34-billion-parameter reasoning vision-language-action model for autonomous vehicle development, combining a 32-billion-parameter Cosmos 3 Super Reasoner with a 2-billion-parameter diffusion-based Action Expert. The model generates future trajectories, Chain-of-Causation reasoning traces, meta-actions, grounded answers, and auto-labels across up to seven cameras, and is available on Hugging Face under the OpenMDW-1.1 license. It aims to unify multiple AV development stages into a single foundation model, supporting offline policy teaching, evaluation, data labeling, and task customization.", "body_md": "Autonomous vehicle (AV) development often relies on separate models for trajectory generation, high-level intent prediction, scene understanding, and data labeling. This separation makes it hard to compare related outputs, investigate model behavior, and reuse the same representations across the development workflow.\n\n[NVIDIA Alpamayo 2 Super](https://huggingface.co/nvidia/Alpamayo2-Super) is an open 34-billion-parameter reasoning vision-language-action (VLA) model designed to accelerate autonomous vehicle (AV) development. It combines the 32-billion-parameter NVIDIA [Cosmos 3 Super](https://huggingface.co/nvidia/Cosmos3-Super) Reasoner with a 2-billion-parameter diffusion-based Action Expert and is post-trained with reinforcement learning. The reasoner interprets multi-camera video, language context, and prior motion history, while the Action Expert converts the model’s resulting internal representation into a future ego-vehicle trajectory.\n\nAlpamayo 2 Super’s perception expands to 360-degree coverage across up to seven cameras and can return multiple complementary outputs: future trajectories, Chain-of-Causation (CoC) reasoning traces, high-level meta-actions, grounded answers to questions about the scene, and reasoning auto-labels.\n\nThis multi-task design gives AV developers a common foundation across multiple stages of the development workflow. The same foundation model can be used as an offline policy teacher, an evaluation critic, a data engine, or a starting point for new task customization, instead of maintaining a separate model for each stage of the workflow.\n\nThis post provides a hands-on introduction to four Alpamayo 2 Super-enabled workflows:\n\n- Generate trajectories and CoC reasoning traces, evaluating the results in open-loop and closed-loop benchmarks.\n- Predict meta-actions such as yield, change lanes, and stop alongside a trajectory.\n- Ask natural-language questions about a multi-camera driving scene.\n- Generate CoC auto-labels with 2D grounding on your own clips.\n\nThe model weights are available on [Hugging Face](https://huggingface.co/nvidia/Alpamayo2-Super) and the inference notebooks on [GitHub](https://github.com/NVlabs/alpamayo2). The model is released under OpenMDW-1.1, the Linux Foundation permissive license for open model distributions, which covers fine-tuning, derivative models, and commercial redistribution. Distilled models can be deployed commercially without further permission from NVIDIA, and model outputs carry no license conditions.\n\n## Planning and reasoning\n\nReasoning through new scenarios is a fundamental problem in autonomous driving. Navigating construction zones, partially occluded pedestrians, unusual right-of-way interactions, and objects entering the roadway requires more than matching a common trajectory pattern. A useful driving model must identify scene context that matters, connect it to the appropriate driving decision, and produce an action consistent with that decision.\n\n### Trajectories and CoC traces: What and why\n\nAlpamayo 2 Super, like its predecessors, jointly produces output trajectories and CoC reasoning traces. The trajectory expresses *what* the ego vehicle could do next. The CoC reasoning trace provides insights into *why* a driving decision was made from observed scene context. Returning both outputs makes it easier to understand the model’s decision-making, curate difficult cases, compare a deployed policy with a larger teacher, and diagnose whether a failure originated in perception, reasoning, or action generation. CoC traces also feed into the NVIDIA Halos safety validation workflows by enabling introspection into the model’s understanding of the scene.\n\nThe Alpamayo 2 Super repository’s [inference notebook](https://github.com/NVlabs/alpamayo2/blob/main/notebooks/inference.ipynb) loads a surround-view clip, prepares ego-motion history, and samples a trajectory with its associated CoC trace. The core inference step and outputs are shown below.\n\n``` python\nfrom alpamayo2_super import helper\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset\nfrom alpamayo2_super.models.alpamayo2_super import Alpamayo2Super\nfrom alpamayo2_super.visualization import plot_inference_result\n\ndata = load_physical_aiavdataset(\n  \"030c760c-ae38-49aa-9ad8-f5650a545d26\",\n   t0_us=2000000,\n)\n\nmodel = Alpamayo2Super.from_pretrained(\"nvidia/Alpamayo2-Super\", dtype=torch.bfloat16, device_map=\"cuda:0\")\nmodel_inputs = helper.prepare_model_inputs(data, model.config, model.tokenizer)\nmodel_inputs = helper.to_device(model_inputs, \"cuda\")\n\ntorch.cuda.manual_seed_all(42)\nwith torch.autocast(\"cuda\", dtype=torch.bfloat16):\n   pred_xyz, pred_rot, logprob, extra = model.sample_trajectories_from_data(\n       data=model_inputs,\n       top_p=0.98,\n       temperature=0.6,\n       num_traj_samples=1,\n       diffusion_kwargs={\"inference_step\": 10},\n       return_extra=True,\n   )\n\nfig, metadata = plot_inference_result(\n   data=data,\n   pred_xyz=pred_xyz,\n   extra=extra,\n)\n```\n\n## Evaluation methods\n\nTo evaluate the model’s output reasoning and trajectory quality, we can use open-loop and closed-loop evaluation methods. **Open-loop evaluation** measures trajectory and reasoning quality on recorded scenes by comparing them to ground-truth labels.\n\nAlpamayo 2 Super achieves the following results:\n\n**Trajectory prediction:** Across 1,434 challenging samples from the[Physical AI AV Dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles), it records a 6.4-second`minADE_6`\n\nof 0.911 m.**AV reasoning:** It scores 0.433 on the[Physical AI AV Reasoning Benchmark](https://huggingface.co/spaces/nvidia/PhysicalAI-AV-OOD-Reasoning-Challenge-2026).**LingoQA:** Alpamayo 2 Super achieves 79.2 on the[LingoQA benchmark](https://github.com/wayveai/LingoQA/), ranking first among 37 evaluated models. With 34 billion parameters, it leads Qwen2.5-VL (72B) by 17.0 points, Qwen3-VL (32B) by 7.0 points, Gemini 2.5 Pro by 15.1 points, and GPT-4o by 23.2 points.\n\nLower `minADE_6`\n\nvalues indicate better trajectory predictions; higher reasoning scores indicate better performance.\n\nThe main challenge with open-loop metrics, however, is that they evaluate predictions against a fixed, prerecorded future and therefore don’t capture what happens after the model’s first action, which may affect the rest of the scene.\n\nFor example, if the ego vehicle changes lanes, an open-loop replay may continue moving an adjacent vehicle along its recorded trajectory rather than accounting for how it would respond to the ego vehicle.\n\n**Closed-loop simulation** executes each predicted action within the scene and, when the simulator includes reactive behavior models, captures how surrounding agents may react. [NVIDIA AlpaSim](https://github.com/NVlabs/alpasim) enables this by repeatedly rendering observations, querying the policy, and applying its actions so developers can measure these closed-loop effects over time.\n\nTo run Alpamayo 2 Super on an AlpaSim evaluation suite, use the corresponding shell command:\n\n```\nuv run alpasim_wizard deploy=local topology=2gpu driver=alpamayo2 \nwizard.log_dir=$PWD/tutorial eval.video.video_layouts=[REASONING_OVERLAY]\n```\n\nwith the following AlpaSim wizard configuration:\n\n```\n# Should be used in defaults list, e.g.\n# - /driver: alpamayo2\n# Type validation happens at driver runtime via OmegaConf.structured merge\n\ndefaults:\n  - alpamayo_configs    # Camera and simulation configs for 4-cam 10Hz\n  - _self_                      # YAML values override schema defaults\n\n# Alpamayo 2 Super Driver Configuration for Alpasim\n\n# Logging level (uses wizard's global setting)\nlog_level: ${wizard.log_level}\n\n# Model configuration\nmodel:\n  model_type: alpamayo2  # Entry-point name in alpasim.models registry\n  # HuggingFace model ID (requires cached download or hf authentication in the driver container)\n  checkpoint_path: \"nvidia/Alpamayo2-Super\"\n  # # Alternative local path to a pre-downloaded model\n  # checkpoint_path: \"/mnt/drivers/alpamayo2/Alpamayo2-Super\"\n  device: \"cuda\"\n  # Enable classifier-free guidance navigation sampling (NOTE: this requires 2 GPUs with at least 70 GB VRAM).\n  # Set to true only when sufficient GPU memory is available.\n  use_classifier_free_guidance_nav: false\n\n# Server configuration\nhost: \"0.0.0.0\"\nport: ???\n\n# Inference configuration\ninference:\n  use_cameras:\n     - camera_cross_left_120fov\n     - camera_front_wide_120fov\n     - camera_cross_right_120fov\n     - camera_front_tele_30fov\n  max_batch_size: 1  # A2Super is memory intensive, start with batch size 1\n  subsample_factor: 1\n  context_length: 4  # A2Super uses 4 temporal frames per camera\n\n# Route configuration — A2Super uses language-only navigation, not waypoint commands\nroute:\n  use_waypoint_commands: false\n\n# Output configuration\noutput_dir: \"/mnt/output/driver\"\n\n# Trajectory optimization (disabled by default for A2Super)\ntrajectory_optimizer:\n  enabled: false\n\nplot_debug_images: false\n```\n\nOn 913 reconstructed scenes, Alpamayo 2 Super obtains an AlpaSim Score of 1.50 ± 0.13. This closed-loop score complements open-loop results by revealing collisions, road departures, close encounters, and other failures that can emerge only after the policy influences future observations.\n\n| Evaluation | Benchmark (set) | Metric | Alpamayo 2 Super | Reference scores | Better |\n| Open-loop |\n|\n\n[Physical AI AV Reasoning Benchmark](https://huggingface.co/spaces/nvidia/PhysicalAI-AV-OOD-Reasoning-Challenge-2026)GPT-5.5: 0.502\n\n[LingoQA](https://arxiv.org/abs/2312.14115)Qwen3-VL 32B: 72.2Qwen2.5-VL 72B: 62.2\n\nGemini 2.5 Pro: 64.1GPT-4o: 56.0\n\n[AlpaSim](https://github.com/NVlabs/alpasim)(913 reconstructed scenes from the[Physical AI AV NuRec](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec)dataset)*Table 1. Open-loop and closed-loop evaluation of Alpamayo 2 Super on long-tail driving scenarios*\n\n## Meta-actions\n\nA trajectory is precise, but it doesn’t always provide a compact description of intent. Meta-actions summarize a plan in terms of high-level decisions such as *yield, change lanes, stop*, or *accelerate*. These outputs can help bridge an end-to-end foundation model and a modular AV stack: a downstream planner can consume the decision, an evaluator can check whether trajectory geometry agrees with intent, and teams can search their data corpus for particular maneuvers.\n\n### Generate meta-actions\n\nThe [meta-actions notebook](https://github.com/NVlabs/alpamayo2/blob/main/notebooks/meta_actions.ipynb) shows how to produce meta-action outputs with an example scene. For a detailed list of supported meta-actions, refer to [these lists](https://github.com/search?q=repo%3ANVlabs%2Falpamayo-coc-autolabeler%20ordered_actions&type=code). The core inference step and outputs are shown below.\n\n``` python\nfrom alpamayo2_super import helper\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset\nfrom alpamayo2_super.models.alpamayo2_super import Alpamayo2Super\nfrom alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs\n\ndata = load_physical_aiavdataset(\n  \"030c760c-ae38-49aa-9ad8-f5650a545d26\",\n   t0_us=2000000,\n)\n\nmodel = Alpamayo2Super.from_pretrained(\"nvidia/Alpamayo2-Super\", dtype=torch.bfloat16, device_map=\"cuda:0\")\ntask_inputs = prepare_text_generation_inputs(\n   data=data,\n   model_config=model.config,\n   tokenizer=model.tokenizer,\n   task=\"meta_action\",\n)\ntask_inputs = helper.to_device(task_inputs, \"cuda\")\n\ntorch.cuda.manual_seed_all(42)\nwith torch.autocast(\"cuda\", dtype=torch.bfloat16):\n   result = generate_text(\n       model,\n       task_inputs,\n       top_p=0.98,\n       temperature=0.6,\n       max_new_tokens=512,\n   )\n\ncot = result[\"cot\"][0]\nmeta_action = result[\"meta_action\"][0]\nprint(\"Chain-of-Causation:\\n\", cot)\nprint(\"\\nMeta-action:\\n\", meta_action)\n```\n\n### Evaluate meta-action accuracy\n\nTo evaluate meta-action accuracy, we compare Alpamayo 2 Super’s output with ground-truth labels and report the resulting classification accuracy in terms of intersection-over-union (IoU) for the three components of its meta-action taxonomy: lateral, longitudinal, and lane-wise.\n\nOn an internal set of 94K clips with ground-truth meta-action data, Alpamayo 2 Super achieves 74.59 lateral IoU, 61.91 longitudinal IoU, and 73.55 lane-wise IoU across its meta-action taxonomy.\n\n## Scene understanding\n\nPlanning is only one way to use a driving foundation model. Visual question answering (VQA) exposes the model’s scene understanding directly through natural language. Developers can ask about key elements in the scene, how they affect driving behavior, why the ego vehicle should slow, or about other aspects of scenarios.\n\nne-change gap.\n\nVQA is valuable for interactive debugging and data operations. It can help a developer inspect why a policy behaved a certain way, build semantic filters over large clip collections, and generate candidate annotations for human review. With surround-view input, questions can reference side and rear context that a front-view-only model would miss.\n\n### Querying multi-camera scenes with VQA and 2D grounding\n\nFor each clip, Alpamayo 2 Super can generate answers and spatially localize referenced actors by predicting 2D bounding boxes in the relevant camera frames. This grounding makes outputs more useful than free-form text alone. Reviewers can verify which specific object the model is referring to, automated checks can flag missing or inconsistent boxes, and downstream models can leverage (e.g., through distillation) these tighter links between visual evidence, reasoning, and action.\n\nThe [scene understanding and VQA notebook](https://github.com/NVlabs/alpamayo2/blob/main/notebooks/vqa.ipynb) shows how to perform VQA with the same inputs. The core inference step and outputs are shown below.\n\n``` python\nfrom alpamayo2_super import helper\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset\nfrom alpamayo2_super.models.alpamayo2_super import Alpamayo2Super\nfrom alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs\n\ndata = load_physical_aiavdataset(\n  \"ea4a6729-bf33-4997-905b-cd58774a3580\",\n   t0_us=7500000,\n)\n\nmodel = Alpamayo2Super.from_pretrained(\"nvidia/Alpamayo2-Super\", dtype=torch.bfloat16, device_map=\"cuda:0\")\ntask_inputs = prepare_vqa_inputs(\n    data=data,\n    model_config=model.config,\n    tokenizer=model.tokenizer,\n    question=\"Describe the driving scene and identify the key traffic elements that should influence ego behavior.\",\n)\ntask_inputs = helper.to_device(task_inputs, \"cuda\")\nwith torch.autocast(\"cuda\", dtype=torch.bfloat16):\n    result = generate_text(\n        model,\n        task_inputs,\n        top_p=1.0,\n        temperature=0.1,\n        max_new_tokens=1024,\n    )\nanswer = result[\"answer\"][0]\nprint(answer)\n```\n\nThe model can return two types of output for the same input: Figure 4, above, shows a natural-language description of the scene, while Figure 5, below, shows the same model spatially localizing referenced objects by predicting 2D bounding boxes in the relevant camera frame.\n\n### Evaluate VQA response and grounding quality\n\nTo evaluate VQA performance, we compare Alpamayo 2 Super’s generated responses with ground truth answers across an internal set of 8K question-answer pairs. The model achieves 0.652 answer similarity (higher is better), compared with Qwen3-VL 32B at 0.450. For 2D grounding, we compare the predicted bounding boxes to ground-truth annotations using IoU and obtain 0.71 compared to 0.17 for Qwen3-VL 32B. Together, these measurements evaluate whether the model both answers questions accurately and associates its responses with the correct visual evidence across the surround-view cameras.\n\n## Auto-labeling\n\nReasoning models need decision-grounded reasoning data, but labeling long-tail driving clips by hand is expensive and slow. Annotators must inspect temporal and multi-camera context, identify the causal actors, describe how they affect the ego vehicle, and keep the label consistent with the intended maneuver. Alpamayo 2 Super can serve as an offline auto-labeler that proposes this structure at scale. This can compress annotation cycles from months to days.\n\n### Generate structured CoC auto-labels\n\nThe [CoC auto-labeling notebook](https://github.com/NVlabs/alpamayo2/blob/main/notebooks/autolabeling.ipynb) accepts clips in the released schema and writes one structured record per selected keyframe.\n\n``` python\nfrom alpamayo2_super import helper\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset\nfrom alpamayo2_super.models.alpamayo2_super import Alpamayo2Super\nfrom alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs\n\ndata = load_physical_aiavdataset(\n  \"b5f3756c-4f0e-4298-a1ff-cc92ed392ae0\",\n   t0_us=11000000,\n)\n\nmodel = Alpamayo2Super.from_pretrained(\"nvidia/Alpamayo2-Super\", dtype=torch.bfloat16, device_map=\"cuda:0\")\n\ntask_inputs = prepare_text_generation_inputs(\n   data=data,\n   model_config=model.config,\n   tokenizer=model.tokenizer,\n   task=\"auto_labeling\",\n)\ntask_inputs = helper.to_device(task_inputs, \"cuda\")\n\ntorch.cuda.manual_seed_all(42)\nwith torch.autocast(\"cuda\", dtype=torch.bfloat16):\n   result = generate_text(\n       model,\n       task_inputs,\n       top_p=0.98,\n       temperature=0.6,\n       max_new_tokens=1024,\n   )\n\nauto_labeling_text = result[\"cot_auto_labeling\"][0]\nauto_labeling_json = result[\"cot_auto_labeling_json\"][0]\nprint(json.dumps(auto_labeling_json, indent=2))\n```\n\nBy default, Alpamayo 2 Super assumes access to future ego-trajectory information. However, it can also auto-label data that *does not* contain future trajectory information:\n\n```\n# ... same imports and model loading as above ...\n\n# Get the model to predict a future trajectory of its own, and use that in auto-labeling as the \"observed\" future motion.\ntrajectory_inputs = helper.prepare_model_inputs(data, model.config, model.tokenizer)\ntrajectory_inputs = helper.to_device(trajectory_inputs, \"cuda\")\n\ntorch.cuda.manual_seed_all(42)\nwith torch.autocast(\"cuda\", dtype=torch.bfloat16):\n   pred_xyz, pred_rot, _, extra = model.sample_trajectories_from_data(\n        data=trajectory_inputs,\n        top_p=0.98,\n        temperature=0.6,\n        num_traj_samples=1,\n        diffusion_kwargs={\"inference_step\": 10},\n        return_extra=True,\n   )\n   future_xyz = pred_xyz[:, 0, 0].detach().cpu()\n   future_rot = pred_rot[:, 0, 0].detach().cpu()\n   # In case you want to see the model's reasoning, uncomment these:\n   # trajectory_cot = str(extra[\"cot\"].reshape(-1)[0])\n   # print(\"trajectory_cot:\\n\", trajectory_cot)\n\ntask_inputs = prepare_text_generation_inputs(\n   data=data,\n   model_config=model.config,\n   tokenizer=model.tokenizer,\n   task=\"auto_labeling\",\n   # This is where the predicted trajectories are passed in: \n   future_xyz=future_xyz,\n   future_rot=future_rot,\n)\ntask_inputs = helper.to_device(task_inputs, \"cuda\")\n\ntorch.cuda.manual_seed_all(42)\nwith torch.autocast(\"cuda\", dtype=torch.bfloat16):\n   result = generate_text(\n       model,\n       task_inputs,\n       top_p=0.98,\n       temperature=0.6,\n       max_new_tokens=1024,\n   )\n\nauto_labeling_text = result[\"cot_auto_labeling\"][0]\nauto_labeling_json = result[\"cot_auto_labeling_json\"][0]\nprint(json.dumps(auto_labeling_json, indent=2))\n```\n\n### Evaluate CoC auto-labeling quality\n\nTo evaluate CoC auto-labeling quality, we compare Alpamayo 2 Super’s generated labels with expert annotations on 8k internal clips. We use an internal judge model to assess similarity. Alpamayo 2 Super achieves a 0.652 similarity score, compared with Qwen3-VL 32B at 0.450. These results show its potential to generate structured reasoning labels at scale while maintaining consistency with expert-authored annotations.\n\n## Build with Alpamayo 2 Super\n\nAlpamayo 2 Super brings surround-view perception, reasoning, planning, scene understanding, and data auto-labeling into one open model workflow. These capabilities give developers a practical foundation for building teacher models, curating long-tail data, inspecting policy decisions, and evaluating AV systems beyond a single open-loop trajectory metric. As a teacher model, it can be distilled into compact models that run on NVIDIA DRIVE AGX Thor inside the vehicle.\n\nExplore the model on [Hugging Face](https://huggingface.co/nvidia/Alpamayo2-Super), run the [inference notebooks](https://github.com/NVlabs/alpamayo2), and share what you build on the [Alpamayo developer forum](https://forums.developer.nvidia.com/c/autonomous-vehicles/alpamayo/766).\n\nFor more details on the broader updates introduced as part of the Alpamayo 2 release, see [the associated Hugging Face blog post](https://huggingface.co/blog/drmapavone/nvidia-alpamayo-2).", "url": "https://wpnews.pro/news/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2", "canonical_source": "https://developer.nvidia.com/blog/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2-super/", "published_at": "2026-08-04 15:00:00+00:00", "updated_at": "2026-08-04 15:35:22.758164+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "autonomous-vehicles", "generative-ai", "ai-research"], "entities": ["NVIDIA", "Alpamayo 2 Super", "Cosmos 3 Super", "Hugging Face", "GitHub", "NVIDIA Halos", "OpenMDW-1.1", "Linux Foundation"], "alternates": {"html": "https://wpnews.pro/news/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2", "markdown": "https://wpnews.pro/news/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2.md", "text": "https://wpnews.pro/news/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2.txt", "jsonld": "https://wpnews.pro/news/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2.jsonld"}}