{"slug": "running-av-reasoning-models-within-onboard-compute-and-latency-limits", "title": "Running AV Reasoning Models Within Onboard Compute and Latency Limits", "summary": "NVIDIA released Alpamayo 2 Edge, a 2B-parameter adaptive reasoning vision-language-action model built on a Nemotron-based backbone and paired with a diffusion action expert, designed to run in real time on NVIDIA DRIVE AGX Thor onboard compute. NVIDIA reports that closed-loop reinforcement learning post-training in AlpaGym improved closed-loop driving performance by 48.9% over the best checkpoint without closed-loop RL, with speculative decoding via DFlash2 and TensorRT quantization recipes used to cut inference cost. The release targets AV developers who need reasoning-based driving models that fit onboard compute and latency budgets.", "body_md": "Autonomous vehicle (AV) developers need reasoning models that can operate within the compute and latency constraints of an onboard system. Larger models provide a foundation for developing these capabilities, but moving them into a vehicle requires additional work to reduce model size, preserve useful reasoning, and validate performance on the target hardware.\n\n[NVIDIA Alpamayo 2 Edge](https://huggingface.co/nvidia/Alpamayo2-Edge) addresses this need with an adaptive reasoning vision-language-action (VLA) model designed for real-time operation on [NVIDIA DRIVE AGX Thor](https://developer.nvidia.com/drive/agx/). Built on the 2B Nemotron-based reasoning backbone of [NVIDIA Cosmos 3 Edge](https://www.nvidia.com/en-us/ai/cosmos/) and paired with a diffusion action expert, it leverages surround-camera observations, temporal context, and navigation inputs to generate Chain-of-Causation (CoC) reasoning and future driving trajectories. It reasons *adaptively*, generating reasoning only when needed, reducing unnecessary token generation and conserving inference compute during routine driving. \n\nAlpamayo 2 Edge is post-trained with closed-loop reinforcement learning (RL) in NVIDIA [AlpaGym](https://github.com/NVlabs/alpagym), improving closed-loop driving performance by 48.9% on top of the best-performing checkpoint without closed-loop RL. Together, these capabilities give developers a strong, edge-ready foundation for bringing reasoning-based driving into an onboard stack.\n\nMaking a reasoning model compact enough for onboard deployment introduces a key challenge: preserving the quality of its reasoning and driving behavior as model size is reduced. This requires a way to measure reasoning quality and use that feedback to improve the model. NVIDIA AlpaJudge is the human-calibrated evaluation and reward model used in Alpamayo 2 Edge development to evaluate reasoning traces, select higher-value training data, and provide RL rewards.\n\nThis post introduces Alpamayo 2 Edge, explains its inference optimization on DRIVE AGX Thor, and explores three ways to use AlpaJudge in AV development workflows:\n\n- Run AlpaJudge-RB and AlpaJudge-RF on a PhysicalAI-AV sample, inspect their scores, and adapt the evaluation to policy-generated reasoning\n- Select training examples that expose useful learning opportunities\n- Use human-aligned reasoning scores as rewards for post-training\n\nTogether, these workflows connect an edge-oriented driving model with a repeatable process for measuring and improving it.\n\n## Bring reasoning-based driving to the edge\n\nAn onboard driving model must interpret a changing scene and produce a useful trajectory within the system’s inference budget. The key question is how much driving and reasoning capability a model retains under that constraint.\n\nTo balance model capacity and efficiency, Alpamayo 2 Edge leverages multiple new features:\n\n- A 2B-parameter Nemotron-based reasoning backbone, sized for onboard inference\n- Adaptive reasoning, reducing unnecessary token generation and conserving inference compute during routine driving\n- Offline distillation, training with CoC reasoning autolabels generated by Alpamayo 2 Super\n- Closed-loop reinforcement learning post-training in [AlpaGym](https://github.com/NVlabs/alpagym) , in which the model learns from the consequences of its actions end-to-end in simulation\n- Multi-resolution inputs to efficiently encode surround-view observations online\n- Speculative decoding with [DFlash2](https://github.com/z-lab/dflash) to accelerate online reasoning without sacrificing quality\n- Recipes for quantization and inference acceleration with NVIDIA TensorRT and NVIDIA TensorRT-Edge-LLM\n\nFigure 2, below, shows how these choices translate into capability, comparing Alpamayo 2 Edge with Qwen-Drive-1.0 across closed-loop driving, open-loop trajectory prediction, and scene understanding and reasoning.\n\nEarly partner feedback also highlights Alpamayo 2 Edge’s driving performance and inference speed:\n\n*“Within days of getting early access to Alpamayo 2 Edge, we had it running closed-loop in NuRec reconstructions of our own London street recordings. In our early simulation tests, Edge kept its lane in more than twice as many runs as Alpamayo 1.5, recovered from lane offsets about 40% more often, and out of the box planned roughly twice as fast as our hand-tuned Alpamayo 1.5 server on the same GPU.”* *—**Metas Simonavicius, CTO, PORT***\n\n*“Alpamayo 2 Edge demonstrated impressive generalization capabilities, reducing displacement error (minADE) by 13.6% and lateral prediction error (RMSE) at 3 seconds by 46.1% compared to Alpamayo 2 Super in our zero-shot evaluation, despite substantial differences in sensor configurations. Fine-tuning on our own driving data delivered a further 44.2% reduction in displacement error compared to the Alpamayo 2 Edge baseline. Within just a few days, we went from receiving the model to running our first tests on a real vehicle.”* — *Luca Crotti, co-founder and motion planning lead, Niulinx*\n\n*“In less than 48 hours, we made NVIDIA’s Alpamayo 2 Edge run at 10 Hz on AGX Thor for real-time planning.”—* *Hans Salomonsson, CEO and co-founder,* *Embedl*\n\nThe Alpamayo 2 Edge repository’s [inference script](https://github.com/NVlabs/alpamayo2-edge/tree/main/src/alpamayo2_edge/inference.py) shows how the model operates: It takes in surround-view camera observations over time, navigation text, and historical ego-motion to produce CoC reasoning traces and future trajectories.\n\n``` python\nimport torch\nimport transformers\n\nfrom alpamayo2_edge.checkpoint import load_model\nfrom alpamayo2_edge.load_physical_aiavdataset import load_physical_aiavdataset\nfrom alpamayo2_edge.helper import prepare_model_inputs, to_device\nfrom alpamayo2_edge.models.utils import fuse_traj_tokens\n\ndata = load_physical_aiavdataset(**clip)\nmodel, loading_info = load_model(\"nvidia/Alpamayo2-Edge\")\n\nsettings = dict(\n    seed=43,\n    top_p=0.98,\n    top_k=None,\n    temperature=0.6,\n    num_traj_sets=1,\n    num_traj_samples=1,\n    max_generation_length=256,\n    diffusion_kwargs={\"inference_step\": 10},\n)\n\nprepared = prepare_model_inputs(data, model.config, model.tokenizer)\nprepared = to_device(prepared, device=\"cuda\")\n\n\"\"\"Seed text and diffusion sampling, then run the model on prepared inputs.\"\"\"\ntorch.manual_seed(settings[\"seed\"])\ngenerator = torch.Generator(device=\"cuda\").manual_seed(settings[\"seed\"])\nwith (\n    torch.inference_mode(),\n    torch.autocast(\n        device_type=torch.device(\"cuda\").type,\n        dtype=model.dtype,\n        enabled=model.dtype != torch.float32,\n    ),\n):\n    xyz, rot, _, extra = model.sample_trajectories_from_data(\n        prepared,\n        num_traj_sets=settings[\"num_traj_sets\"],\n        num_traj_samples=settings[\"num_traj_samples\"],\n        top_p=settings[\"top_p\"],\n        top_k=settings[\"top_k\"],\n        temperature=settings[\"temperature\"],\n        max_generation_length=settings[\"max_generation_length\"],\n        diffusion_kwargs={**settings[\"diffusion_kwargs\"], \"generator\": generator},\n        return_extra=True,\n    )\n```\n\nIn particular, note that the model will now produce special `<NOMINAL>` tokens instead of reasoning traces sometimes, indicating routine driving and conserving inference compute.\n\n### Retaining quality during deployment\n\nWhen downloading the model from Hugging Face, by default, Alpamayo 2 Edge’s weights are represented in BF16 and it is directly runnable as a PyTorch model on any supported NVIDIA GPU. To run at real-time on DRIVE AGX Thor, however, Alpamayo 2 Edge must be further inference-optimized.\n\nThe [Alpamayo Developer Hub](https://github.com/NVlabs/alpamayo-recipes) includes the recipe for the DRIVE AGX Thor deployment workflow: stage the Alpamayo 2 Edge checkpoint and DFlash2 drafter, calibrate FP8 scales, export ONNX graphs, build TensorRT engines with TensorRT Edge-LLM plugins, and run the Python harness. The recipe’s reference configuration was measured on DRIVE AGX Thor-X with DRIVE OS 7.2.7.0, CUDA 13.4, and TensorRT 11.3.1.9.\n\nAfter following the Alpamayo 2 Edge deployment recipe to stage the model, evaluation inputs, and engines, run the bundle with\n\n```\npython -m alpamayo2edge_edgellm.runtime.run \\\n  --engines \"$WORK/engines\" \\\n  --model \"$MODEL\" \\\n  --reference \"$INPUTS\" \\\n  --dflash2 \\\n  --rolling \\\n  --warmup 1 \\\n  --repeat 5 \\\n  --output \"$WORK/results.json\"\n```\n\nThe recipe documents export, build, calibration provenance, board requirements, and per-stage measurements so developers can reproduce the latency and paired-quality check on their DRIVE AGX Thor configuration. For reference, the resulting optimized model has a pixel-to-trajectory inference latency of 160 ms to 190 ms depending on the amount of reasoning. The same recipe check compares optimized model outputs with a BF16 PyTorch reference built from identical inputs and seeded action noise.\n\n## Evaluate reasoning with AlpaJudge\n\nTrajectory metrics describe how a model plans or drives. Reasoning evaluation adds another dimension: whether the model’s explanation identifies the relevant scene context and supports the decision it describes. [AlpaJudge](https://huggingface.co/nvidia/alpajudge) provides the human-calibrated critic for this aspect of Alpamayo 2 Edge development (as well as prior models). The following workflows describe how to conduct that evaluation and use it for model improvement.\n\n### Choose the judge for your available inputs\n\nAlpaJudge provides two checkpoint variants. Choose the variant according to the evidence available for the example you want to evaluate.\n\n| Variant | Required inputs | Returned assessment | \n| AlpaJudge-RB, reference-based | Question, reference answer, and candidate answer; no images | One alignment probability in [0, 1] | \n| AlpaJudge-RF, reference-free | Three front-wide and three front-tele frames, ego meta-action history and future, and a candidate rationale | Five rubric probabilities plus action, component, and overall scores | \n\n*Table 1. AlpaJudge variants use different evidence to assess a candidate rationale*\n\nRB measures agreement with the supplied reference, rather than independently checking the scene. RF does not require a reference rationale, but it does require synchronized visual evidence and formatted meta-action context.\n\n### Workflow 1: Run AlpaJudge on a PhysicalAI-AV sample\n\n[The example notebook](https://github.com/NVlabs/alpajudge/blob/main/examples/alpajudge_pai.ipynb) provides a useful starting point, visualizing the data, meta-action context, candidate rationale, and reference rationale tied to a PhysicalAI-AV sample. The same record can then be rescored with a policy-generated CoC trace. Preserve clip ID, timestamp, camera names, and action context in the output so a low score can be traced back to the scene.\n\nIn open-loop evaluation, the candidate explanation and trajectory can be compared with recorded annotations. In closed-loop evaluation, the explanation at each policy decision can be scored using the observation and action context from that rollout. A high reasoning score is evidence that a rationale matches the judge’s rubric, while collision, road-departure, close-encounter, and trajectory measures describe different aspects of driving behavior. Report them together.\n\n#### Step 1: Set up the package and open the notebook\n\nFrom the AlpaJudge repository root, install the locked environment, authenticate for the model and PhysicalAI-AV dataset, and open the example notebook:\n\n```\nuv sync --locked --dev\nhf auth login\nuv run jupyter lab examples/alpajudge_pai.ipynb\n```\n\nThe notebook uses clip `030c760c-ae38-49aa-9ad8-f5650a545d26` at 5.1 s. It decodes front-wide and front-tele frames at −1.0, −0.5, and 0.0 s, oldest first. The example rationale and meta-action strings are manually authored; the dataset supplies the camera frames. Replace all text fields together when using a different moment.\n\n#### Step 2: Download the two judge checkpoints\n\nThe notebook downloads `nvidia/alpajudge` once and selects the `reference-based/` and `reference-free/` subfolders. These are separate checkpoints with separate loaders. Its snapshot call filters to those two folders; set `ALPAJUDGE_MODEL_REVISION` to an immutable revision for a reproducible run. A model load needs a CUDA GPU and roughly 5 GB for weights before activations.\n\n```\ncheckpoint_root = snapshot_download(\n    repo_id=\"nvidia/alpajudge\",\n    revision=MODEL_REVISION,\n    allow_patterns=[\"reference-based/*\", \"reference-free/*\"],\n)\n```\n\n#### Step 3: Score the sample and inspect both outputs\n\nRun the notebook’s fixed construction-zone sample first. With its recorded checkpoint and inputs, the smoke test reports RB alignment 0.999 and RF overall 0.842. The five RF subscores assess action/reference agreement, action safety, component grounding, component completeness, and causal consistency. The RF model defines action = p_match × (1 + p_safe)/2 and component = mean(p_grounded, p_complete, p_causal), then combines them overall = 0.6 × action + 0.4 × component.\n\nCore notebook calls:\n\n```\nrb_model, rb_tokenizer = load_reference_based(\n    str(checkpoint_root),\n    subfolder=\"reference-based\",\n    device=\"cuda\",\n)\nrb = score_reference_based(\n    rb_model,\n    rb_tokenizer,\n    [{\n        \"question\": QUESTION,\n        \"answer\": REFERENCE_RATIONALE,\n        \"student\": CANDIDATE_RATIONALE,\n    }],\n)[0].item()\n\nrf_model, rf_processor = load_reference_free(\n    str(checkpoint_root),\n    subfolder=\"reference-free\",\n    device=\"cuda\",\n)\nrf = score_reference_free(\n    rf_model,\n    rf_processor,\n    front_wide_frames=front_wide_frames,\n    front_tele_frames=front_tele_frames,\n    meta_action_history=META_ACTION_HISTORY,\n    meta_action_future=META_ACTION_FUTURE,\n    rationale=CANDIDATE_RATIONALE,\n)\n\nprint(rb, rf.overall, rf.subscores)\n```\n\nTo evaluate Alpamayo 2 Edge itself, run its inference script on the selected clip and inspect the generated rationale in output.json under text.cot. Use that same clip and decision time when loading AlpaJudge-RF’s six frames, and replace the notebook’s hand-authored rationale with the generated string. Provide meta-action history and future that match this decision rather than copying the notebook’s construction-scene example. If an expert reference CoC trace is available, pass it to RB as `answer` and the Edge CoC as `student`. \n\n### Workflow 2: Select higher-value training examples\n\nAlpaJudge’s score is useful for training data selection because the highest and lowest scoring examples often serve different purposes. A very low reference-based score can indicate a mismatched label, a failed rollout, or a genuinely difficult case. Review these cases before discarding them; a blanket removal of low-scoring clips would also remove rare failures the policy needs to learn. Very high scores can identify scenes the current model already explains consistently and is not learning much from anymore. The middle range often contains data that is highly informative for further training.\n\nApplying this to your own dataset begins with candidate CoC rollouts per clip and a fixed AlpaJudge checkpoint. Score each candidate against the paired reference (if using AlpaJudge-RB), record the score, clip ID, and any scenario metadata, then review low-scoring and safety-critical examples manually. Set selection thresholds on a development split, stratify the retained set by scene type and maneuver, and keep the held-out evaluation split untouched. Resample informative clips for further training and compare the new checkpoint with the same baseline set.\n\n### Workflow 3: Use reasoning scores for post-training\n\n[The public Alpamayo RL recipe](https://github.com/NVlabs/alpamayo-recipes/blob/main/recipes/alpamayo1_x_rl/README.md) offers a concrete pattern for adding a reasoning reward to open-loop GRPO training. It generates groups of policy rollouts, decodes each CoC and trajectory, computes trajectory error and comfort, then calls a pluggable reasoning grader before combining the terms into one scalar reward. AlpaJudge-RB can be that grader when each sample has a reference reasoning trace: pass the fixed driving question, reference as answer, and sampled CoC as student, then use the returned alignment probability as the reasoning signal.\n\nAn adapter can implement the recipe’s `BaseReasoningGrader.score(predictions, references) -> Tensor[N]` contract by batching the text pairs through score_reference_based. Register that adapter in `GRADER_REGISTRY` and point the reasoning reward config to the corresponding checkpoint. The reward function should keep its trajectory gate and log both component scores, so the policy is less likely to earn a high total reward from a fluent but wrong driving explanation.\n\n## Get started with Alpamayo 2 Edge and AlpaJudge\n\nThese workflows form a repeatable development loop: run the edge model, score its reasoning next to trajectory and safety metrics, select informative clips, and compare the next candidate against the initial baseline. For teams that also train policies with RL, AlpaJudge can provide a reasoning reward through an adapter such as the one described above.\n\nExplore [Alpamayo 2 Edge](https://huggingface.co/nvidia/Alpamayo2-Edge) and [AlpaJudge](https://huggingface.co/nvidia/alpajudge) on Hugging Face. For discussions and developer questions, visit the [Alpamayo developer forum](https://forums.developer.nvidia.com/c/autonomous-vehicles/alpamayo/766). Check out the following for more information on:\n\n- [Alpamayo Open Platform](https://www.nvidia.com/en-us/solutions/autonomous-vehicles/alpamayo/) and its latest releases.\n- [State-of-the-art research](https://research.nvidia.com/labs/aspire/) informing Alpamayo’s development.", "url": "https://wpnews.pro/news/running-av-reasoning-models-within-onboard-compute-and-latency-limits", "canonical_source": "https://developer.nvidia.com/blog/running-av-reasoning-models-within-onboard-compute-and-latency-limits/", "published_at": "2026-11-10 16:00:00+00:00", "updated_at": "2026-11-10 16:24:08.934008+00:00", "lang": "en", "topics": ["autonomous-vehicles", "ai-agents", "ai-infrastructure", "machine-learning", "ai-research"], "entities": ["NVIDIA", "Alpamayo 2 Edge", "NVIDIA DRIVE AGX Thor", "NVIDIA Cosmos 3 Edge", "AlpaGym", "AlpaJudge", "DFlash2", "TensorRT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/running-av-reasoning-models-within-onboard-compute-and-latency-limits", "markdown": "https://wpnews.pro/news/running-av-reasoning-models-within-onboard-compute-and-latency-limits.md", "text": "https://wpnews.pro/news/running-av-reasoning-models-within-onboard-compute-and-latency-limits.txt", "jsonld": "https://wpnews.pro/news/running-av-reasoning-models-within-onboard-compute-and-latency-limits.jsonld"}}