Qwen-Drive 1.0: Turning a General VLM into a Driving Foundation Model #
The most recent Vision-Language-Action approaches for autonomous driving use a pretrained VLM mainly as a source of semantic understanding and reasoning, then specialize it on driving data and connect it to an action or trajectory predictor. Qwen-Drive-1.0 asks a somewhat broader question: can a general-purpose VLM be turned into a foundation model for driving without giving up the capabilities that made the original VLM useful?
This matters because language supervision alone is a poor constraint on actual 3D geometry (a model may describe a scene convincingly while having only an approximate understanding of distances and occupancy) and aggressive domain adaptation can destroy some of the general visual and world knowledge that may be particularly valuable in rare or out-of-distribution situations.
Qwen-Drive addresses this by keeping the architecture of Qwen3.5-4B intact while attaching two specialized components: a BEV (bird’s-eye-view) perception head for 3D detection, occupancy prediction, and map segmentation, and a ~1.1B-parameter Planning Expert that generates five-second trajectories using flow matching.
The resulting model substantially improves driving-specific understanding over the original Qwen3.5-4B while losing less than one point on the paper’s VLM benchmark average, and it achieves competitive planning performance across open-loop, pseudo-closed-loop, and closed-loop evaluations. The interesting result is that explicit 3D perception, language-based reasoning, and continuous control can coexist on top of the general-purpose VLM.
The approach #
The BEV perception head
The BEV head reads two feature streams per camera view: raw vision-encoder features carrying low-level appearance, and the same image tokens after they have traversed the full VLM, carrying broader scene context.
A depth-based lift-splat transform projects the encoder features into a 3D volume using a per-pixel depth distribution learned without explicit depth supervision. In parallel, a query-based BEV transformer aggregates a feature pyramid built from the VLM-side features onto the BEV plane. Its queries are initialized from the height-collapsed 3D volume, giving them an explicit geometric prior rather than starting from learned queries alone.
The resulting BEV representation feeds three tasks:
- A DETR-style deformable decoder predicts 3D objects
- A UNet-style head segments the map
- The occupancy branch expands the BEV features back along the height dimension and fuses them with the original 3D volume before a shallow 3D UNet predicts voxel semantics.
The first training stage also turns the head into a probe of the backbone: with the vision encoder and VLM frozen, training the head alone tests how readily their pretrained representations can support explicit 3D prediction.
The Planning Expert
The Planning Expert is a 32-layer diffusion transformer of roughly 1.1B parameters trained on top of the frozen VLM. It conditions on the cached keys and values from the VLM’s eight grouped-query softmax-attention layers, with each cache feeding four consecutive expert layers. The VLM therefore acts as a read-only context provider: its cached K/V states are concatenated with those of the trajectory tokens inside the expert’s attention layers.
The expert emits a 50-waypoint trajectory covering 5 seconds at 10 Hz. It is trained with flow matching using x-prediction: instead of predicting velocity or noise, it directly predicts the clean trajectory at the end of the flow. Additional penalties on first- and second-order temporal differences are introduced to supress waypoint jitter and abrupt acceleration changes. At inference, generation starts from Gaussian noise and uses a 10-step Euler solver.
Training stages and the RL formulation
Training proceeds in four stages, each unfreezing a different part of the system:
- Perception head pretraining : encoder and VLM frozen, only the new BEV head learns.
- Joint perception and VQA training : head, encoder and VLM all trainable, with the head using a 20× higher learning rate than the backbone.
- Planning Expert pretraining : VLM and encoder frozen again, the expert trained purely by flow matching.
- Reinforcement learning : VLM still frozen and only the Planning Expert reward-optimized.
The Stage-2 data pipeline combines 24 public driving VQA datasets. Their examples are rewritten into a common conversational schema and then checked for consistency against the original annotations, cutting the public pool from 5.53M to 3.09M samples. Roughly 20% of each source is then sampled and mixed with general-purpose VLM data and self-constructed driving examples. They include Chain-of-Causation reasoning traces: explanations linking scene elements to driving decisions, generated from logged trajectories and audited by judge models answering concrete classification questions rather than producing scalar quality scores.
Imitation learning has a basic limitation: the logged trajectory is only one of many possible safe trajectories, and it does not directly optimize properties such as collision avoidance or driving progress. RL can optimize these objectives directly, but applying it to a flow-matching planner is tricky because its Euler sampling process is normally deterministic.
Qwen-Drive solves this by adding noise only to the last three of ten sampling steps. Instead of perturbing every waypoint independently, it adds noise along a few low-frequency cosine modes, producing smooth changes such as shifting or bending the whole trajectory rather than random jitter. A restoring term keeps these perturbed trajectories close to those preferred by the pretrained planner.
This creates enough controlled randomness for RL to explore and compare alternative trajectories without destabilizing the original planner.
Experiments
On 3D perception, Qwen-Drive leads the reproduced baselines on nuScenes detection and map segmentation, but trails on occupancy, which the authors attribute to machine-generated voxel labels. More interestingly, the frozen-backbone probe performs poorly: training only the BEV head trails a BEVFormerV2 variant with the same encoder by 6.34 mAP, while unfreezing the encoder and VLM in stage 2 adds 10.46 mAP. The pretrained VLM appears to provide a strong initialization, but not an explicit 3D representation that can simply be read out.
On driving VQA, Qwen-Drive improves the Qwen3.5-4B base from 63.52 to 69.43 while retaining nearly all general VLM performance (66.41 vs. 67.40). Notably, even the unadapted base already outperforms every driving-specific model in the comparison, including several much larger ones.
Planning results are more mixed. On NAVSIM, RL reaches 90.7 PDMS, narrowly ahead of ExploreVLA and EponaV2 at 90.4. On WOD-E2E, however, the test-set gain over UniPlan is only 0.04 RFS, while on PhysicalAI-AV Qwen-Drive’s 0.39 m minADE trails Alpamayo-1.5’s 0.17 m.
Closed-loop AlpaSim is similarly less favorable: RL substantially improves Qwen-Drive, but Alpamayo-1.5 still leads on the main scores, while Alpamayo-R1 is stronger on most metrics. Curiously, the paper reproduces both Alpamayo variants but discusses mainly the weaker one.
Limitations
The generated trajectory does not always follow the textual rationale that conditions it, and the authors concede that reasoning’s measured benefit may come from the extra conditioning context the trace supplies, not from its content. Causal attribution is also unstable when causes act at different time scales, such as a red light 20 meters ahead versus a child 5 meters ahead.
Conclusions
Driving VLAs mostly start from the same recipe (teach a pretrained vision-language model to plan) but differ in the trade-offs. Alpamayo-1.5 uses a much larger 10B model trained on roughly 80,000 hours of driving and three million Chain-of-Causation traces, and beats Qwen-Drive on several planning benchmarks. But its general VLM score in Qwen-Drive’s evaluation drops, while Qwen-Drive retains almost all of Qwen3.5-4B’s original capability. UniDriveVLA tackles the same perception-reasoning conflict by separating understanding, perception and planning into experts, while DriveWAM and SimWAM instead build planning around video-generation priors.
Qwen-Drive takes a safer route: adapt a small general VLM for driving perception and reasoning, then freeze it and put planning in a separate expert. In that sense it resembles the opposite of the scaling strategy seen in PaLM-E, where larger models increasingly retained their general language capabilities after embodied training.
What I like most is that the paper is unusually careful about its limitations: the above-human WOD validation result is identified as in-sample, and the authors avoid claiming that explicit 3D supervision alone explains the planning gains. There are still questionable choices — LingoQA is judged with Qwen-Plus rather than the official LingoJudge, and the closed-loop discussion focuses on Alpamayo-1.5 even though the reproduced Alpamayo-R1 baseline is stronger on most metrics.