Post-training 4B model with RL to fly Tesla Roadster 21% better than GPT-6 Astra

wpnews.pro

Last Thursday (October 15), Tesla's new Roadster drove off a ramp that curled over until the car was upside down, rolled upright in the air and landed. It then used thrusters to hop over a Cybertruck. Turns out the future does have flying cars.

The demo had already been pushed back two weeks, "due to high winds" (Elon Musk on X). That made us wonder: with a language model in the pilot's seat, how would the car do in that wind?

We built an RL environment around a physics simulator of a flying roadster to find out, handed the controls to LLMs, and turned the wind up. We let some of the latest best models fly it: DeepSeek V4.1 Flash, Meta's Muse Spark Open, Reflection's Beam 1.0, NVIDIA's Nemotron 3.5 Super VL, TypeSafe's Jev 1.14, and even OpenAI's GPT-6 Astra.

Asked for decisions in real time, these models crashed almost every flight: most take 1 to 6 s to answer, and the car needs a new action every 0.2 s. When we d the world so the models could think as long as they wanted, they still lost to a 4B model we trained with RL in a storm.

Storm, 33 m/s wind: DeepSeek, with unlimited time to think, comes down leaning and hits the road with its body. Our RL-trained 4B lands level.

TL;DR

We built an RL environment around a physics simulator of a flying roadster with four air thrusters, gusty wind and the reveal's two stunts: a ramp flip and a kickflip over a Cybertruck. An LLM flies it with one command every 0.2 s while the car keeps moving.
The latest open models can't fly it in real time. DeepSeek V4.1 Flash, Nemotron 3.5 Super, Muse Spark Open and Beam 1.0 take 1 to 6 s per decision and land 0% of kickflips and 3% or less of ramp flips. GPT-6 Astra also lands 0% on the clock.
RL post-training made a 4B model our best pilot. Qwen3-4B, trained for about $13, answers in 0.09 s and lands 90% of kickflips in real time. With the world d for the model to think, the best open model, DeepSeek V4.1 Flash, lands 87% and the other open models land 58 to 62%. Even GPT-6 Astra lands only 74%.
Everything is open source, and you can fly your own model without a GPU (Try it ). Watch how the models fly the car. Pick a stunt, a wind level and a model below to replay the same flight on a held-out scene. The full page also shows the pilot decision every 0.2 seconds, the telemetry the model saw, and how long each reply took.

RL environment for flying Tesla Roadster in the wind # #

Four air thrusters lift the car, and push it down when it's upside down

The car is a rigid body, 4.5 m long and 2,000 kg dry, simulated at 500 Hz. Lift comes from four cold-gas thrusters at the corners of the floor of the car. Each thruster is good for 12 kN, fed from one tank of compressed air (110 to 170 kg per flight). They push straight out of the floor. If the left pair fires harder, the car rolls, and if the front pair fires harder, it pitches.

When the car is upside down, the same thrusters push it toward the ground, and that is what makes both stunts hard.

A car falling flat or sideways catches far more wind

A car is built to move nose first. When it falls flat, or the wind hits it from the side, it shows five to nine times as much area to the air, so the wind pushes it around far more. Near the road the wind is weaker, and gusts come on top of the average. Each flight gets a mean wind from a random direction, in one of three bands: light (up to 14 m/s), gale (14 to 24 m/s) or storm (24 to 33 m/s).

The wheels can take a 4 m drop, but the roof can't touch the ground

The car can touch the ground with its four sprung wheels or with its body. It counts as a crash when:

| What touches | Crash when it's moving down faster than |

| Wheels | 9 m/s (about a 4 m drop) | 
| Floor | 3 m/s (about a 0.5 m drop) |

Two stunts from the reveal: a ramp flip and a kickflip over a Cybertruck

In the ramp flip, a ramp curls over and the car leaves the deck upside down 10 to 14 m up, then has to roll upright and land. In the kickflip, the car comes in at 50 to 94 km/h, clears a parked Cybertruck, rolls a full 360 degrees and lands.

A flight succeeds only if the car ends up on all four wheels on the 14 m road, within about 20 degrees of level, no longer spinning, and without a crash. Every task randomizes the speed, the air in the tank, the wind and gusts, and the ramp size or truck position. The held-out tasks use seeds that no model trained on.

What the model sees, and what it sends back

At the start of each flight, the model gets a briefing. It describes the car, the crash rules and the five commands it can send, and then the task. Here is the task part of the briefing for one held-out storm flight:

After that, every 0.2 s of simulated time, the model gets one line of telemetry and replies with one JSON object. Here is the line our trained Qwen 4B model saw 1.2 s into that flight, while it was upside down above the truck:

The model replied {"mode": "rotate", "axis": "roll", "to": 360.0, "lift": 1.0} in 0.15 s, which kept the roll going. The whole flight took 19 replies:

| Time | What the telemetry showed | The model's reply | Time to answer | | 0.0 s | On the road, truck 25.7 m ahead (reached in 1.1 s) | climb at 7 m/s | 0.09 s | | 0.2 to 0.4 s | Lifting off, 0.5 m then 1.2 m up | climb at 7 m/s again | 0.09 s | | 0.6 s | 2.3 m up, truck 11.8 m ahead (reached in 0.5 s) | rotate roll to 360°, lift 1.0 | 0.15 s | | 0.8 to 1.6 s | Rolling over the truck: 21°, 95°, 182°, 267°, 335° | the same rotate , five times | 0.15 s | | 1.8 s | Upright at 6.7 m, rolled 360°, truck 16 m behind | land | 0.08 s | | 2.0 to 3.4 s | Coming down, then two wheels touch | land , eight more times | 0.08 s | | 3.6 s | All four wheels on the road, 3 m from the center line | coast | 0.05 s |

The truck part of each line also gives the time until the car reaches the truck and the time until it's past it, at the current speed, so the model doesn't have to do that division itself. Every LLM in this post sees the same lines, and the hand-written pilot below reads the same numbers.

The five commands are coast (all thrusters off), climb (stay level at a given vertical speed), hold (hold a given roll and pitch), rotate (spin about one axis to a total angle, braking to stop on it) and land (level out and come down softly). The model picks one command, with its numbers, in each reply. A flight computer then turns that command into thrust for each of the four thrusters, 50 times a second, until the next reply arrives.

The car doesn't stop while the model thinks

If a model takes 2 s to answer, the car flies ten 0.2 s intervals on the old command before the new one takes over. We time every reply end to end. Only the first decision is untimed, like a plan made before takeoff. We picked 0.2 s because the kickflip's roll takes about a second and has to start within a few tenths of a second of the right moment. With one second between decisions, no pilot could time it, whether model or script. To separate judgment from speed, we also fly every model a second way, with the simulator d until each reply arrives, so the model has unlimited time to think.

Half the reward is for a clean landing

For RL the reward runs from 0 to 1. It gives partial credit for height, clearing the truck, rotation and touchdown quality, then 0.4 for a successful landing and 0.1 for a clean finish (soft, with air left, centered on the road).

A pilot written by hand shows that both stunts can be flown

Before training anything, we wanted to know that both stunts can be flown with these five commands. So we wrote a pilot by hand: a few dozen lines of if-then rules that read the same telemetry and send commands through the same 0.2 s interface. On the kickflip, for example, it starts climbing when the truck is 1.1 s away, starts the roll once the car is 1.4 m above the truck's roof, leans against any sideways drift, and lands once the car is past the truck.

The hand-written pilot answers instantly and sees the same inputs to the LLMs. It lands every light-wind ramp flip and 98% of light-wind kickflips, but only 61% of kickflips in a storm. We call it the scripted pilot in the rest of this post.

The scripted pilot flying both stunts in light wind.

The latest open models can't fly in real time # #

We flew four open models (DeepSeek V4.1 Flash, Muse Spark Open, Nemotron 3.5 Super and Beam 1.0), GPT-6 Astra in storms, and Jev 1.14, a model built for fast decisions that picks from a menu of commands instead of writing JSON. Every model flies the same held-out flights as our trained pilot: 192 per stunt in light wind and in a gale, and 384 in a storm. All runs use temperature 0 and each model's default reasoning setting. We called the open models through OpenRouter, with Nemotron, Muse and Beam each pinned to one provider.

With time to think, the models make good calls

When the simulator waits for each reply, the four open models fly the kickflip well in light wind. DeepSeek V4.1 Flash lands 98%, and Muse, Nemotron and Beam land 72 to 82%.

Wind makes it harder even then. In a storm, DeepSeek drops to 87%, Muse to 62%, Nemotron to 60% and Beam to 58%. GPT-6 Astra lands 74% of storm kickflips.

On a running clock, they land 0% of kickflips

With the car moving, all four open models land 0% of kickflips. Their median decision takes 1 to 6 s, so a typical command arrives 5 to 29 intervals after it was needed. In the real-time runs, with 16 flights in parallel, replies were slower still (DeepSeek's median was 2 to 3 s).

GPT-6 Astra takes about 2 s per decision and lands none of its 768 storm flights in real time. On the same clock, our trained 4B lands 90% of storm kickflips.

DeepSeek V4.1 Flash on the same flight, twice. On the left the simulator waits for each reply, and the car lands. On the right the clock keeps running, the replies arrive late, and the car crashes.

Answering faster doesn't fix it

In our first round, with reasoning off, DeepSeek answered in 0.3 s but made worse calls, and Muse and Beam at low effort still took about 2 s per decision. Jev answers in about 0.17 s, fast enough for the clock, but it lands only 2% of storm kickflips live. A pilot has to answer fast and make good calls.

The ramp is easier, but the replies still come too late

The car leaves the deck upside down 10 to 14 m up, and a simple plan works: roll upright, lean against the wind, then brake and land. With the simulator d, the big models fly it. Muse lands 88 to 97%, DeepSeek about 80%, and GPT-6 Astra 383 of 384 storm flights.

On the clock, none of them gets past 3%. The first command is usually a roll upright, which is correct, but the fall takes under 2 s, and the next reply comes after the car has already hit the road.

A ramp flip in real time. DeepSeek's first command, roll upright, is correct, but its next reply arrives after the car has hit the road. On the right, the trained 4B answers every 0.2 s and lands.

| Ramp flip, share that land | Light wind | Gale | Storm | | Qwen3-4B, untrained (real time) | 100% | 99% | 85% | | Muse Spark Open, unlimited time to think | 97% | 92% | 88% | | DeepSeek V4.1 Flash, unlimited time | 82% | 78% | 79% | | Beam 1.0, unlimited time | 72% | 72% | 68% | | Nemotron 3.5 Super VL, unlimited time | 67% | 56% | 50% | | DeepSeek V4.1 Flash, real time | 1% | 2% | 3% |

Muse, Beam and Nemotron land 1% or less in real time.

The untrained Qwen3-4B answers in 0.07 s and lands 85 to 100% of ramp flips, but it never lands a kickflip. Like Jev, it's fast enough for the clock. Unlike Jev, it's a model we can train, so it's where we started.

Post-training a Qwen3-4B pilot # #

We started from Qwen3-4B-Instruct-2507 with a rank-64 LoRA (small add-on weights that train while the base model stays frozen). The training code is SkyRL, an open-source RL library. The final model took about 4 GPU-hours of training, or $13.

A warm start, then 60 steps of RL

Training has two stages:

Warm start (2 hours 20 minutes): supervised fine-tuning on flights that landed: 64 scripted light-wind flights per stunt, plus 350 gale and storm kickflips that DeepSeek V4.1 Flash landed on training seeds with the simulator d. We train one epoch, with the loss only on the pilot's replies. Most of the time goes to moving optimizer state off the GPU, so that the model fits in GPU memory.
RL (1 hour 35 minutes): GRPO flies the same task several times and pushes the model toward the attempts that beat the group average. Each of the 60 steps flies 16 tasks, 8 flights each, across all three wind levels. Every 0.2 s decision is one conversation turn, and the reward is for the whole flight. The trainer shares the GPU with the vLLM server that flies the model.

We save a checkpoint every 5 steps, pick one using separate validation seeds only, and then fly the held-out seeds once. The recipe itself came out of many rounds of experiments scored on validation seeds, and we held the test seeds back for this one final run.

The RL-trained 4B landing a storm kickflip in real time.

The rest of this section is how we got to that recipe.

Why RL alone never learned the kickflip

The first run, RL from the untrained model, picked up the ramp flip in 10 steps. The kickflip reward started at 0.07 and stayed there. Of 68 kickflip decisions, 66 were a roll, and 21 of 22 first moves rolled at t=0 with the wheels still on the road, which puts the car on its roof.

All 8 flights in a group crashed the same way and got the same reward. GRPO learns by comparing samples within a group, so when they're all the same it has nothing to go on.

The same light-wind flight before and after training. Untrained, Qwen3-4B starts the roll with its wheels on the road and tips onto its roof. After imitation and RL, it climbs first and then rolls.

We tried two cheap fixes. First, we added a line to the briefing saying that rolling on the road tips the car over. The opening move became a roll with maximum lift, which still ended on the roof.

Second, we started half the practice flights partway through, after the scripted pilot had flown the opening (a reverse curriculum, borrowed from robotics). Training success rose to about 70%, but held-out success from t=0 stayed at zero, because the model only learned to finish a flip that the scripted pilot had started.

The warm start used two teachers, each in the wind it flew best

The warm start is what worked, and most LLM RL recipes start with one anyway. The hard part was choosing what to imitate, because neither pilot we had was good in every wind:

The scripted pilot is good in light wind and bad in storms (61% of storm kickflips).
DeepSeek with unlimited time is good in storms, but it takes about a second per decision, too slow to fly the car live.

So the warm start copies the scripted pilot's light-wind flights and DeepSeek's landed gale and storm flights. On its own, the warm start already lands 97%, 93% and 84% of held-out kickflips in light wind, gale and storm, which is close to DeepSeek except in storms.

RL improves further on what the teachers taught

RL starts from what the warm start copied, then practices where the teachers were weakest. Two choices in its training mix came from reading failed flights:

Storm counts double. Storms are where the reward is hardest to earn, so the training mix has twice as many storm flights as light or gale.
Practice the slow approaches. An earlier model lost to DeepSeek in light wind on one kind of flight: a slow car (14 to 16 m/s) with the truck about 3 s away. It sentland at the same moment it always did, about 1.8 s in, and came down on the truck. On the same flights, DeepSeek held its height until the telemetry said it was past the truck, then landed. In the final run, half of the training kickflips are these slow, far-truck cases, so RL has to learn to wait.

The RL-trained Qwen3-4B flies best once the wind picks up # #

Kickflip. Share of held-out test flights that land, with the number landed in brackets. There are 192 flights per wind level in light wind and gale and 384 in a storm, and every row flies the same tasks.

| Scripted pilot (hand-written rules, decides instantly) | 98% (188) | 83% (160) | 61% (233) | 
| DeepSeek V4.1 Flash, real time | 0% (0) | 0% (0) | 0% (0) | 
| DeepSeek V4.1 Flash, unlimited time to think | 98% (189) | 90% (172) | 87% (334) | 
| Qwen3-4B, no training (real time) | 0% (0) | 0% (0) | 0% (0) | 
| Qwen3-4B after imitation only: fine-tuned on the teachers' flights, no RL yet | 97% (187) | 93% (179) | 84% (322) | 
| **Qwen3-4B after imitation + RL (real time)** | **96% (185)** | **96% (184)** | **90% (347)** |

The trained pilot answers in 0.09 s (served by vLLM, one flight at a time), inside the 0.2 s interval, so none of its commands arrive late. Every flight ends the same way whether the simulator runs in real time or waits for it.

RL's part shows up in the storm. The same model after imitation only lands 322 of the 384 storm flights, and after RL it lands 347. Flight by flight, 26 storm tasks land only after RL and 1 only without it (exact paired test, p < 0.000001). In light wind and gale the warm start was already close, and RL moves them by a few flights either way.

Against DeepSeek with unlimited time to think, flight by flight on the same tasks:

Gale: 13 flights only the 4B landed, 1 only DeepSeek (exact paired test, p = 0.002).
Storm: 29 to 16 (p = 0.07). The 4B is ahead, but p is above the 0.05 threshold we set before running the test, so we don't count the storm lead as significant.
Light wind: 2 to 6 (p = 0.29). DeepSeek is a few flights better, and so is the scripted pilot (188 to 185). Against GPT-6 Astra with unlimited time, on all 384 storm kickflips, the 4B is clearly ahead: it lands 347 to Astra's 286, or 21% more. 70 land only for the 4B and 9 only for Astra (exact paired test, p < 0.001). On the storm ramp, Astra is slightly ahead, 8 flights to 1 (p = 0.04).

RL runs vary. Rerunning the same recipe landed 85 to 92 of 96 validation storm flights, and the run we kept was the best of them. A rerun would most likely land about as many storm kickflips as DeepSeek.

Ramp flip. Same test set and sizes as the kickflip table.

| Scripted pilot (hand-written rules, decides instantly) | 100% (192) | 100% (192) | 96% (368) | 
| DeepSeek V4.1 Flash, real time | 1% (2) | 2% (3) | 3% (11) | 
| DeepSeek V4.1 Flash, unlimited time to think | 82% (157) | 78% (149) | 79% (304) | 
| Muse Spark Open, unlimited time to think | 97% (186) | 92% (176) | 88% (336) | 
| Qwen3-4B, no training (real time) | 100% (192) | 99% (190) | 85% (325) | 
| Qwen3-4B after imitation only: fine-tuned on the teachers' flights, no RL yet | 100% (192) | 100% (192) | 96% (369) | 
| **Qwen3-4B after imitation + RL (real time)** | **100% (192)** | **100% (192)** | **98% (376)** |

On the ramp, imitation does most of the work. After imitation alone the 4B already flies it almost exactly like the scripted pilot, and RL adds a little on top. 7 storm ramps land only after RL and none only without it (p = 0.02), and 8 land where the scripted pilot fails, with none the other way.

The stronger the wind, the bigger the 4B's lead

The scripted pilot is nearly perfect in light wind and drops sharply in a storm, from 98% to 61%. DeepSeek holds up better, from 98% to 87%, but needs about a second per decision.

The trained 4B goes from 96% to 90%. It loses a little to both teachers in light wind and beats both once the wind picks up. Choosing the teachers by wind level got it most of the way there. The warm start alone lands 84% in storms, and RL takes it to 90%.

The same storm flight, flown by the scripted pilot (left) and the RL-trained 4B (right). The scripted pilot's body hits the road before its wheels do, and the 4B touches down on its wheels.

The failed flights look different, too. In storms, 34 of DeepSeek's 50 failed flights end with the car's body hitting the road before its wheels. 23 of the trained 4B's 37 end in a level landing just off the 14 m road.

Among storm flights that land, the 4B's are a little shorter than DeepSeek's (3.6 s against 3.9 s, median) and finish with more air in the tank (24% against 18%), so it has thrust left to settle the car. We didn't reward short flights directly. The reward pays for a soft, centered landing with air left, and RL found shorter flights on its own.

How far the car leans on the way down, DeepSeek against the trained 4B on the same storm flight. DeepSeek is still leaning about 20° into the wind when its body hits the road.

Lessons learned # #

1. The simulator must capture realistic scenarios

We scored every model twice on the same flights, and the ranking flipped. With the simulator d, DeepSeek V4.1 Flash was the best kickflip pilot we tested. With the car moving, it landed nothing.

Most LLM benchmarks give the model as much time as it needs to answer, just like our d runs. If your model has to act while the world keeps moving, like a vehicle or a voice agent, the time it takes to decide is part of the score, so measure how fast it decides along with how well.

2. DeepSeek beat bigger models while running fewer parameters per token

With unlimited time to think, DeepSeek V4.1 Flash beat every other open model we tested by a wide margin: 98% of kickflips in light wind and 87% in a storm, against 72 to 82% and 58 to 62% for the rest. It runs only 16B of its 552B parameters per output token. That is the fewest of the group except Nemotron 3.5 Super (12B of 120B), and well under Muse Spark Open (42B of 816B) and Beam 1.0 (23B of 501B). The best pilot of all was smaller still, a dense 4B after RL. On a task this narrow, how a model was trained mattered more than how many of its parameters run per token.

3. RL improved the 4B model beyond its teachers

The warm start copied two teachers, each good in a different wind setting. After RL, on the same held-out flights, the 4B model beats the scripted pilot by 30 points in storms, beats DeepSeek in gales, and is slightly ahead of it in storms. It doesn't beat either teacher in light wind, where both were already near perfect.

Imitation brought the 4B up to the teachers' level, and RL on the real objective moved it past them where there was room to improve. In storms that was 25 more landings out of 384 than the warm start alone, with only 1 flight going the other way.

4. GRPO needs attempts that differ at the hard decision

Our first runs failed for a reason that's easy to miss. GRPO compares a group of attempts at the same task. When all eight crash the same way, they all get the same reward, and the model has nothing to learn from. Starting practice partway through the flight raised training success to 70% and left held-out success at zero, because the model never practiced the opening move, which was the hard part. A short warm start fixed it by making the attempts differ at that decision.

What this doesn't show # #

Obviously, this says nothing about flying a real car. A real car would use a conventional controller, and even our scripted pilot decides instantly, which no LLM does.

It also isn't how a robotics team would build this pilot. Hierarchical systems like Figure's Helix run a 7B vision-language model at 7 to 9 Hz on top of an 80M-parameter policy at 200 Hz. The large model chooses what to do, and the small policy turns that into motor commands 200 times a second. Our stack has the same two layers, with a 50 Hz flight computer holding attitude and vertical speed underneath the LLM, but our fast layer is scripted and makes no decisions.

We deliberately left the one timing-sensitive decision, when to start the roll, to the language model, because we wanted to know whether an LLM can make that call against a clock. We haven't trained a classic controller policy (for example, PPO with a small network) on the simulator, and we'd expect one to land nearly everything.

The RL environment and every vehicle parameter in it are our own assumptions, and it's one environment with two stunts. Held-out means unseen seeds (wind, speeds, ramp and truck placement) of the same two stunts, not new stunts. DeepSeek's storm flights were part of the warm start, so in storms the 4B is compared with a model whose flights it learned from.

Using SkyPilot for RL training # #

The final model took about 4 GPU-hours (2 h 20 min of warm start, 1 h 35 min of RL), roughly $13 (at ~$3.39/h per H100). Finding the recipe took many more runs. We ran each training run and each evaluation as a SkyPilot managed job, up to six at a time, and SkyPilot found a free GPU for each one. When a cluster node went bad mid-run, SkyPilot reported the stuck jobs and we relaunched them on other nodes.

Training, rollouts and evals run side by side as one Job Group

An RL run does three things at once. It updates the model, it generates rollouts with the latest weights, and it scores checkpoints. A SkyPilot Job Group launches these as one managed job, with each part on its own hardware:

The trainer runs GRPO on its own H100 and writes new LoRA weights every step to a volume it shares with the inference server.
The inference server runs vLLM on another H100 and generates the rollouts with those weights.
An eval watcher starts an evaluation for each new checkpoint. Each evaluation gets its own GPU, runs alongside training, and joins the same Job Group as one more task.

The numbers in this post come from the same recipe run as separate training and evaluation jobs. The Job Group version is train_eval.yaml in the repo. For every evaluation, vLLM serves the checkpoint and tools/pilot_bench.py flies it over the OpenAI-compatible API, the same code path as every other model in the benchmark.

Try it # #

SkyPilot is open source (pip install skypilot) and everything above runs on it. The RL environment, prompts and SkyPilot files are in the flying-roadster-rl example in the SkyPilot repo.

Fly your own model. The benchmark talks to any OpenAI-compatible endpoint and defaults to SkyPilot Tokens, which serves DeepSeek V4.1 Flash, Muse Spark Open and other open models. Create an account there for an API key, clone the repo, cd llm/flying-roadster-rl, and run:

Drop --realtime to give the model unlimited time to think. --endpoint openrouter uses $OPENROUTER_API_KEY instead. For a local server or any other OpenAI-compatible API, pass --base-url (and --api-key-env if the key lives in another variable). A local server needs no key.

To train your own pilot, with checkpoints evaluated as they land, launch the Job Group. You'll need a Kubernetes cluster with two free H100s and a shared volume, plus an S3 bucket:

Without a shared volume, train.yaml and eval.yaml run training and evaluation as separate jobs.

If you have questions or a baseline we should add, open an issue on the repo. If you want help post-training a model on your own task, talk to us.

source & further reading

skypilot.ai — original article