If our browser has ever gone offline, we have met the Chrome Dino game. A T-rex runs across a desert, and we press a key to jump over cacti and duck under pterodactyls. Nobody plans while playing it. we see a cactus, and our finger is already moving.
Daniel Kahneman called this kind of thinking System 1: fast, automatic and pattern-driven. System 2 is the slow one that plans, compares and calculates. Here, I built a System 1 Decision model for a machine and tested it in a Chrome-Dino-style game where the runner is a drone.
My Objective is to Train a Type 1 decision model: fast, automatic and pattern-driven, in the sense of Kahneman’s System 1. The model flies a drone through a Chrome-Dino-style side-scroller. Each frame, it observes the world and picks one of five actions with minimal latency. The goal is to survive as long as possible and fly as far as possible, using learned reflexes rather than planning.
Note: AI has been used to refine this article to improve its readability, clarity, and understandability.
DroneDash keeps Chrome Dino’s idea and swaps the runner. It is a Gymnasium environment (DroneDash-v0) with a desktop app on top. The drone sits at a fixed x-position while the world scrolls left at 10 to 14.4 m/s, inside a 10 m tall airspace where the ground and the ceiling are both crashes. The simulation steps at 30 Hz, and the pilot makes exactly one decision per step.
The obstacles are what a drone meets along a road: buildings, trees, street lights, kites, storm clouds, birds, other drones, hot-air balloons, planes (high up) and downdrafts. Gusts, speed surges and fog arrive with no warning; the drone only feels a gust once it hits. A person can only Start, /Resume and End a run.
Table 1 lists what each action asks for, not what the drone does on the next frame. Between the command and the motion sits a small physics model, and that gap is what makes flying a timing problem: a slow move needs time to build up speed, so the pilot must start it before the obstacle is close, or fall back on a burst. So before we look at the pilot, let’s see how DroneDash turns an action into vertical motion every frame.
A slow action sets a target climb rate v*. The vertical speed chases it with a gain of 8 s⁻¹ and an acceleration cap of 12 m/s², and wind adds a term the sensors cannot see:
The cap matters. A slow move cannot reach full speed instantly, so a pilot that waits too long has to burst. arc’s physics features approximate the lag of ASCEND as v²/(2 a_max) = 3²/(2 · 12) = 0.375 m.
L1 (Level 1) runs at 10 m/s with sparse obstacles; L2 to L5 run at 12 m/s and obstacles get denser, with more birds, drones and interruptions. Endless mode climbs a level every 1.75 km.
Sensing: The drone has 13 rays from −60° to +60° and a list of up to 12 nearby obstacles, out to 20 m (8 m in fog). Birds and drones wander, and 30% of birds dive or pop up now and then, so an object tracker stands in for a real drone’s sensing software. It fits a sine to the last 1.5 s of sightings of each mover, evaluates it over the window in which the drone will pass, and widens it by margins calibrated on 1.4 million simulated sightings. For a bird, the predicted band is about 4.6 m tall at first sight and 1.1 m when the pass is within 0.4 s. The tracker never reads the game’s hidden path settings.
At frame t, the pilot sees an observation o_t built only from its sensors: a SELF vector s_t, the ray vector r_t and up to 12 obstacle rows h_t. It returns one of five actions. Its only links to the past are its own last action (part of s_t) and the tracker’s estimates inside each obstacle row.
Seeds are split into fixed TRAIN, VAL, TEST and STRESS lists. VAL picks settings; TEST and STRESS run once per finished pilot. Before training, I fixed the targets: a ratio of at least 0.90 on L5 VAL, an act() p99 under 1 ms, and a stress drop of at most 0.20.
At play time only the pilot runs; the game app never imports the planner. Everything ran on one MacBook Air (Apple M5, 16 GB): the game and the pilot on one CPU thread, training on Apple MPS.
arc follows from three facts about the task: obstacles have no natural order, the actions differ in how they move the drone, and the key question is whether a slow move still has time. So it encodes a frame as a set of tokens, makes the actions tokens too, and computes the time-to-contact physics inside the network. The width is d = 64 throughout, with 4 heads of d_k = 16.
Let’s start with what arc sees. Everything is scaled to roughly [−1, 1]:
The kind id goes through a learned embedding (13 ids, 9 of them in the current game). Each action is described by 3 values: the height it gains 0.5 s after starting (/ 3), its hitbox scale and its duration. Actions on cooldown, or blocked during a burst, are masked.
Each input goes through a two-layer MLP with GELU and comes out as a 64-number token.
Obstacles have no natural order, so there are no position embeddings and arc treats them as a set. Two unit tests pin this down: shuffling the obstacle tokens leaves the move, safety and threat outputs unchanged (within 1e-5), while DECIDE’s attention moves with the obstacles, and permuting the action tokens permutes the logits the same way. Because each action is a token that describes itself, a masked one (JUMP or DUCK on cooldown) simply cannot be chosen.
Early versions of the pilot kept answering one question badly: is there still time for a slow move, or does this need a burst? So I stopped making the network learn it and computed it inside the model, from inputs it already had. For an obstacle with gap g, closing speed v_c and climb needed Δy↑:
A positive slack means a slow move started now still clears the obstacle. A negative slack means it won’t, and only a burst (or another plan) can. The five numbers cost 320 weights (5 more inputs to the obstacle encoder’s first 64-wide layer) and add no inputs to the ONNX graph.
With the tokens in place, we can look at how they talk to each other. They go through two pre-norm transformer blocks: 4 heads, d_k = 16, and a feed-forward of 64 → 256 → 64 with GELU. The mask is built once as a static 20 × 20 pattern S, then combined every frame with the column validity c (filled obstacle slots, available actions):
The mask encodes three rules:
After the encoder, a choice head runs one more block of the same kind over only the 6 tokens that matter for the decision: the 5 actions and DECIDE. Here every token sees the available actions and DECIDE. This is where the options finally meet and get compared.
The move is a bilinear score between DECIDE and each action. Two small heads read off safety and threat:
A fourth output is DECIDE’s attention over the 12 obstacles, from the last encoder block and averaged over heads. With --show-attention, the game outlines the obstacle arc is watching. A value head V(h_D) exists only for PPO and is not exported.
Let’s put the pieces together. Shortened from models/arc.py:
def forward(self, self_f, rays, haz, haz_type, haz_mask, act_ids, act_desc, act_avail): B = haz.shape[0] if self.ttc_features: # + 5 physics numbers haz = torch.cat([haz, hazard_physics(self_f, haz)], -1) ctx = torch.cat([self.self_enc(self_f)[:, None], # SELF [B, 1, 64] self.ray_enc(rays)[:, None], # RAYS [B, 1, 64] self.haz_enc(haz) + self.haz_type(haz_type)], 1) # 12 OBSTACLES acts = self.act_id(act_ids) + self.act_desc(act_desc) # 5 ACTIONS x = torch.cat([ctx, acts, self.decide.expand(B, 1, self.d)], 1) # 20 tokens [B, 20, 64]allow = (self.mask_struct[None] & col_ok[:, None, :]) | self.mask_eye[None] for blk in self.enc: # entity encoder, 2 blocks x, p = blk(x, allow) h = x[:, ACT0:] # 5 ACTIONS + DECIDE [B, 6, 64] for blk in self.head: # choice head, 1 block h, _ = blk(h, hallow) # hallow: available actions + DECIDE h = self.norm(h) a, dec = h[:, :N_ACT], h[:, -1] logits = torch.einsum("bd,bad->ba", self.q_proj(dec), self.k_proj(a)) / math.sqrt(self.d) logits = logits.masked_fill(~act_avail, float("-inf")) # cooldown -> never chosen # ... decide_attn: DECIDE's attention over the 12 obstacles, last encoder block return logits, self.safe(a).squeeze(-1), self.threat(dec), decide_attn, self.value(dec).squeeze(-1)
With d = 64, one transformer block holds the four attention projections, the feed-forward and two LayerNorms:
The token encoders are mostly four small MLPs (4,416 to 5,376 weights each), plus the two embeddings and DECIDE. Five sixths of the model (83%) sits in the three transformer blocks.
The trained network is exported to ONNX (opset 18), and the game only ever talks to this graph:
At play time, act() copies the frame into preallocated arrays, runs one ONNX session on one CPU thread, applies the masked softmax at temperature T, and takes the argmax. If the winning probability is below τ, the decision is flagged as hesitating. The flag goes to the HUD and the logs; it is never handed to a planner.
DAgger trains arc from the planner: System 2 teaches System 1. PPO then fine-tunes arc in the game, which gives arc_ppo, the same network with new weights. On the Mac, the teacher data takes about 12 minutes, imitation 8, DAgger 20, and PPO about 3 hours.
At every labelled frame, the teacher clones the simulator with the random interruptions off (except the gust the drone already feels), then cuts the copy down to what the drone could know: nothing beyond 20 m (8 m in fog), birds and drones as the tracker’s predicted bands (its best guess if those leave no safe move), and trees, street lights and kites grown by 0.2 m. For each first action, it searches 9 macro steps of 5 frames (1.5 s), keeps the best 2 partial plans, and scores every plan:
One detail in the search mattered more than any hyperparameter. Beam pruning ranks partial plans by their score so far, and a steady climb pays its small cost on every step before it reaches the obstacle. Pruned that way, every plan that kept climbing lost to a last-moment JUMP, at any beam width up to 8, and the student copied those late jumps. So I changed the pruning rule: a branch that repeats ASCEND or DESCEND is always kept. With that one change, the teacher starts climbing as soon as a building enters its 1.5 s plan.
The restriction to the drone’s own view is deliberate. Earlier teachers planned with information the pilot did not have, such as a bird’s exact sine path, and their students held where the teacher moved: the labels said “move” in states where the pilot’s inputs looked safe. When I gave the pilot the same band the teacher planned with, bird crashes in 18 long endless runs fell from 13 to none, and the median run grew from 1.7 km to 7.4 km. That band came from the game’s hidden settings, so I replaced it, for both the teacher and the pilot, with the tracker’s prediction.
The teacher flew the TRAIN seeds and labelled 602,038 frames from 1,104 episodes, in 691 s. Levels were mixed 10/15/20/25/30% from L1 to L5. A teacher that always flies well only shows good states, so each frame has a 15% chance of starting a 1 to 5 frame burst of random actions (chosen among those the teacher marks safe), and the teacher has to show how to recover. Every frame stores four labels: the soft scores q, the safe flag per action, the threat level and the teacher’s chosen action.
The first pilot, arc_A, learns all four labels at once (masked actions excluded), with frames sampled by weight:
The last loss term looks redundant, but it isn’t. With soft labels at temperature 2, “start climbing now” often beat “hold a little longer” by about 0.01 in score: a nearly uniform label. A pilot trained only on q learned to move when a crash was imminent and never earlier. The hard label says: the teacher climbed here.
I trained it for 15 epochs on Apple MPS: AdamW (lr 3e-4, weight decay 0.01), cosine with 5% warm-up, batch 512, gradient clip 1.0, about 8 minutes. I picked the feed-forward width with an ablation and kept 4× d: 2,009 m and 33 of 50 survivors on L3 VAL, against 1,870 m and 30 at 2× d (130,311 weights).
Imitation has a well-known flaw: the student only sees states the teacher reached. After a small mistake, it is somewhere the teacher never went, and errors compound. DAgger (Dataset Aggregation) lets the student fly and asks the teacher to label what it reaches:
Each round adds about 60k frames and retrains warm-started for 3 epochs on all the data so far, 826,339 frames by the end of round 4. Calibration then flies arc greedily on L5 VAL until 20,000 frames are labelled: T = 0.32 cuts the expected calibration error (15 bins) from 0.458 to 0.155, and arc agrees with the teacher on 96.5% of frames. The model is exported to ONNX as arc.
On the 50 L5 VAL seeds, arc flies 1,818 m, and 24 of the 50 runs last the full 200 s. The planner flies 1,953 m on the same seeds, so arc’s VAL ratio is 0.93, above the 0.90 target. The whole DAgger pipeline, from teacher data to the ONNX file, takes about an hour.
DAgger can only make the student as good as the teacher’s 1.5 s view; PPO lets arc practise whole episodes. Each update runs 8 games for 1,024 steps, then 4 epochs in minibatches of 1,024, and the first 50k steps train only the critic. Actions are sampled at arc’s temperature (0.32), so practice flies like arc.
The reward is the game’s own, plus training-only shaping:
I left out two shaping terms from earlier runs on purpose: a cost on vertical speed and a bonus for a clear path ahead. Both made earlier PPO pilots start their moves late and cruise high, close to the ceiling.
My earlier PPO runs without the anchor started their moves about 0.5 s later than the pilot they began from, even with a strong teacher loss, because the soft labels barely tell “climb now” from “hold a little longer”. The KL term holds the policy itself. Let’s see it in code, shortened from train/ppo.py:
def ref_kl(logits, ref_logp, avail, temperature): """Mean KL(softmax(logits / T) || frozen arc) over the available actions.""" logp = torch.log_softmax(logits / temperature, -1).masked_fill(~avail, 0.0) p = logp.exp() * avail return (p * (logp - ref_logp.masked_fill(~avail, 0.0))).sum(-1).mean()# inside each minibatchratio = (new_logp - old_logp).exp() # rho_tpg = torch.max(-adv * ratio, -adv * ratio.clamp(1 - 0.1, 1 + 0.1)).mean()loss = pg - 0.005 * entropy + 0.5 * v_lossloss = loss + 0.5 * ref_kl(out["logits"], ref_logp, act_avail, 0.32) # stay close to arcloss = loss + 0.5 * teacher_loss # replayed teacher frames
Every 20 updates, I flew 10 VAL seeds on each of L1, L3 and L5 and kept the update with the best mean distance ratio. That update is calibrated and exported as arc_ppo.
PPO ran 1,000 updates, 8.19 million game steps, in 190 minutes. The practice flights changed in the direction I wanted:
The best update, 560, flew 1,937 m on the 50 L5 VAL seeds with 30 of 50 runs surviving, against arc’s 1,818 m and 24. Its calibration is close to arc’s:
Confound. Update 560 was picked on VAL seeds, so VAL flatters arc_ppo a little. On VAL seeds 10 to 49, which the selection never saw, the L5 gain shrinks from +120 to +76 ± 142 m. The fair comparison is TEST, which both pilots saw exactly once.
Let’s start with the headline number, the L5 TEST distance ratio from Equation 2, on 100 seeds neither pilot had seen. arc scores 0.89 and arc_ppo 0.94, at 0.14 ms per decision.
A single forward pass flies 89% (arc) and 94% (arc_ppo) as far as a 1.5 s beam search on the hardest level. On L1, 93% and 98% of runs fly the full 200 s. Neither pilot calls a planner, so the look-ahead lives in the weights.
Stress is the surprise. At 14.4 m/s, with gust, fog and surge at once, both pilots fly about as far as the planner: ratios of 0.98 and 1.00, so the stress drop is negative (−0.10 and −0.06). That does not mean the reflex beats planning there. Almost nobody survives stress, the planner included (3% of runs), so all three crash at a similar distance.
Over 10,000 calls on one CPU thread, act() takes 0.142 / 0.160 ms (p50 / p99) for arc and 0.141 / 0.155 ms for arc_ppo, well under the 1 ms target. In the game, the HUD shows more (0.45 ms in Figure 1), because the measurement includes a process that is also drawing the frame.
Note. G6 first failed*. On battery, in macOS Low Power Mode, arc’s p99 was 1.6 ms. On AC power, the same file measured 0.16 ms. If you benchmark on a laptop, check the power settings before you blame the model.*
Before training, I wrote down a pass mark for each stage.
A good pilot should look like a drone, not a pinball: mostly HOLD, a sensible height, slow moves first, started early. Measured greedily on 8 VAL seeds per level (“lead” is how far ahead the obstacle in its path was when the pilot left HOLD):
The timing came from the teacher, and it is early, not last-moment. In endless mode, over 18 long VAL runs on the same seeds, arc’s median run is 5.1 km with 39% of runs past 6 km. arc_ppo’s median is 6.8 km with 72% past 6 km, and it flew farther on 12 of the 18 seeds (+1,613 ± 832 m per run, paired).
The two pilots tie on mean distance over L1 to L5 TEST, at 1,982 m each. They get there by crashing into different things:
PPO nearly removed the ceiling crashes (31 to 5 on TEST, 22 to 4 in stress) and cut bursts by a third. It hits more buildings (24 to 34) and traffic drones (7 to 11), and the same pattern holds in stress (13 to 25 and 8 to 16). It is ahead on L1 and L5 and behind on L2 (−103 ± 51 m, paired).
The ceiling result fits the reward: arc’s ceiling crashes were mostly needless bursts, and PPO pays for every burst and more for a needless one. One hypothesis for the extra building and traffic crashes is that the same costs make arc_ppo slower to burst when a burst really is needed. I haven’t tested that.
Confound. The reference planner is also the teacher, so the ratio measures how close a student gets to its own teacher, not to an optimal pilot. And each pilot was trained once, so I can’t separate the method from run-to-run noise.
I started with one question: can a single forward pass, with no search, no memory and no look-ahead, fly a drone as far as the planner that taught it? Nearly. On the hardest level, on seeds it had never seen, arc flies 89% as far as its teacher and arc_ppo 94%, each in 0.14 ms per decision from a 1 MB file. The 1.5 s of search did not disappear. It moved from run time into 179,847 weights.
Four things made the difference, and I think they carry over to any project that distils a planner into a policy:
There are honest limits. DroneDash is a 2D simulation, not a real drone, and I haven’t tested any of this on hardware. arc is bounded by a teacher that sees 1.5 s ahead, and under stress it flies the full 200 s in only 2% of runs. Its remaining failures, needless bursts near the ceiling and climbs into storm clouds, are the next things to fix.
The code, the model card and the full run log are on GitHub: github.com/jaiganesan-n/reflex-drone.
Want to learn more? Check out my new article: Expert Transplant: I Replaced Gemma 3 1B’s FFNs With Another Model’s MoE Experts on a MacBook Air
Reflex Drone: I Trained a System 1 (Decision Model) Pilot ‘Arc’ That Flies by Navigating Obstacles was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.