Scaling Pre-Training on Super Smash Bros. Melee

An engineer trained HAL, a 247M-parameter causal Transformer, to play Super Smash Bros. Melee using human data only, reaching the top ~1% of human play on the ranked ladder. The model reads the last 256 frames of game state and controller inputs (no pixels) and outputs autoregressive action chunks, replanning at under 10 Hz, and the author reports it is the strongest base Melee model in the world. The work characterizes Melee pre-training scaling laws, finding model FLOPs should scale faster than data, and reports that observation delay hurts more than model size helps.

HAL https://ericyuegu.com/ Part 2 of 3 Scaling pre-training on Melee I trained a model named HAL to play Super Smash Bros. Melee using human data only and reached top ~1% of human play on ranked ladder<sup 1 user-content-fn-bots on ladder </sup . To my knowledge, HAL is the strongest base Melee model in the world. Here’s a short highlight reel of HAL playing Fox against humans names redacted with rank . Our model plays aggressively by chasing down opponents, finding surprising moves, but occasionally fumbling edge guard scenarios. I’ve included annotations for the non-Melee players. You can find full VODs on Twitch https://www.twitch.tv/hal 20xx/video/2890236934?filter=highlights&sort=time . Main takeaways This post is written with ML practitioners and technical readers in mind. The first blog post https://ericyuegu.com/behavior-cloning-melee in this series covered our motivations, problem setup, and behavior cloning BC approach. In this post, we… - add offline reinforcement learning offline-rl techniques like return conditioning, and find that requesting a constant 99th-percentile return on every frame gives stable play - modify the architecture to support autoregressive action chunking action-chunking , so the model only needs to replan at <10 Hz - characterize pre-training “scaling laws” compute-optimal-model-scaling for Melee , and find that model FLOPs should scale faster than data - show that observation delay hurts more than model size helps latency-aware-scaling - use a KV cache kv-cache that isn’t numerically exact, and find it works anyway - explore lots of other architecture, feature, and learning-objective ideas in the Appendix appendix The final recipe is a 247M-parameter causal Transformer that reads the last 256 frames of game state and controller inputs no pixels , then outputs a chunk of future controller inputs, decoded autoregressively frame by frame. Because it predicts coherent multi-frame chunks, HAL only needs to replan every few frames, which hides both inference time and netplay delay, enabling it to play online in real-time on varying hardware and network conditions. We can also steer its play style and difficulty at test time. Figure 1 fig-1 shows the whole recipe, which we unpack section by section. Why not reinforcement learning? There exists another Melee agent trained with behavior cloning + online RL named Phillip https://github.com/vladfi1/slippi-ai that is quite competitive with pros. If that’s the case, why not move on to online RL already? - My intention was always to pre-train on human data to warm start self-play RL. However, I was confident that pre-training a stronger base model leads to a stronger post-trained model . - My previous post left a lot of low-hanging fruit in feature representations, model architecture, learning objective, and optimization that I wanted to revisit. - It’s well known that RL is inefficient at exploration. If I can learn better priors and preserve more diversity, this can improve results downstream<sup 2 user-content-fn-rl exploration </sup . - I wanted to apply robotic learning techniques and find “scaling laws” in Melee . - In some ways, my problem setup is an idealized robotics environment. There is no deployment or embodiment gap between train and test, but I also have tons of diverse, expert human data. - I wanted to make the model available for people to play against. To do that, I needed to find a principled way to handle network latency for players all around the world. Over the last few months, I finally found the time and compute to chase these rabbit holes, the results of which I’m pleased to share. These directions were not necessarily the shortest path to superhuman Melee AI—at the end of the day, this was simply what piqued my curiosity. How to read results Over the course of several months, I ran over 50 experiments and trained hundreds of models. Except when stated otherwise, I tried to ablate ideas by training two models with equivalent compute, parameters, and data, then measuring differences in closed-loop eval<sup 3 user-content-fn-closedloop </sup over 96 episodes, 2 minutes each. I seeded a fixed set of stage and character match-ups that match the dataset’s prior distribution, then played each model against the in-game level 9 CPU, same as with our previous post. This was a cheap, reproducible baseline roughly on-par with the small-scale models. A player wins the game if they take all 4 of their opponent’s stocks , or lives. Our primary metric is net stocks per minute NSM : All else equal, we prefer a winning policy to take stocks faster and a losing one to lose stocks slower. NSM scales roughly linearly to Elo within , with +0.3 NSM ≈ +100 Elo. Outside of that range, NSM saturates asymptotically like Elo—our flagship model in the video above has only 1.7 NSM over the level 9 CPU but a 100% winrate over 192 games. At this sample size, NSM is accurate to small changes in behavior even across short episodes, which facilitates shorter evals. We should be careful when comparing absolute NSM across experiments, because configurations in model size, learning objective, and training duration varied across experiments. Some models were more undertrained than others, while later experiments tended to stack the accumulated techniques of earlier findings. Another flaw is that NSM is biased towards faster characters and smaller stages. To mitigate this, we can pair each treatment eval episode with a baseline episode of the same matchup, and call a result significant if the 95% confidence interval of a paired t-test https://en.wikipedia.org/wiki/Student%27s t-test Paired samples over per-game differences excludes zero. In general, I assumed that beneficial changes would have diminishing marginal returns but not destructively interfere. Offline RL Behavior cloning has well-known failure modes<sup 4 user-content-fn-dagger </sup . But, given the size of our dataset relative to the state space, we actually have excellent coverage of failure states<sup 5 user-content-fn-bc coverage </sup . Fundamentally, if our goal is for the model to be really good at winning Melee , imitating everything humans do isn’t always optimal. Even after filtering bad examples from our dataset, there are bound to be human mistakes and players who focus on style over winning. Can we make better use of the signals we have from the human dataset? Advantage-Weighted Regression https://arxiv.org/abs/1910.00177 AWR is behavior cloning where each training example’s loss is weighted by its advantage —how much better or worse the return after that action was than a baseline. The return is simply a discounted sum of rewards from a frame to the end of the replay. Through experimentation, I defined a symmetric reward function between two players: where is the percent damage a player takes on frame , is 1 on the frame they lose a stock, and is 1 if that was their last stock. How did I decide on these reward coefficients? A typical stock is played to around 90-120% damage. With discounting, I wanted to properly incentivize the model to prefer taking stocks over dealing more damage especially the last stock of the game , but in circumstances where that isn’t possible, to take favorable trades. It was important to tune , which determines the discount rate or half-life of the reward. Too short, and the earlier frames that actually caused the outcome don’t receive attribution. Too long, and unrelated frames get affected, possibly even encouraging degenerate behavior like keeping an opponent alive to farm more damage. I found that a half-life of around 8 seconds was 0.37 NSM better than 3 seconds and 0.05 better than 14 seconds. Including damage in the reward function gave an extra 0.2 NSM—otherwise the signal was too sparse. As for the baseline, I regress a value function head on the return. Similar states show up with many different continuations in the data, so learns their average the return a typical human gets from here . The advantage asks whether this particular action did better or worse than usual from the same state, and its clipped exponential weights the action’s negative log-likelihood in the loss: In Video 2 vid-2 , I color each character by its advantage, from purple negative through grey to orange positive . The return sums rewards from the current frame forward, so the color reflects the next several seconds of play more than the character’s current position. Under the dense reward, Kirby turns purple while hitting Falco at the start of the clip, because Falco is about to punish Kirby. Kirby then turns orange while taking those hits, because Kirby goes on to win the game. Under the sparse reward, only the final stock counts, so both characters stay close to grey for most of the clip. As a learning aid, I also created a visualization of how AWR re-weights the gradient in replays leading up to certain events, using real data and a heuristic for : AWR improved my model by 0.29 NSM over behavior cloning. Notably, AWR never assigns negative weight to any examples. Pushing the policy away from recorded actions could potentially learn worse or invalid actions. Instead, we can think of AWR as simply up-sampling and down-sampling training examples in proportion to how “good” the actions were. I found AWR to be more principled than training only on player perspectives that won, since there is valuable training signal from imitating skilled players who lost. What didn’t work While designing a reward function using damage, stocks, and winning worked, weighting training samples by skill ranking i.e. weighting the gradient for a Master rank player that of a Platinum player did not help. I didn’t have much luck learning a good Q-function, but I didn’t try very hard. I also tried IQL, which learns Q and V with TD learning and uses their difference as the advantage. Attaching Q and value heads to the policy scored −1.85 NSM, compared to +0.16 for plain behavior cloning, even with twice the data. Separate critic networks did about as badly −1.55 . The Q and V estimates drifted up past the clip ceiling and stayed there, so the advantages were basically noise. In hindsight, this made sense: function approximation, bootstrapping, and off-policy are known as the “ deadly triad https://arxiv.org/abs/1812.02648 ” of deep RL. Fortunately, I could directly supervise on dense ground-truth rewards and had no need to bootstrap a Q-function with TD learning. Return conditioning In general, more conditioning is a great way to factorize multimodal distributions. I added two inputs: which player to imitate player-conditioning and how much return it should go for. These double as levers for the model’s play style and “difficulty” at test time. Return conditioning is inspired by Decision Transformer https://arxiv.org/abs/2106.01345 . Unlike AWR which reshapes the training distribution, return conditioning is a knob I can turn at test time without retraining. At each frame, the label is the discounted return over the next 60 frames, with the same rewards and as AWR. The label only enters the action decoder, so the trunk and value head never see the future return see implementation return-conditioning-implementation . A problem I encountered was that I didn’t have a reliable learned distribution that I could sample the return-conditioning from. I had trained my value function on ground truth discounted return until the end of each replay, which has a pretty weak correlation to any arbitrary 1-second window during the game Pearson r = 0.471 . I didn’t want to take the effort to train a second value function just for this purpose, so at inference time, I request a constant return. As it turns out, requesting a constant 99th percentile return on every frame results in stable play, +0.6 NSM over no conditioning. | Requested return, next 60 frames | NSM | Stocks taken / min | Stocks lost / min | |---|---|---|---| | No conditioning | 0.80 | 1.42 | 0.62 | | 90th percentile | 1.20 | 2.01 | 0.81 | | 99th percentile | 1.41 | 2.00 | 0.59 | 90th percentile play is a lot more aggressive than no conditioning. In Video 3 vid-3 , 99th percentile play is more careful than 90th percentile without losing the aggression. This is unsurprising, because in the data, 1-second returns fall into a starkly bimodal distribution: windows with only damage <50 return , and 8% of windows that contain a stock event 100 return . Requesting a 90th percentile return in these scenarios result in the model choosing suboptimal actions. What’s interesting is that the model is able to generalize decision making associated with high return scenarios to other scenarios on stage at low percents typically nowhere near a kill , as seen in the highlight reel highlights . Return conditioning implementation I figured 60 frames was a reasonable horizon because our action chunks are only half that length. The label enters the action decoder as a FiLM<sup 6 user-content-fn-film </sup layer on each block’s attention and MLP RMSNorms, shared across the whole action chunk. The projections are zero-initialized, so training starts from the unconditioned model. Unlike the past-action FiLM in Inside each frame inside-each-frame , the return is an optional extra signal layered on a policy that already works, so starting from “no effect” is the right default. Player conditioning I had over half a million games from 38 pros with nearly 20k unique player codes, plus anonymized ranked games tagged by rank Platinum, Diamond, or Master . I told the model who it was playing as using a learned embedding for player code or rank. Here’s a 16M model playing as Fox against the level 9 CPU, conditioned on a couple players known for playing Fox: | Conditioned on | Net stock / min | Net damage / min | |---|---|---| | No one | 0.55 | 55.9 | | Cody | 0.63 | 87.0 | | Master | 0.53 | 60.1 | | Mang0 | 0.48 | 62.9 | | Platinum | 0.23 | 46.6 | Platinum conditioning is clearly worse than Master or either of the pro players, which is a nice sanity check. Across training checkpoints, player ID conditioning increased the Pearson r correlation between validation loss and net stocks/minute: | Evaluation suite | Player-conditioned BC | Unconditioned BC | |---|---|---| | Fox only | −0.84 | −0.59 | | Multiple character matchups | −0.92 | −0.59 | Action chunking Up until now, we trained a model that makes only next-frame predictions. Even with history, human data tends to be multimodal and non-Markovian due to different play styles and reaction times. What if we also want to support larger models with slow inference or variable network delay? We can predict actions for multiple steps ahead, also known as action chunking . In robotics, action chunking is thought to help https://www.haonanyu.blog/post/action chunking/ because it shortens the prediction horizon, improves temporal consistency, and helps with representation learning<sup 7 user-content-fn-chunking </sup . However, action chunking in robotics uses diffusion or flow matching models to predict continuous value trajectories<sup 8 user-content-fn-flow </sup . We currently have an autoregressive AR architecture that predicts discrete actions button presses . One option is to adapt flow matching to discrete actions, which I cover below flow-matching . A simple direction is to extend the next-token prediction objective into a multi-token prediction https://arxiv.org/abs/2404.19737 MTP objective. And this works pretty well You can add independent output heads, each tasked with predicting actions at a fixed frame offset . They don’t need to be consecutive either: adds about 0.5 NSM over next-frame only prediction. I reckon this works because lookahead reinforces the importance of choice points https://arxiv.org/pdf/2404.19737 page=26 in the loss, forcing the model to learn better representations to plan ahead. But that predicts future actions independently. We care about the joint probability distribution https://en.wikipedia.org/wiki/Joint probability distribution of actions over time, not just the marginal distribution. Two marginals that are individually likely but jointly incoherent might half-commit to running away vs. attacking in the same action chunk Figure 3a fig-3a . One way to improve MTP is presented in DeepSeek-V3 https://arxiv.org/pdf/2412.19437v1 page=10 , which feeds the th future token into the th MTP module. However, I have something better: another Transformer. Each frame’s action has four controller groups: c-stick, main stick, triggers, and buttons. As in the previous post, HAL already predicts these one after another within a frame, which is important for coordinating inputs like ↓ + B on the same frame. Now we extend that across time: each frame in the chunk is also conditioned on every frame before it. Given the observed state history , a chunk of future actions factorizes as where are the groups already predicted within frame . Since we want pretty long action chunks , rather than using a separate MTP module for each future offset, we can use a decoder-only Transformer as the action expert<sup 9 user-content-fn-action expert </sup and causally attend to the future action chunk Figure 3b fig-3b . Compared to DeepSeek-V3’s MTP design, we save parameters and improve expressivity. We don’t have new game state for these frames which have yet to happen, so our “input token” for the action expert is created by embedding the previous action at each time step and concatenating from the trunk, along with a position embedding. During training, teacher forcing supplies the ground-truth actions in parallel. At inference, those conditioning actions are instead the model’s own predictions . Unlike VLAs https://en.wikipedia.org/wiki/Vision%E2%80%93language%E2%80%93action model , we don’t need to worry about memory-intensive modalities like vision, so we can supervise loss for an action chunk at every prefix in the training example in parallel, just like we did for next-token prediction Figure 4 fig-4 . A decoder with dense supervision along the last 128 positions in its 256 context length trains 4 times faster and performs 1 NSM better than a model trained with loss only at its final token position, even though the losses at prefixes are correlated. While decoding a long action chunk at inference can be slower than flow matching, we can choose not to decode the entire chunk—only what is needed to support the replan horizon see Real-time inference real-time-inference . To sanity check that AR action chunking actually helps, I trained four 7.1M models that differ only in the decoder: one that predicts every frame and every controller group independently, one AR across controller groups within the same frame, one AR across frames, and one with both. We can compare the difference in performance when varying planning frequencies, with no added delay before each plan starts. When replanning every 4–6 frames, AR across frames is most beneficial, adding 0.4–1.0 NSM Figure 5 fig-5 . Another way to look at this result is that AR prediction across frames holds roughly the same performance while replanning less often. Replanning only once every six frames gives a lot of flexibility around inference and network latency. Qualitatively, Video 4 vid-4 shows that autoregressive action chunking results in smoother, more coherent actions across frames at long replan intervals, with fewer erratic, jerky controller movements. Flow matching One disadvantage of AR action chunking is that an early mistake locks you in. Unlike text domains, high frequency closed control problems like Melee are brutally unforgiving of errant actions. While in principle both paradigms should be able to factorize the joint distribution, I initially expected flow matching in practice to learn more coherent action chunks because it has multiple chances to revise earlier actions while looking at later ones. Unfortunately, even the analog inputs main-stick, c-stick, triggers are heavily clustered around a few hot spots due to notches https://multishine.tech/collections/wp-content/uploads/2022/01/melee-wavedash-notches-gamecube.png . Sparse and relatively discrete datasets are a poor setting for continuous flow matching. I tried naively flow matching on raw x, y values for the analog groups main stick, c-stick, triggers and one-hot button vectors and this totally failed. The only flow matching variant that I got working was endpoint flow over discretized actions<sup 10 user-content-fn-discreteflow </sup . Endpoint flow Each controller group at each of ten future offsets is a vector over its classes, linearly interpolated from Gaussian noise at to the one-hot target at . Instead of regressing a velocity with MSE, the model predicts logits for every slot’s class with cross-entropy, so estimates the clean chunk given . Each solver step moves every slot part of the remaining way toward that guess Figure 6 fig-6 <sup 11 user-content-fn-endpoint step </sup . Even with the chance to revise the whole chunk at every step, endpoint flow still didn’t beat AR decoding : −0.28 vs. −0.10 NSM at 7M, and +0.06 vs. +0.40 NSM at 15M, with the same trunk, data, and compute. Real-time inference For a near-lagless playing experience, Slippi netplay https://github.com/project-slippi/slippi-launcher/blob/main/FAQ.md what-are-delay-frames-and-how-do-i-change-them uses rollback netcode https://en.wikipedia.org/wiki/Netcode /media/File:Rollback Netcode Flowchart.png to process inputs client-side before resolving discrepancies when packets arrive from the opponent’s computer. It’s standard to play with 2 frames of action delay as network buffer. This means given an observation at time , if we press a controller button at , it won’t register in the game until . We need to account for this delay in order to run the model accurately in real-time over the internet. Let’s define a few terms: - Inference delay : frames the model takes to predict an action chunk. - Network delay : frames a controller input sits in Slippi’s queue. - Total observation delay - Replan interval : frames of each action chunk played open-loop before the next chunk replaces it. - Prediction horizon : length of each action chunk. A chunk has to cover the delay plus the open-loop interval, so . Generally, we request the next action chunk after the previous model inference finishes, so . We could technically overlap model inferences and pay the overhead, but as we’ve seen, short replanning intervals are not crucial to performance. Here’s an interactive visualization of these constraints: At test-time, I use real-time chunking<sup 12 user-content-fn-rtc </sup . The first actions of every chunk are already locked in by earlier chunks, so I force those actions in the action head and only sample actions after them. Given that I control the replan interval, I can stop decoding the action chunk at exactly , earlier than it was trained for. The model works well with observation delay despite not explicitly being trained for it because it has observation history and predicts action chunks autoregressively. To find the optimal prediction horizon, I played one fixed checkpoint against the level 9 CPU on a grid of observation delays and replan intervals Figure 8 fig-8 . I found that each additional frame of reaction delay hurts performance twice as much as each additional frame of open-loop action execution. Scaling Now that we have a working recipe, it’s time to scale up Bigger models are usually smarter, but they’re also slower. All else equal, they have to contend with more control delay. I had two central questions: 1. Open-loop scaling : What is the compute-optimal model size for the amount of data that I have? 2. Latency-aware closed-loop scaling : What model size plays best on fixed inference hardware, once inference delay is taken into account? To answer these questions, I ran two series of experiments: 1. Holding compute constant iso-FLOP , I trained models with varying parameter and dataset sizes, similar to Chinchilla https://arxiv.org/abs/2203.15556 . 2. Holding data constant iso-data , I trained models of varying sizes and evaluated them at the fastest replanning frequency their size would allow on my RTX 3060<sup 13 user-content-fn-affordable gpu </sup . Compute-optimal model scaling I ran compute-optimal scaling experiments and found that compute-optimal model FLOPS should grow faster than data. The usual rule of thumb is that training costs FLOPS. That assumes each token passes through every weight once, but my models don’t work that way—each token passes through the trunk once but through the decoder 36 times, once for each future action prediction<sup 14 user-content-fn-long action chunks </sup . Following DeepSeek<sup 15 user-content-fn-computesize </sup , I measure model scale as training FLOPS per supervised position instead: I trained a grid of model sizes on a range of data budgets, then fit to their validation NLL. The fit tracks the measured runs closely, with an RMS error of 0.004 Figure 9a fig-9a . Slicing that surface at a fixed training budget gives an iso-FLOP curve Figure 9b fig-9b . The minimum of each curve is the compute-optimal model size for that budget. Following those minima across budgets, the optimal model size and data grow roughly as These scaling terms differ from DeepSeek’s findings for LLMs and are consistent with what others have reported https://seohong.me/blog/behavioral-cloning-mystery/ in behavior cloning. Why? Perhaps extra model capacity is important for modeling the multimodality and noise of human demonstration data. Another way to interpret it is that data in this domain especially from the same episodes is highly correlated, and counts less than a fresh text token does. This exponent could also be an artifact of how I scaled the model parameters faster in the trunk than the decoder. For example, adding just a bit of width to the decoder but reducing the trunk params considerably so that the total model params dropped from 15M → 11M resulted in an equally strong policy. I did not run separate experiments to find the optimal decoder vs. trunk size. My dataset is finite, so the more useful view is: for supervised positions, the compute-optimal model grows roughly as , so the budget grows as . I made a tactical YOLO run with a model size that looks about larger than optimal in Figure 9c fig-9c . Its effective data seen is higher than suggested: it trained on more data, supervised sparser action chunks, and propagated loss at fewer prefixes per sequence, so its looks deceptively large and surprisingly small. It folded in many of the architectural tricks and learning objectives above that improve sample efficiency but were not included in smaller scaling runs. I was also confident that with inference optimizations, the larger model could run as fast as the smaller models. It does share one metric with every scaling run: the validation loss four frames ahead. On that metric Figure 9d fig-9d , it keeps improving well past the compute of any run in the fit due to the aforementioned tricks. Validation loss vs. closed-loop performance Does lower loss actually mean better play? Broadly. Across the scaling runs, each played at a 4-frame delay, lower validation NLL goes with more net stocks Spearman ρ = −0.66 over 28 runs; Figure 9e fig-9e . It’s too coarse to pick between checkpoints across runs, though. Within a single run Figure 9f fig-9f , validation loss sometimes stalls and NSM with it. Once the loss starts falling again, NSM climbs steadily. This finding is roughly consistent with my experience in robotics. Latency-aware scaling Open-loop validation loss isn’t the only metric we care about. For the second question, I evaluated each model at a delay that naive inference no KV cache takes on an RTX 3060 at batch 32. Here, both delay and replan interval increase with model size. The 6-frame model executes actions predicted 7–12 frames ahead, whereas the models in Figure 5 fig-5 execute actions predicted 1–6 frames ahead with no added delay. I trained each model size across 4 data budgets Figure 10 fig-10 . It’s clear that the smallest model is capacity-limited and doesn’t benefit from more data. An inverted U shape shows up as expected: the largest models lose more from slower reactivity than they gain from greater representational capacity. The loss-based fit didn’t predict this ordering because the cost of observation delay is high, and outweighs the benefit lower open-loop validation loss. Inference optimizations If a smaller, more reactive model is better than a bigger, slower model, then I want my flagship model as close to 60 fps as I can. Due to the complex architecture, the model has a lot of small elementwise operations and kernel launches. We used CUDA graphs https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html to capture and replay the compiled inference kernels with less dispatch overhead. This reduced batch 1 p95 latency on an RTX 3060 for the 247M model from 11.5 to 6.5 ms. That’s under one 16.7 ms frame, so the flagship gets a big model’s capacity at a small model’s delay. KV cache Melee has very few long-range dependencies: on average, the attention heads in my models focused on positions 60 frames in the past. Notably, I did not train with parallel sliding window attention on entire replays. During training, I prioritized replay diversity by sampling short windows of 256 frames to fit more distinct replays per batch<sup 16 user-content-fn-multiple windows </sup using plain causal self-attention. However, at inference time, this poses an issue. If I want to use a rolling KV cache https://huggingface.co/blog/not-lain/kv-caching to cut down on inference latency, the cache would retain state that contains information on tokens that have fallen outside the window Figure 11 fig-11 . The numerically correct way to match training is to fully recompute attention values on the full context window every inference. Fortunately, the DeepSeek-V4.1-Flash paper https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek V41 Tech Report.pdf emboldened me to try dropping old tokens from the KV cache anyway. In practice, play doesn’t suffer: the rolling cache scored 1.006 NSM vs. 0.979 NSM for full recomputation. Across 96 eval matches, the sampled tokens between full recomputation and rolling KV cache disagree in 0.84% of frames. I suspect that the model’s closed-loop generalization compensates for these numerical differences. KV caching reduced median latency by 42.3% at batch 64 on the RTX PRO 6000, with prediction horizon 8 and CUDA graphs on. | Mode | Median | p95 | |---|---|---| | Full recomputation | 44.62 ms | 44.78 ms | | Rolling KV cache | 25.74 ms | 25.85 ms | Results Our final run trained on a single B200 for 16 hours. With return conditioning set to 99th percentile, player conditioning set to Cody, playing as Fox, replanning every 2 frames, our 247M-parameter flagship model scored 1.7 NSM against the in-game level 9 CPU. This same configuration with 3 frames of observation delay scored 2200 Elo on the human ladder. Against Phillip We also benchmark against Phillip https://github.com/vladfi1/slippi-ai , currently the strongest Melee model in the world, which has been trained on the same dataset and undergone extensive self-play RL up to 1 week wall-clock time . Evaluation results are across all character matchups over 96-192 games: | Opponent | NSM | Win Rate | |---|---|---| | Level-9 CPU | 1.215 | 97.4% | | Our BC baseline Feb 2025 | 0.883 | 88.5% | | Phillip base | 1.050 | 94.8% | | Phillip RL | -0.661 | 25.0% | The clips below come from the 96 games against Phillip RL , including several where HAL loses the exchange. HAL plays the first character named in each caption. Qualitatively, our larger model with AR action chunking and return-conditioning plays in an aggressive yet coherent style, with fewer errant inputs that put its characters into disadvantageous states. AWR made the models play more strategically, claiming center stage control and holding edge guards longer and more consistently. Next up for us is self-play RL Our goal of creating the strongest Melee AI in the world is well within reach. Thank you for reading Acknowledgements Thank you to Jeffrey Cheng, Lev Stambler, and Sung Jae Bae for feedback on this article. A big thanks to Andrew Hinh and the Modal https://modal.com team for sponsoring the compute that made these experiments possible Appendix I share below both ideas that worked and ideas that I tried but couldn’t get to work. I didn’t always have time to prove why certain ideas performed better or worse, or that failed ideas hadn’t failed because of an implementation bug. Even so, I hope that simply reporting what I tried and found is interesting anyway. Inputs and outputs Last blog post, I made a few simplifying assumptions to the action space by discretizing analog controls and reducing concurrent button presses to the latest button. This can cause a lot of error accumulation in closed-loop. First, I revisited the way we map controller groups main stick, c-stick, triggers, buttons to target labels for the model. I included the L and R digital shoulder buttons and represented all possible combinations of 8 buttons A, B, X, Y, Z, L-digital, R-digital, D-pad right as a one-hot with classes instead of only classes. I also added more joint x, y locations to the main stick discretization, which previously hurt performance but performs better as we scale up capacity Figure 12 fig-12 , Video 5 vid-5 . Together, these changes give a nice 0.29 NSM bump. Encoding the main stick as polar coordinates provided marginal improvement within noise , and flow matching directly on continuous analog values was worse see Flow matching flow-matching . From this, I concluded main stick discretization was no longer a bottleneck. All of the game state and player features for a single frame are concatenated into one input token as before. This time, I also added projectiles and Ice Climbers’ Nana<sup 17 user-content-fn-nana </sup , with masking bits on frames when they don’t occur. Both appear in only a fraction of matchups, so they noticeably help there but otherwise don’t budge the bottom line much. For past actions, adding a tiny learned embedding and a linear projection of the raw x, y values for each controller group, then concatenating them with the state features, was noticeably helpful. Inside each frame Going back inside each frame, I made further improvements to how we predict each controller group given we have information on past-predicted frames in the action chunk. 1. We can use AdaRMSNorm to strongly modulate past-action conditioning on subsequent heads. This works much better than the feature concatenation that I used in the previous blog post. Curiously, I found zero-init FiLM to work worse than ordinary Kaiming uniform init for this. 2. We can add a trunk skip: mapping straight into each controller group head’s logits. Though the decoder’s tokens already have , I hypothesize this offloads a lot of work and feature bandwidth to the trunk, which can heavily pattern match controller actions simply based on the game state history, while turning the tiny, latency-sensitive decoder heads into small residual writers that only have to edit the priors at each time offset based on locally sampled actions. 3. Finally, using a SiLU MLP head over a linear head helps a lot. Essentially, at every turn, adding more expressivity and capacity to the decoder was disproportionately helpful to closed-loop performance. Other ideas that didn’t work I tried adding forms of cross-attention between the action expert’s tokens future actions and the trunk’s history tokens . It didn’t help much 0.07 NSM , but I kept a single cross-attention layer. It seems that a single latent mostly contains enough bandwidth to predict a chunk of future actions. Hand-engineered features like distance to opponent, distance to nearest ledge, hitlag left, etc. made model performance worse. Button presses and other controller actions tend to be correlated, but fairly discrete and sparse mostly zeros . I was inspired to try FAST https://www.pi.website/research/fast , but since discrete cosine transform struggles with sparse binary signals, I tried run-length encoding https://en.wikipedia.org/wiki/Run-length encoding actions over a 16-frame horizon before tokenizing them. A button sequence like AAANNNNNNXXXNNBB would turn into A, 3 , None, 6 , X, 3 , None, 2 , B, 2 . This didn’t pan out. The model was fairly non-reactive and would hold buttons for far too long. Performance hovered around −2 NSM. I also tried “chording”, or jointly representing the complete controller state for one frame in a single vocabulary. Since , I scanned the training data and built a vocab of roughly 8000 classes that covered 99.9% of frames. Unfortunately, the model didn’t converge to coherent play. I tried flattening the hierarchical autoregression by putting all of the controller actions across frames into one long sequence of tokens with the state. Basically, training a model to learn and so on. This increased sequence lengths roughly 5-fold for a 0.1 NSM gain, or at equal training budget, a −0.26 NSM change. I also tried training a “state expert” head with flow matching alongside the action expert head to predict next-frame features of the game and player states. It seems this somewhat fought for representation capacity and led to a −0.05 NSM change in performance—within noise. Even as I scaled up the model size up and discretized future state targets in the action chunk, this was strictly worse than devoting all parameters to policy alone. I also tried Explorative Modeling XM https://explorative-modeling.github.io/ , which generates candidates per training step and backpropagates loss for only the closest candidate. XM promises to teach the model to handle multimodal distributions by factoring the generation procedure, so I hoped that in lieu of using AR to factorize the action chunk, the XM model could generate all the controller parts at once. At , we saw net −0.8 NSM. Footnotes 1. 2200 Elo https://slippi.gg/user/hal-9000 within 5 hours. I do not condone running bots on Slippi without permission ↩ user-content-fnref-bots on ladder 2. Nair et al., 2018 https://arxiv.org/abs/1709.10089 ↩ user-content-fnref-rl exploration 3. As opposed to open-loop evaluation or training, where the model’s loss is measured against a teacher human demonstration, closed-loop means the policy controls the game. Each action changes the next observation it receives, just like real deployment. ↩ user-content-fnref-closedloop 4. Small mistakes take a cloned policy to states the demonstrations never visited, where it makes bigger mistakes. DAgger https://arxiv.org/abs/1011.0686 Ross et al., 2011 formalizes this as compounding error due to covariate shift. ↩ user-content-fnref-dagger 5. Our behavior cloning policy from part 1 https://ericyuegu.com/behavior-cloning-melee was excellent at returning to stage and recovering from unlikely inputs. ↩ user-content-fnref-bc coverage 6. FiLM https://arxiv.org/abs/1709.07871 Perez et al., 2018 conditions a network by predicting a per-feature scale and shift. Adaptive normalization, such as adaLN in DiT https://arxiv.org/abs/2212.09748 Peebles and Xie, 2023 , applies the same idea after a normalization layer. ↩ user-content-fnref-film 7. Popularized by ACT https://arxiv.org/abs/2304.13705 Zhao et al., 2023 and Diffusion Policy https://arxiv.org/abs/2303.04137 Chi et al., 2023 . There’s some countervailing evidence https://arxiv.org/abs/2608.02547 that action chunking works because it implicitly ensembles different policies with random delay. ↩ user-content-fnref-chunking 8. Black et al., 2024 https://arxiv.org/abs/2410.24164 trains a flow matching action expert on top of a pre-trained VLM backbone. ↩ user-content-fnref-flow 9. Term used interchangeably with “action output head.” ↩ user-content-fnref-action expert 10. Closely related to discrete flow matching https://arxiv.org/abs/2407.15595 Gat et al., 2024 , which defines flows directly over categorical variables. ↩ user-content-fnref-discreteflow 11. , with . ↩ user-content-fnref-endpoint step 12. Real-time chunking Physical Intelligence, 2025 https://www.pi.website/research/real time chunking ↩ user-content-fnref-rtc 13. I hoped that a mid-range consumer GPU from five years ago would be representative of the hardware most players have. ↩ user-content-fnref-affordable gpu 14. For the scaling runs, I made my action chunks longer to cover inference delays for potentially very large models. ↩ user-content-fnref-long action chunks 15. DeepSeek LLM https://arxiv.org/abs/2401.02954 Bi et al., 2024 replaces parameter count with non-embedding FLOPS per token, , with , and finds the compute-optimal and from iso-FLOP curves. The first term is ; the second counts attention over the context. I leave the attention term out: over a 128-frame context it adds , about 7% for my smallest model and 2% for my largest. Kaplan et al. 2020 https://arxiv.org/abs/2001.08361 drop it for the same reason. ↩ user-content-fnref-computesize 16. Trajectories within the same replay tend to be heavily correlated. Sampling 4 windows instead of 1 window from each replay within a few batches of each other reduced performance by 0.73 NSM. ↩ user-content-fnref-multiple windows 17. Ice Climbers are two characters: Popo, controlled by the player, and Nana, a partner that mostly repeats Popo’s inputs a few frames later but can be desynced https://www.ssbwiki.com/Desyncing in advanced play. ↩ user-content-fnref-nana Cite this post Gu, Eric. “Scaling pre-training on Melee.” ericyuegu.com , October 2026. https://ericyuegu.com/hillclimbing-melee https://ericyuegu.com/hillclimbing-melee @misc{gu2026melee, author = {Gu, Eric}, title = {Scaling pre-training on Melee}, year = {2026}, month = oct, url = {https://ericyuegu.com/hillclimbing-melee} }