{"slug": "scaling-pre-training-on-super-smash-bros-melee", "title": "Scaling Pre-Training on Super Smash Bros. Melee", "summary": "An engineer trained HAL, a 247M-parameter causal Transformer, to play Super Smash Bros. Melee using human data only, reaching the top ~1% of human play on the ranked ladder. The model reads the last 256 frames of game state and controller inputs (no pixels) and outputs autoregressive action chunks, replanning at under 10 Hz, and the author reports it is the strongest base Melee model in the world. The work characterizes Melee pre-training scaling laws, finding model FLOPs should scale faster than data, and reports that observation delay hurts more than model size helps.", "body_md": "[HAL](https://ericyuegu.com/)Part 2 of 3\n\n# Scaling pre-training on Melee\n\nI trained a model named HAL to play *Super Smash Bros. Melee* using **human data only** and reached **top ~1% of human play** on ranked ladder<sup>[1](#user-content-fn-bots_on_ladder)</sup>.\nTo my knowledge, HAL is the strongest base *Melee* model in the world.\n\nHere’s a short highlight reel of HAL playing Fox against humans (names redacted with rank). Our model plays aggressively by chasing down opponents, finding surprising moves, but occasionally fumbling edge guard scenarios. I’ve included annotations for the non-Melee players.\n\nYou can find [full VODs on Twitch](https://www.twitch.tv/hal_20xx/video/2890236934?filter=highlights&sort=time).\n\n## Main takeaways\n\nThis post is written with ML practitioners and technical readers in mind.\nThe [first blog post](https://ericyuegu.com/behavior-cloning-melee) in this series covered our motivations, problem setup, and behavior cloning (BC) approach.\n\nIn this post, we…\n\n- add [offline reinforcement learning](#offline-rl) techniques like return conditioning, and find that requesting a constant 99th-percentile return on every frame gives stable play\n- modify the architecture to support [autoregressive action chunking](#action-chunking) , so the model only needs to replan at <10 Hz\n- characterize [pre-training “scaling laws”](#compute-optimal-model-scaling) for*Melee* , and find that model FLOPs should scale faster than data\n- show that [observation delay hurts more than model size helps](#latency-aware-scaling)\n- use a [KV cache](#kv-cache) that isn’t numerically exact, and find it works anyway\n- explore lots of other architecture, feature, and learning-objective ideas in the [Appendix](#appendix) !\n\nThe final recipe is a 247M-parameter causal Transformer that reads the last 256 frames of game state and controller inputs (no pixels), then outputs a chunk of future controller inputs, decoded autoregressively frame by frame.\n\nBecause it predicts coherent multi-frame chunks, HAL only needs to replan every few frames, which hides both inference time and netplay delay, enabling it to play online in real-time on varying hardware and network conditions. We can also steer its play style and difficulty at test time.\n\n[Figure 1](#fig-1) shows the whole recipe, which we unpack section by section.\n\n## Why not reinforcement learning?\n\nThere exists another *Melee* agent trained with behavior cloning + online RL named [Phillip](https://github.com/vladfi1/slippi-ai) that is quite competitive with pros.\nIf that’s the case, why not move on to online RL already?\n\n- My intention was always to pre-train on human data to warm start self-play RL. However, I was confident that **pre-training a stronger base model leads to a stronger post-trained model** .\n  - My previous post left a lot of low-hanging fruit in feature representations, model architecture, learning objective, and optimization that I wanted to revisit.\n  - It’s well known that RL is inefficient at exploration. If I can learn better priors and preserve more diversity, this can improve results downstream<sup>[2](#user-content-fn-rl_exploration)</sup> .\n- **I wanted to apply robotic learning techniques and find “scaling laws” in *Melee*.**  - In some ways, my problem setup is an idealized robotics environment. There is no deployment or embodiment gap between train and test, but I also have tons of diverse, expert human data.\n- I wanted to make the model available for people to play against. To do that, I needed to find a principled way to handle network latency for players all around the world.\n\nOver the last few months, I finally found the time and compute to chase these rabbit holes, the results of which I’m pleased to share.\nThese directions were not necessarily the shortest path to superhuman *Melee* AI—at the end of the day, this was simply what piqued my curiosity.\n\n## How to read results\n\nOver the course of several months, I ran over 50 experiments and trained hundreds of models.\n\nExcept when stated otherwise, I tried to ablate ideas by training two models with equivalent compute, parameters, and data, then measuring differences in closed-loop eval<sup>[3](#user-content-fn-closedloop)</sup> over 96 episodes, 2 minutes each.\nI seeded a fixed set of stage and character match-ups that match the dataset’s prior distribution, then played each model against the in-game level 9 CPU, same as with our previous post.\nThis was a cheap, reproducible baseline roughly on-par with the small-scale models.\n\nA player wins the game if they take all 4 of their opponent’s *stocks*, or lives.\n**Our primary metric is net stocks per minute (NSM)**:\n\nAll else equal, we prefer a winning policy to take stocks faster and a losing one to lose stocks slower.\n\nNSM scales roughly linearly to Elo within , with +0.3 NSM ≈ +100 Elo. Outside of that range, NSM saturates asymptotically like Elo—our flagship model in the video above has only 1.7 NSM over the level 9 CPU but a 100% winrate over 192 games. At this sample size, NSM is accurate to small changes in behavior even across short episodes, which facilitates shorter evals.\n\nWe should be careful when comparing absolute NSM across experiments, because configurations in model size, learning objective, and training duration varied across experiments. Some models were more undertrained than others, while later experiments tended to stack the accumulated techniques of earlier findings.\n\nAnother flaw is that NSM is biased towards faster characters and smaller stages.\nTo mitigate this, we can pair each treatment eval episode with a baseline episode of the same matchup, and call a result significant if the 95% confidence interval of a [paired t-test](https://en.wikipedia.org/wiki/Student%27s_t-test#Paired_samples) over per-game differences excludes zero.\n\nIn general, I assumed that beneficial changes would have diminishing marginal returns but not destructively interfere.\n\n## Offline RL\n\nBehavior cloning has well-known failure modes<sup>[4](#user-content-fn-dagger)</sup>.\nBut, given the size of our dataset relative to the state space, we actually have excellent coverage of failure states<sup>[5](#user-content-fn-bc_coverage)</sup>.\n\nFundamentally, if our goal is for the model to be really good at winning *Melee*, imitating everything humans do isn’t always optimal.\nEven after filtering bad examples from our dataset, there are bound to be human mistakes and players who focus on style over winning.\nCan we make better use of the signals we have from the human dataset?\n\n**[Advantage-Weighted Regression](https://arxiv.org/abs/1910.00177) (AWR) is behavior cloning where each training example’s loss is weighted by its *advantage*** —how much better or worse the return after that action was than a baseline.\n\nThe *return*  is simply a discounted sum of rewards from a frame to the end of the replay.\n\nThrough experimentation, I defined a symmetric *reward function* between two players:\n\nwhere is the percent damage a player takes on frame , is 1 on the frame they lose a stock, and is 1 if that was their last stock.\n\nHow did I decide on these reward coefficients? A typical stock is played to around 90-120% damage. With discounting, I wanted to properly incentivize the model to prefer taking stocks over dealing more damage (especially the last stock of the game), but in circumstances where that isn’t possible, to take favorable trades.\n\nIt was important to tune , which determines the discount rate or half-life of the reward. Too short, and the earlier frames that actually caused the outcome don’t receive attribution. Too long, and unrelated frames get affected, possibly even encouraging degenerate behavior like keeping an opponent alive to farm more damage. I found that a half-life of around 8 seconds was 0.37 NSM better than 3 seconds and 0.05 better than 14 seconds. Including damage in the reward function gave an extra 0.2 NSM—otherwise the signal was too sparse.\n\nAs for the baseline, I regress a value function head on the return. Similar states show up with many different continuations in the data, so learns their average (the return a typical human gets from here). The advantage asks whether this particular action did better or worse than usual from the same state, and its clipped exponential weights the action’s negative log-likelihood in the loss:\n\nIn [Video 2](#vid-2), I color each character by its advantage, from purple (negative) through grey to orange (positive).\nThe return sums rewards from the current frame forward, so the color reflects the next several seconds of play more than the character’s current position.\nUnder the dense reward, Kirby turns purple while hitting Falco at the start of the clip, because Falco is about to punish Kirby.\nKirby then turns orange while taking those hits, because Kirby goes on to win the game.\nUnder the sparse reward, only the final stock counts, so both characters stay close to grey for most of the clip.\n\nAs a learning aid, I also created a visualization of how AWR re-weights the gradient in replays leading up to certain events, using real data and a heuristic for :\n\n**AWR improved my model by 0.29 NSM over behavior cloning.**\n\nNotably, AWR never assigns *negative* weight to any examples.\nPushing the policy away from recorded actions could potentially learn worse or invalid actions.\nInstead, we can think of AWR as simply up-sampling and down-sampling training examples in proportion to how “good” the actions were.\n\nI found AWR to be more principled than training only on player perspectives that won, since there is valuable training signal from imitating skilled players who lost.\n\n## #### What didn’t work\n\nWhile designing a reward function using damage, stocks, and winning worked, weighting training samples by skill ranking (i.e. weighting the gradient for a Master rank player that of a Platinum player) did not help.\n\nI didn’t have much luck learning a good Q-function, but I didn’t try very hard.\n\nI also tried IQL, which learns Q and V with TD learning and uses their difference as the advantage.\nAttaching Q and value heads to the policy scored −1.85 NSM, compared to +0.16 for plain behavior cloning, even with twice the data.\nSeparate critic networks did about as badly (−1.55).\nThe Q and V estimates drifted up past the clip ceiling and stayed there, so the advantages were basically noise.\nIn hindsight, this made sense: function approximation, bootstrapping, and off-policy are known as the “[deadly triad](https://arxiv.org/abs/1812.02648)” of deep RL.\n\nFortunately, I could directly supervise on dense ground-truth rewards and had no need to bootstrap a Q-function with TD learning.\n\n### Return conditioning\n\nIn general, more conditioning is a great way to factorize multimodal distributions.\n\nI added two inputs: [which player to imitate](#player-conditioning) and how much return it should go for.\nThese double as levers for the model’s play style and “difficulty” at test time.\n\nReturn conditioning is inspired by [Decision Transformer](https://arxiv.org/abs/2106.01345).\nUnlike AWR which reshapes the training distribution, return conditioning is a knob I can turn at test time without retraining.\n\nAt each frame, the label is the discounted return over the next 60 frames, with the same rewards and  as AWR.\nThe label only enters the action decoder, so the trunk and value head never see the future return (see [implementation](#return-conditioning-implementation)).\n\nA problem I encountered was that I didn’t have a reliable learned distribution that I could sample the return-conditioning from. I had trained my value function on ground truth discounted return until the end of each replay, which has a pretty weak correlation to any arbitrary 1-second window during the game (Pearson r = 0.471). I didn’t want to take the effort to train a second value function just for this purpose, so at inference time, I request a constant return.\n\nAs it turns out, **requesting a constant 99th percentile return on every frame results in stable play, +0.6 NSM over no conditioning.**\n\n| Requested return, next 60 frames | NSM | Stocks taken / min | Stocks lost / min | \n|---|---|---|---|\n| No conditioning | 0.80 | 1.42 | 0.62 | \n| 90th percentile | 1.20 | 2.01 | 0.81 | \n| 99th percentile | 1.41 | 2.00 | 0.59 | \n\n90th percentile play is a lot more aggressive than no conditioning.\nIn [Video 3](#vid-3), 99th percentile play is more careful than 90th percentile without losing the aggression.\n\nThis is unsurprising, because in the data, 1-second returns fall into a starkly bimodal distribution: windows with only damage (<50 return), and 8% of windows that contain a stock event (>100 return). Requesting a 90th percentile return in these scenarios result in the model choosing suboptimal actions.\n\nWhat’s interesting is that the model is able to generalize decision making associated with high return scenarios to other scenarios on stage at low percents (typically nowhere near a kill), as seen in the [highlight reel](#highlights).\n\n## #### Return conditioning implementation\n\nI figured 60 frames was a reasonable horizon because our action chunks are only half that length.\n\nThe label enters the action decoder as a FiLM<sup>[6](#user-content-fn-film)</sup> layer on each block’s attention and MLP RMSNorms, shared across the whole action chunk.\nThe projections are zero-initialized, so training starts from the unconditioned model. Unlike the past-action FiLM in [Inside each frame](#inside-each-frame), the return is an optional extra signal layered on a policy that already works, so starting from “no effect” is the right default.\n\n### Player conditioning\n\nI had over half a million games from 38 pros with nearly 20k unique player codes, plus anonymized ranked games tagged by rank (Platinum, Diamond, or Master). I told the model who it was playing as using a learned embedding for player code or rank.\n\nHere’s a 16M model playing as Fox against the level 9 CPU, conditioned on a couple players known for playing Fox:\n\n| Conditioned on | Net stock / min | Net damage / min | \n|---|---|---|\n| No one | 0.55 | 55.9 | \n| Cody | 0.63 | 87.0 | \n| Master | 0.53 | 60.1 | \n| Mang0 | 0.48 | 62.9 | \n| Platinum | 0.23 | 46.6 | \n\nPlatinum conditioning is clearly worse than Master or either of the pro players, which is a nice sanity check.\n\nAcross training checkpoints, player ID conditioning increased the Pearson r correlation between validation loss and net stocks/minute:\n\n| Evaluation suite | Player-conditioned BC | Unconditioned BC | \n|---|---|---|\n| Fox only | −0.84 | −0.59 | \n| Multiple character matchups | −0.92 | −0.59 | \n\n## Action chunking\n\nUp until now, we trained a model that makes only next-frame predictions. Even with history, human data tends to be multimodal and non-Markovian due to different play styles and reaction times. What if we also want to support larger models with slow inference or variable network delay?\n\n**We can predict actions for multiple steps ahead,** also known as *action chunking*.\nIn robotics, [action chunking is thought to help](https://www.haonanyu.blog/post/action_chunking/) because it shortens the prediction horizon, improves temporal consistency, and helps with representation learning<sup>[7](#user-content-fn-chunking)</sup>.\n\nHowever, action chunking in robotics uses diffusion or flow matching models to predict continuous value trajectories<sup>[8](#user-content-fn-flow)</sup>.\nWe currently have an autoregressive (AR) architecture that predicts discrete actions (button presses).\nOne option is to adapt flow matching to discrete actions, which I cover [below](#flow-matching).\n\nA simple direction is to extend the next-token prediction objective into a [multi-token prediction](https://arxiv.org/abs/2404.19737) (MTP) objective.\nAnd this works pretty well!\nYou can add  independent output heads, each tasked with predicting actions at a fixed frame offset .\nThey don’t need to be consecutive either:  adds about 0.5 NSM over next-frame only prediction.\nI reckon this works because lookahead reinforces the importance of [choice points](https://arxiv.org/pdf/2404.19737#page=26) in the loss, forcing the model to learn better representations to plan ahead.\n\nBut that predicts future actions independently.\nWe care about the [joint probability distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution) of actions over time, not just the marginal distribution.\nTwo marginals that are individually likely but jointly incoherent might half-commit to running away vs. attacking in the same action chunk ([Figure 3a](#fig-3a)).\n\nOne way to improve MTP is presented in [DeepSeek-V3](https://arxiv.org/pdf/2412.19437v1#page=10), which feeds the th future token into the th MTP module.\nHowever, I have something better: another Transformer.\n\nEach frame’s action has four controller groups: c-stick, main stick, triggers, and buttons.\nAs in the previous post, HAL already predicts these one after another within a frame, which is important for coordinating inputs like `↓ + B` on the same frame.\nNow we extend that across time: each frame in the chunk is also conditioned on every frame before it.\nGiven the observed state history , a chunk of  future actions factorizes as\n\nwhere are the groups already predicted within frame .\n\nSince we want pretty long action chunks (), rather than using a separate MTP module for each future offset, we can use a decoder-only Transformer as the action expert<sup>[9](#user-content-fn-action_expert)</sup> and causally attend to the future action chunk ([Figure 3b](#fig-3b)).\nCompared to DeepSeek-V3’s MTP design, we save parameters and improve expressivity.\n\nWe don’t have new game state for these frames which have yet to happen, so our “input token” for the action expert is created by embedding the previous action at each time step and concatenating from the trunk, along with a position embedding. During training, teacher forcing supplies the ground-truth actions in parallel. At inference, those conditioning actions are instead the model’s own predictions .\n\nUnlike [VLAs](https://en.wikipedia.org/wiki/Vision%E2%80%93language%E2%80%93action_model), we don’t need to worry about memory-intensive modalities like vision, so we can supervise loss for an action chunk at every prefix in the training example in parallel, just like we did for next-token prediction ([Figure 4](#fig-4)).\n**A decoder with dense supervision along the last 128 positions in its 256 context length trains >4 times faster and performs >1 NSM better** than a model trained with loss only at its final token position, even though the losses at prefixes are correlated.\n\nWhile decoding a long action chunk at inference can be slower than flow matching, we can choose not to decode the entire chunk—only what is needed to support the replan horizon (see [Real-time inference](#real-time-inference)).\n\nTo sanity check that AR action chunking actually helps, I trained four 7.1M models that differ only in the decoder: one that predicts every frame and every controller group independently, one AR across controller groups within the same frame, one AR across frames, and one with both.\n\nWe can compare the difference in performance when varying planning frequencies, with no added delay before each plan starts.\nWhen replanning every 4–6 frames, AR across frames is most beneficial, adding 0.4–1.0 NSM ([Figure 5](#fig-5)).\nAnother way to look at this result is that **AR prediction across frames holds roughly the same performance while replanning  less often.**\nReplanning only once every six frames gives a lot of flexibility around inference and network latency.\n\nQualitatively, [Video 4](#vid-4) shows that autoregressive action chunking results in smoother, more coherent actions across frames at long replan intervals, with fewer erratic, jerky controller movements.\n\n## #### Flow matching\n\nOne disadvantage of AR action chunking is that an early mistake locks you in.\nUnlike text domains, high frequency closed control problems like *Melee* are brutally unforgiving of errant actions.\n\nWhile in principle both paradigms should be able to factorize the joint distribution, I initially expected flow matching in practice to learn more coherent action chunks because it has multiple chances to revise earlier actions while looking at later ones.\nUnfortunately, even the analog inputs (main-stick, c-stick, triggers) are heavily clustered around a few hot spots due to [notches](https://multishine.tech/collections/wp-content/uploads/2022/01/melee-wavedash-notches-gamecube.png).\nSparse and relatively discrete datasets are a poor setting for continuous flow matching.\n\nI tried naively flow matching on raw `(x, y)` values for the analog groups (main stick, c-stick, triggers) and one-hot button vectors and this totally failed.\n\nThe only flow matching variant that I got working was endpoint flow over discretized actions<sup>[10](#user-content-fn-discreteflow)</sup>.\n\n##### Endpoint flow\n\nEach controller group at each of ten future offsets is a vector over its classes, linearly interpolated from Gaussian noise  at  to the one-hot target  at .\nInstead of regressing a velocity with MSE, the model predicts logits  for every slot’s class with cross-entropy, so  estimates the clean chunk given .\nEach solver step moves every slot part of the remaining way toward that guess ([Figure 6](#fig-6))<sup>[11](#user-content-fn-endpoint_step)</sup>.\n\nEven with the chance to revise the whole chunk at every step, **endpoint flow still didn’t beat AR decoding**: −0.28 vs. −0.10 NSM at 7M, and +0.06 vs. +0.40 NSM at 15M, with the same trunk, data, and compute.\n\n### Real-time inference\n\nFor a near-lagless playing experience, [Slippi netplay](https://github.com/project-slippi/slippi-launcher/blob/main/FAQ.md#what-are-delay-frames-and-how-do-i-change-them) uses [rollback netcode](https://en.wikipedia.org/wiki/Netcode#/media/File:Rollback_Netcode_Flowchart.png) to process inputs client-side before resolving discrepancies when packets arrive from the opponent’s computer.\nIt’s standard to play with 2 frames of action delay as network buffer.\nThis means given an observation at time , if we press a controller button at , it won’t register in the game until .\n\nWe need to account for this delay in order to run the model accurately in real-time over the internet. Let’s define a few terms:\n\n- **Inference delay :** frames the model takes to predict an action chunk.\n- **Network delay :** frames a controller input sits in Slippi’s queue.\n- **Total observation delay** \n- **Replan interval :** frames of each action chunk played open-loop before the next chunk replaces it.\n- **Prediction horizon :** length of each action chunk. A chunk has to cover the delay plus the open-loop interval, so .\n\nGenerally, we request the next action chunk after the previous model inference finishes, so . (We could technically overlap model inferences and pay the overhead, but as we’ve seen, short replanning intervals are not crucial to performance.)\n\nHere’s an interactive visualization of these constraints:\n\nAt test-time, I use real-time chunking<sup>[12](#user-content-fn-rtc)</sup>.\nThe first  actions of every chunk are already locked in by earlier chunks, so I force those actions in the action head and only sample actions after them.\nGiven that I control the replan interval, I can stop decoding the action chunk at exactly , earlier than it was trained for.\n\nThe model works well with observation delay despite not explicitly being trained for it because it has observation history and predicts action chunks autoregressively.\n\nTo find the optimal prediction horizon, I played one fixed checkpoint against the level 9 CPU on a grid of observation delays and replan intervals ([Figure 8](#fig-8)).\nI found that **each additional frame of reaction delay hurts performance twice as much as each additional frame of open-loop action execution.**\n\n## Scaling\n\nNow that we have a working recipe, it’s time to scale up!\n\nBigger models are usually smarter, but they’re also slower. All else equal, they have to contend with more control delay.\n\nI had two central questions:\n\n1. **Open-loop scaling** : What is the compute-optimal model size for the amount of data that I have?\n2. **Latency-aware closed-loop scaling** : What model size plays best on fixed inference hardware, once inference delay is taken into account?\n\nTo answer these questions, I ran two series of experiments:\n\n1. Holding compute constant (iso-FLOP), I trained models with varying parameter and dataset sizes, similar to [Chinchilla](https://arxiv.org/abs/2203.15556) .\n2. Holding data constant (iso-data), I trained models of varying sizes and evaluated them at the fastest replanning frequency their size would allow on my RTX 3060<sup>[13](#user-content-fn-affordable_gpu)</sup> .\n\n### Compute-optimal model scaling\n\nI ran compute-optimal scaling experiments and found that **compute-optimal model FLOPS should grow faster than data.**\n\nThe usual rule of thumb is that training costs  FLOPS.\nThat assumes each token passes through every weight once, but my models don’t work that way—each token passes through the trunk once but through the decoder 36 times, once for each future action prediction<sup>[14](#user-content-fn-long_action_chunks)</sup>.\n\nFollowing DeepSeek<sup>[15](#user-content-fn-computesize)</sup>, I measure model scale as training FLOPS per supervised position instead:\n\nI trained a grid of model sizes on a range of data budgets, then fit  to their validation NLL.\nThe fit tracks the measured runs closely, with an RMS error of 0.004 ([Figure 9a](#fig-9a)).\n\nSlicing that surface at a fixed training budget gives an iso-FLOP curve ([Figure 9b](#fig-9b)).\nThe minimum of each curve is the compute-optimal model size for that budget.\n\nFollowing those minima across budgets, the optimal model size and data grow roughly as\n\nThese scaling terms differ from DeepSeek’s findings for LLMs () and are consistent with what [others have reported](https://seohong.me/blog/behavioral-cloning-mystery/) in behavior cloning.\n\nWhy? Perhaps extra model capacity is important for modeling the multimodality and noise of human demonstration data. Another way to interpret it is that data in this domain (especially from the same episodes) is highly correlated, and counts less than a fresh text token does.\n\nThis exponent could also be an artifact of how I scaled the model parameters faster in the trunk than the decoder. For example, adding just a bit of width to the decoder but reducing the trunk params considerably so that the total model params dropped from 15M → 11M resulted in an equally strong policy. I did not run separate experiments to find the optimal decoder vs. trunk size.\n\nMy dataset is finite, so the more useful view is: for supervised positions, the compute-optimal model grows roughly as , so the budget grows as .\n\nI made a tactical YOLO run with a model size that looks about  larger than optimal in [Figure 9c](#fig-9c).\nIts effective data seen is higher than suggested: it trained on  more data, supervised sparser action chunks, and propagated loss at fewer prefixes per sequence, so its  looks deceptively large and  surprisingly small.\nIt folded in many of the architectural tricks and learning objectives above that improve sample efficiency but were not included in smaller scaling runs.\nI was also confident that with inference optimizations, the larger model could run as fast as the smaller models.\n\nIt does share one metric with every scaling run: the validation loss four frames ahead.\nOn that metric ([Figure 9d](#fig-9d)), it keeps improving well past the compute of any run in the fit due to the aforementioned tricks.\n\n## #### Validation loss vs. closed-loop performance\n\nDoes lower loss actually mean better play?\nBroadly.\nAcross the scaling runs, each played at a 4-frame delay, lower validation NLL goes with more net stocks (Spearman ρ = −0.66 over 28 runs; [Figure 9e](#fig-9e)).\nIt’s too coarse to pick between checkpoints across runs, though.\n\nWithin a single run ([Figure 9f](#fig-9f)), validation loss sometimes stalls and NSM with it.\nOnce the loss starts falling again, NSM climbs steadily.\n\nThis finding is roughly consistent with my experience in robotics.\n\n### Latency-aware scaling\n\nOpen-loop validation loss isn’t the only metric we care about.\n\nFor the second question, I evaluated each model at a delay that naive inference (no KV cache) takes on an RTX 3060 at batch 32.\nHere, both delay and replan interval increase with model size.\nThe 6-frame model executes actions predicted 7–12 frames ahead, whereas the models in [Figure 5](#fig-5) execute actions predicted 1–6 frames ahead with no added delay.\n\nI trained each model size across 4 data budgets ([Figure 10](#fig-10)).\n\nIt’s clear that the smallest model is capacity-limited and doesn’t benefit from more data.\nAn inverted U shape shows up as expected: **the largest models lose more from slower reactivity than they gain from greater representational capacity.**\nThe loss-based fit didn’t predict this ordering because the cost of observation delay is high, and outweighs the benefit lower open-loop validation loss.\n\n## Inference optimizations\n\nIf a smaller, more reactive model is better than a bigger, slower model, then I want my flagship model as close to 60 fps as I can.\n\nDue to the complex architecture, the model has a lot of small elementwise operations and kernel launches.\n\nWe used [CUDA graphs](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html) to capture and replay the compiled inference kernels with less dispatch overhead.\nThis reduced batch 1 p95 latency on an RTX 3060 for the 247M model from 11.5 to 6.5 ms.\nThat’s under one 16.7 ms frame, so the flagship gets a big model’s capacity at a small model’s delay.\n\n### KV cache\n\n*Melee* has very few long-range dependencies: on average, the attention heads in my models focused on positions 60 frames in the past.\nNotably, I did *not* train with parallel sliding window attention on entire replays.\nDuring training, I prioritized replay diversity by sampling short windows of 256 frames to fit more distinct replays per batch<sup>[16](#user-content-fn-multiple_windows)</sup> using plain causal self-attention.\n\nHowever, at inference time, this poses an issue.\nIf I want to use a rolling [KV cache](https://huggingface.co/blog/not-lain/kv-caching) to cut down on inference latency, the cache would retain state that contains information on tokens that have fallen outside the window ([Figure 11](#fig-11)).\nThe numerically correct way to match training is to fully recompute attention values on the full context window every inference.\n\nFortunately, the [DeepSeek-V4.1-Flash paper](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf) emboldened me to try dropping old tokens from the KV cache anyway.\nIn practice, play doesn’t suffer: the rolling cache scored 1.006 NSM vs. 0.979 NSM for full recomputation.\nAcross 96 eval matches, the sampled tokens between full recomputation and rolling KV cache disagree in 0.84% of frames.\nI suspect that the model’s closed-loop generalization compensates for these numerical differences.\n\nKV caching reduced median latency by 42.3% at batch 64 on the RTX PRO 6000, with prediction horizon 8 and CUDA graphs on.\n\n| Mode | Median | p95 | \n|---|---|---|\n| Full recomputation | 44.62 ms | 44.78 ms | \n| Rolling KV cache | **25.74 ms** | **25.85 ms** | \n\n## Results\n\nOur final run trained on a single B200 for 16 hours.\n\nWith return conditioning set to 99th percentile, player conditioning set to Cody, playing as Fox, replanning every 2 frames, our 247M-parameter flagship model scored 1.7 NSM against the in-game level 9 CPU. This same configuration (with 3 frames of observation delay) scored 2200 Elo on the human ladder.\n\n### Against Phillip\n\nWe also benchmark against [Phillip](https://github.com/vladfi1/slippi-ai), currently the strongest *Melee* model in the world, which has been trained on the same dataset and undergone extensive self-play RL (up to 1 week wall-clock time).\nEvaluation results are across all character matchups over 96-192 games:\n\n| Opponent | NSM | Win Rate | \n|---|---|---|\n| Level-9 CPU | 1.215 | 97.4% | \n| Our BC baseline (Feb 2025) | 0.883 | 88.5% | \n| Phillip (base) | 1.050 | 94.8% | \n| Phillip (RL) | -0.661 | 25.0% | \n\nThe clips below come from the 96 games against Phillip (RL), including several where HAL loses the exchange. HAL plays the first character named in each caption.\n\nQualitatively, our larger model with AR action chunking and return-conditioning plays in an aggressive yet coherent style, with fewer errant inputs that put its characters into disadvantageous states. AWR made the models play more strategically, claiming center stage control and holding edge guards longer and more consistently.\n\nNext up for us is self-play RL!\nOur goal of creating the strongest *Melee* AI in the world is well within reach.\n\nThank you for reading!\n\n## Acknowledgements\n\nThank you to Jeffrey Cheng, Lev Stambler, and Sung Jae Bae for feedback on this article.\n\nA big thanks to Andrew Hinh and the [Modal](https://modal.com) team for sponsoring the compute that made these experiments possible!\n\n## Appendix\n\nI share below both ideas that worked and ideas that I tried but couldn’t get to work. I didn’t always have time to prove why certain ideas performed better or worse, or that failed ideas hadn’t failed because of an implementation bug. Even so, I hope that simply reporting what I tried and found is interesting anyway.\n\n### Inputs and outputs\n\nLast blog post, I made a few simplifying assumptions to the action space by discretizing analog controls and reducing concurrent button presses to the latest button. This can cause a lot of error accumulation in closed-loop.\n\nFirst, I revisited the way we map controller groups (main stick, c-stick, triggers, buttons) to target labels for the model.\nI included the `L` and `R` digital shoulder buttons and represented all possible combinations of 8 buttons (`A, B, X, Y, Z, L-digital, R-digital, D-pad right`) as a one-hot with  classes instead of only  classes.\nI also added more joint `(x, y)` locations to the main stick discretization, which previously hurt performance but performs better as we scale up capacity ([Figure 12](#fig-12), [Video 5](#vid-5)).\n\nTogether, these changes give a nice 0.29 NSM bump.\nEncoding the main stick as polar coordinates provided marginal improvement (within noise), and **flow matching directly on continuous analog values was worse** (see [Flow matching](#flow-matching)).\nFrom this, I concluded main stick discretization was no longer a bottleneck.\n\nAll of the game state and player features for a single frame are concatenated into one input token as before.\nThis time, I also added projectiles and Ice Climbers’ Nana<sup>[17](#user-content-fn-nana)</sup>, with masking bits on frames when they don’t occur.\nBoth appear in only a fraction of matchups, so they noticeably help there but otherwise don’t budge the bottom line much.\n\nFor past actions, adding a tiny learned embedding and a linear projection of the raw `(x, y)` values for each controller group, then concatenating them with the state features, was noticeably helpful.\n\n### Inside each frame\n\nGoing back inside each frame, I made further improvements to how we predict each controller group given we have information on past-predicted frames in the action chunk.\n\n1. We can use AdaRMSNorm to strongly modulate past-action conditioning on subsequent heads. This works much better than the feature concatenation that I used in the previous blog post. Curiously, I found zero-init FiLM to work worse than ordinary Kaiming uniform init for this.\n2. We can add a trunk skip: mapping straight into each controller group head’s logits. Though the decoder’s tokens already have , I hypothesize this offloads a lot of work and feature bandwidth to the trunk, which can heavily pattern match controller actions simply based on the game state history, while turning the tiny, latency-sensitive decoder heads into small residual writers that only have to edit the priors at each time offset based on locally sampled actions.\n3. Finally, using a SiLU MLP head over a linear head helps a lot.\n\nEssentially, at every turn, adding more expressivity and capacity to the decoder was disproportionately helpful to closed-loop performance.\n\n### Other ideas that didn’t work\n\nI tried adding forms of cross-attention between the action expert’s tokens (future actions) and the trunk’s (history tokens). It didn’t help much (0.07 NSM), but I kept a single cross-attention layer. It seems that a single latent mostly contains enough bandwidth to predict a chunk of future actions.\n\nHand-engineered features like distance to opponent, distance to nearest ledge, hitlag left, etc. made model performance worse.\n\nButton presses and other controller actions tend to be correlated, but fairly discrete and sparse (mostly zeros).\nI was inspired to try [FAST](https://www.pi.website/research/fast), but since discrete cosine transform struggles with sparse binary signals, I tried [run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding) actions over a 16-frame horizon before tokenizing them.\nA button sequence like `AAANNNNNNXXXNNBB` would turn into `(A, 3), (None, 6), (X, 3), (None, 2), (B, 2)`.\nThis didn’t pan out.\nThe model was fairly non-reactive and would hold buttons for far too long.\nPerformance hovered around −2 NSM.\n\nI also tried “chording”, or jointly representing the complete controller state for one frame in a single vocabulary. Since , I scanned the training data and built a vocab of roughly 8000 classes that covered 99.9% of frames. Unfortunately, the model didn’t converge to coherent play.\n\nI tried flattening the hierarchical autoregression by putting all of the controller actions across frames into one long sequence of tokens with the state. Basically, training a model to learn and so on. This increased sequence lengths roughly 5-fold for a 0.1 NSM gain, or at equal training budget, a −0.26 NSM change.\n\nI also tried training a “state expert” head with flow matching alongside the action expert head to predict next-frame features of the game and player states. It seems this somewhat fought for representation capacity and led to a −0.05 NSM change in performance—within noise. Even as I scaled up the model size up and discretized future state targets in the action chunk, this was strictly worse than devoting all parameters to policy alone.\n\nI also tried [Explorative Modeling (XM)](https://explorative-modeling.github.io/), which generates  candidates per training step and backpropagates loss for only the closest candidate.\nXM promises to teach the model to handle multimodal distributions by factoring the generation procedure, so I hoped that in lieu of using AR to factorize the action chunk, the XM model could generate all the controller parts at once.\nAt , we saw net −0.8 NSM.\n\n## Footnotes\n\n1. \n[2200 Elo](https://slippi.gg/user/hal-9000) within 5 hours. I do not condone running bots on Slippi without permission![↩](#user-content-fnref-bots_on_ladder)\n2. \n[Nair et al., 2018](https://arxiv.org/abs/1709.10089)[↩](#user-content-fnref-rl_exploration)\n3. \nAs opposed to open-loop evaluation or training, where the model’s loss is measured against a teacher (human) demonstration, *closed-loop* means the policy controls the game. Each action changes the next observation it receives, just like real deployment.[↩](#user-content-fnref-closedloop)\n4. \nSmall mistakes take a cloned policy to states the demonstrations never visited, where it makes bigger mistakes. [DAgger](https://arxiv.org/abs/1011.0686) (Ross et al., 2011) formalizes this as compounding error due to covariate shift.[↩](#user-content-fnref-dagger)\n5. \nOur behavior cloning policy from [part 1](https://ericyuegu.com/behavior-cloning-melee) was excellent at returning to stage and recovering from unlikely inputs.[↩](#user-content-fnref-bc_coverage)\n6. \n[FiLM](https://arxiv.org/abs/1709.07871) (Perez et al., 2018) conditions a network by predicting a per-feature scale and shift. Adaptive normalization, such as adaLN in[DiT](https://arxiv.org/abs/2212.09748) (Peebles and Xie, 2023), applies the same idea after a normalization layer.[↩](#user-content-fnref-film)\n7. \nPopularized by [ACT](https://arxiv.org/abs/2304.13705) (Zhao et al., 2023) and[Diffusion Policy](https://arxiv.org/abs/2303.04137) (Chi et al., 2023). There’s some countervailing[evidence](https://arxiv.org/abs/2608.02547) that action chunking works because it implicitly ensembles different policies with random delay.[↩](#user-content-fnref-chunking)\n8. \n [(Black et al., 2024)](https://arxiv.org/abs/2410.24164) trains a flow matching action expert on top of a pre-trained VLM backbone.[↩](#user-content-fnref-flow)\n9. \nTerm used interchangeably with “action output head.” [↩](#user-content-fnref-action_expert)\n10. \nClosely related to [discrete flow matching](https://arxiv.org/abs/2407.15595) (Gat et al., 2024), which defines flows directly over categorical variables.[↩](#user-content-fnref-discreteflow)\n11. \n, with . [↩](#user-content-fnref-endpoint_step)\n12. \n[Real-time chunking (Physical Intelligence, 2025)](https://www.pi.website/research/real_time_chunking)[↩](#user-content-fnref-rtc)\n13. \nI hoped that a mid-range consumer GPU from five years ago would be representative of the hardware most players have. [↩](#user-content-fnref-affordable_gpu)\n14. \nFor the scaling runs, I made my action chunks longer to cover inference delays for potentially very large models. [↩](#user-content-fnref-long_action_chunks)\n15. \n[DeepSeek LLM](https://arxiv.org/abs/2401.02954) (Bi et al., 2024) replaces parameter count with non-embedding FLOPS per token, , with , and finds the compute-optimal  and  from iso-FLOP curves. The first term is ; the second counts attention over the context. I leave the attention term out: over a 128-frame context it adds , about 7% for my smallest model and 2% for my largest.[Kaplan et al. (2020)](https://arxiv.org/abs/2001.08361) drop it for the same reason.[↩](#user-content-fnref-computesize)\n16. \nTrajectories within the same replay tend to be heavily correlated. Sampling 4 windows (instead of 1 window) from each replay within a few batches of each other reduced performance by 0.73 NSM. [↩](#user-content-fnref-multiple_windows)\n17. \nIce Climbers are two characters: Popo, controlled by the player, and Nana, a partner that mostly repeats Popo’s inputs a few frames later but can be [desynced](https://www.ssbwiki.com/Desyncing) in advanced play.[↩](#user-content-fnref-nana)\n\n## Cite this post\n\nGu, Eric. “Scaling pre-training on Melee.” *ericyuegu.com*, October 2026. [https://ericyuegu.com/hillclimbing-melee](https://ericyuegu.com/hillclimbing-melee)\n\n```\n@misc{gu2026melee,\n  author = {Gu, Eric},\n  title  = {Scaling pre-training on Melee},\n  year   = {2026},\n  month  = oct,\n  url    = {https://ericyuegu.com/hillclimbing-melee}\n}\n```\n\n", "url": "https://wpnews.pro/news/scaling-pre-training-on-super-smash-bros-melee", "canonical_source": "https://ericyuegu.com/hillclimbing-melee/", "published_at": "2026-10-16 19:19:48+00:00", "updated_at": "2026-10-16 19:50:16.907012+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "neural-networks", "ai-agents"], "entities": ["HAL", "Super Smash Bros. Melee", "Twitch", "Phillip", "slippi-ai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/scaling-pre-training-on-super-smash-bros-melee", "markdown": "https://wpnews.pro/news/scaling-pre-training-on-super-smash-bros-melee.md", "text": "https://wpnews.pro/news/scaling-pre-training-on-super-smash-bros-melee.txt", "jsonld": "https://wpnews.pro/news/scaling-pre-training-on-super-smash-bros-melee.jsonld"}}