Scaling Pre-Training on Super Smash Bros. Melee

HALPart 2 of 3

I trained a model named HAL to play Super Smash Bros. Melee using human data only and reached top ~1% of human play on ranked ladder¹. To my knowledge, HAL is the strongest base Melee model in the world.

Here’s a short highlight reel of HAL playing Fox against humans (names redacted with rank). Our model plays aggressively by chasing down opponents, finding surprising moves, but occasionally fumbling edge guard scenarios. I’ve included annotations for the non-Melee players.

You can find full VODs on Twitch.

Main takeaways #

This post is written with ML practitioners and technical readers in mind. The first blog post in this series covered our motivations, problem setup, and behavior cloning (BC) approach.

In this post, we…

• add offline reinforcement learning techniques like return conditioning, and find that requesting a constant 99th-percentile return on every frame gives stable play
• modify the architecture to support autoregressive action chunking , so the model only needs to replan at <10 Hz
• characterize pre-training “scaling laws” forMelee , and find that model FLOPs should scale faster than data
• show that observation delay hurts more than model size helps
• use a KV cache that isn’t numerically exact, and find it works anyway
• explore lots of other architecture, feature, and learning-objective ideas in the Appendix !

The final recipe is a 247M-parameter causal Transformer that reads the last 256 frames of game state and controller inputs (no pixels), then outputs a chunk of future controller inputs, decoded autoregressively frame by frame.

Because it predicts coherent multi-frame chunks, HAL only needs to replan every few frames, which hides both inference time and netplay delay, enabling it to play online in real-time on varying hardware and network conditions. We can also steer its play style and difficulty at test time.

Figure 1 shows the whole recipe, which we unpack section by section.

Why not reinforcement learning? #

There exists another Melee agent trained with behavior cloning + online RL named Phillip that is quite competitive with pros. If that’s the case, why not move on to online RL already?

• My intention was always to pre-train on human data to warm start self-play RL. However, I was confident that pre-training a stronger base model leads to a stronger post-trained model .
  • My previous post left a lot of low-hanging fruit in feature representations, model architecture, learning objective, and optimization that I wanted to revisit.
  • It’s well known that RL is inefficient at exploration. If I can learn better priors and preserve more diversity, this can improve results downstream² .
• I wanted to apply robotic learning techniques and find “scaling laws” in Melee. - In some ways, my problem setup is an idealized robotics environment. There is no deployment or embodiment gap between train and test, but I also have tons of diverse, expert human data.
• I wanted to make the model available for people to play against. To do that, I needed to find a principled way to handle network latency for players all around the world.

Over the last few months, I finally found the time and compute to chase these rabbit holes, the results of which I’m pleased to share. These directions were not necessarily the shortest path to superhuman Melee AI—at the end of the day, this was simply what piqued my curiosity.

How to read results #

Over the course of several months, I ran over 50 experiments and trained hundreds of models.

Except when stated otherwise, I tried to ablate ideas by training two models with equivalent compute, parameters, and data, then measuring differences in closed-loop eval³ over 96 episodes, 2 minutes each. I seeded a fixed set of stage and character match-ups that match the dataset’s prior distribution, then played each model against the in-game level 9 CPU, same as with our previous post. This was a cheap, reproducible baseline roughly on-par with the small-scale models.

A player wins the game if they take all 4 of their opponent’s stocks, or lives. Our primary metric is net stocks per minute (NSM):

All else equal, we prefer a winning policy to take stocks faster and a losing one to lose stocks slower.

NSM scales roughly linearly to Elo within , with +0.3 NSM ≈ +100 Elo. Outside of that range, NSM saturates asymptotically like Elo—our flagship model in the video above has only 1.7 NSM over the level 9 CPU but a 100% winrate over 192 games. At this sample size, NSM is accurate to small changes in behavior even across short episodes, which facilitates shorter evals.

We should be careful when comparing absolute NSM across experiments, because configurations in model size, learning objective, and training duration varied across experiments. Some models were more undertrained than others, while later experiments tended to stack the accumulated techniques of earlier findings.

Another flaw is that NSM is biased towards faster characters and smaller stages. To mitigate this, we can pair each treatment eval episode with a baseline episode of the same matchup, and call a result significant if the 95% confidence interval of a paired t-test over per-game differences excludes zero.

In general, I assumed that beneficial changes would have diminishing marginal returns but not destructively interfere.

Offline RL #

Behavior cloning has well-known failure modes⁴. But, given the size of our dataset relative to the state space, we actually have excellent coverage of failure states⁵.

Fundamentally, if our goal is for the model to be really good at winning Melee, imitating everything humans do isn’t always optimal. Even after filtering bad examples from our dataset, there are bound to be human mistakes and players who focus on style over winning. Can we make better use of the signals we have from the human dataset?

Advantage-Weighted Regression (AWR) is behavior cloning where each training example’s loss is weighted by its advantage —how much better or worse the return after that action was than a baseline.

The return is simply a discounted sum of rewards from a frame to the end of the replay.

Through experimentation, I defined a symmetric reward function between two players:

where is the percent damage a player takes on frame , is 1 on the frame they lose a stock, and is 1 if that was their last stock.

How did I decide on these reward coefficients? A typical stock is played to around 90-120% damage. With discounting, I wanted to properly incentivize the model to prefer taking stocks over dealing more damage (especially the last stock of the game), but in circumstances where that isn’t possible, to take favorable trades.

It was important to tune , which determines the discount rate or half-life of the reward. Too short, and the earlier frames that actually caused the outcome don’t receive attribution. Too long, and unrelated frames get affected, possibly even encouraging degenerate behavior like keeping an opponent alive to farm more damage. I found that a half-life of around 8 seconds was 0.37 NSM better than 3 seconds and 0.05 better than 14 seconds. Including damage in the reward function gave an extra 0.2 NSM—otherwise the signal was too sparse.

As for the baseline, I regress a value function head on the return. Similar states show up with many different continuations in the data, so learns their average (the return a typical human gets from here). The advantage asks whether this particular action did better or worse than usual from the same state, and its clipped exponential weights the action’s negative log-likelihood in the loss:

In Video 2, I color each character by its advantage, from purple (negative) through grey to orange (positive). The return sums rewards from the current frame forward, so the color reflects the next several seconds of play more than the character’s current position. Under the dense reward, Kirby turns purple while hitting Falco at the start of the clip, because Falco is about to punish Kirby. Kirby then turns orange while taking those hits, because Kirby goes on to win the game. Under the sparse reward, only the final stock counts, so both characters stay close to grey for most of the clip.

As a learning aid, I also created a visualization of how AWR re-weights the gradient in replays leading up to certain events, using real data and a heuristic for :

AWR improved my model by 0.29 NSM over behavior cloning.

Notably, AWR never assigns negative weight to any examples. Pushing the policy away from recorded actions could potentially learn worse or invalid actions. Instead, we can think of AWR as simply up-sampling and down-sampling training examples in proportion to how “good” the actions were.

I found AWR to be more principled than training only on player perspectives that won, since there is valuable training signal from imitating skilled players who lost.

#### What didn’t work #

While designing a reward function using damage, stocks, and winning worked, weighting training samples by skill ranking (i.e. weighting the gradient for a Master rank player that of a Platinum player) did not help.

I didn’t have much luck learning a good Q-function, but I didn’t try very hard.

I also tried IQL, which learns Q and V with TD learning and uses their difference as the advantage. Attaching Q and value heads to the policy scored −1.85 NSM, compared to +0.16 for plain behavior cloning, even with twice the data. Separate critic networks did about as badly (−1.55). The Q and V estimates drifted up past the clip ceiling and stayed there, so the advantages were basically noise. In hindsight, this made sense: function approximation, bootstrapping, and off-policy are known as the “deadly triad” of deep RL.

Fortunately, I could directly supervise on dense ground-truth rewards and had no need to bootstrap a Q-function with TD learning.

Return conditioning

In general, more conditioning is a great way to factorize multimodal distributions.

I added two inputs: which player to imitate and how much return it should go for. These double as levers for the model’s play style and “difficulty” at test time.

Return conditioning is inspired by Decision Transformer. Unlike AWR which reshapes the training distribution, return conditioning is a knob I can turn at test time without retraining.

At each frame, the label is the discounted return over the next 60 frames, with the same rewards and as AWR. The label only enters the action decoder, so the trunk and value head never see the future return (see implementation).

A problem I encountered was that I didn’t have a reliable learned distribution that I could sample the return-conditioning from. I had trained my value function on ground truth discounted return until the end of each replay, which has a pretty weak correlation to any arbitrary 1-second window during the game (Pearson r = 0.471). I didn’t want to take the effort to train a second value function just for this purpose, so at inference time, I request a constant return.

As it turns out, requesting a constant 99th percentile return on every frame results in stable play, +0.6 NSM over no conditioning.

Requested return, next 60 frames	NSM	Stocks taken / min	Stocks lost / min
No conditioning	0.80	1.42	0.62
90th percentile	1.20	2.01	0.81
99th percentile	1.41	2.00	0.59

90th percentile play is a lot more aggressive than no conditioning. In Video 3, 99th percentile play is more careful than 90th percentile without losing the aggression.

This is unsurprising, because in the data, 1-second returns fall into a starkly bimodal distribution: windows with only damage (<50 return), and 8% of windows that contain a stock event (>100 return). Requesting a 90th percentile return in these scenarios result in the model choosing suboptimal actions.

What’s interesting is that the model is able to generalize decision making associated with high return scenarios to other scenarios on stage at low percents (typically nowhere near a kill), as seen in the highlight reel.

#### Return conditioning implementation #

I figured 60 frames was a reasonable horizon because our action chunks are only half that length.

The label enters the action decoder as a FiLM⁶ layer on each block’s attention and MLP RMSNorms, shared across the whole action chunk. The projections are zero-initialized, so training starts from the unconditioned model. Unlike the past-action FiLM in Inside each frame, the return is an optional extra signal layered on a policy that already works, so starting from “no effect” is the right default.

Player conditioning

I had over half a million games from 38 pros with nearly 20k unique player codes, plus anonymized ranked games tagged by rank (Platinum, Diamond, or Master). I told the model who it was playing as using a learned embedding for player code or rank.

Here’s a 16M model playing as Fox against the level 9 CPU, conditioned on a couple players known for playing Fox:

Conditioned on	Net stock / min	Net damage / min
No one	0.55	55.9
Cody	0.63	87.0
Master	0.53	60.1
Mang0	0.48	62.9
Platinum	0.23	46.6

Platinum conditioning is clearly worse than Master or either of the pro players, which is a nice sanity check.

Across training checkpoints, player ID conditioning increased the Pearson r correlation between validation loss and net stocks/minute:

Evaluation suite	Player-conditioned BC	Unconditioned BC
Fox only	−0.84	−0.59
Multiple character matchups	−0.92	−0.59

Action chunking #

Up until now, we trained a model that makes only next-frame predictions. Even with history, human data tends to be multimodal and non-Markovian due to different play styles and reaction times. What if we also want to support larger models with slow inference or variable network delay?

We can predict actions for multiple steps ahead, also known as action chunking. In robotics, action chunking is thought to help because it shortens the prediction horizon, improves temporal consistency, and helps with representation learning⁷.

However, action chunking in robotics uses diffusion or flow matching models to predict continuous value trajectories⁸. We currently have an autoregressive (AR) architecture that predicts discrete actions (button presses). One option is to adapt flow matching to discrete actions, which I cover below.

A simple direction is to extend the next-token prediction objective into a multi-token prediction (MTP) objective. And this works pretty well! You can add independent output heads, each tasked with predicting actions at a fixed frame offset . They don’t need to be consecutive either: adds about 0.5 NSM over next-frame only prediction. I reckon this works because lookahead reinforces the importance of choice points in the loss, forcing the model to learn better representations to plan ahead.

But that predicts future actions independently. We care about the joint probability distribution of actions over time, not just the marginal distribution. Two marginals that are individually likely but jointly incoherent might half-commit to running away vs. attacking in the same action chunk (Figure 3a).

One way to improve MTP is presented in DeepSeek-V3, which feeds the th future token into the th MTP module. However, I have something better: another Transformer.

Each frame’s action has four controller groups: c-stick, main stick, triggers, and buttons. As in the previous post, HAL already predicts these one after another within a frame, which is important for coordinating inputs like ↓ + B on the same frame. Now we extend that across time: each frame in the chunk is also conditioned on every frame before it. Given the observed state history , a chunk of future actions factorizes as

where are the groups already predicted within frame .

Since we want pretty long action chunks (), rather than using a separate MTP module for each future offset, we can use a decoder-only Transformer as the action expert⁹ and causally attend to the future action chunk (Figure 3b). Compared to DeepSeek-V3’s MTP design, we save parameters and improve expressivity.

We don’t have new game state for these frames which have yet to happen, so our “input token” for the action expert is created by embedding the previous action at each time step and concatenating from the trunk, along with a position embedding. During training, teacher forcing supplies the ground-truth actions in parallel. At inference, those conditioning actions are instead the model’s own predictions .

Unlike VLAs, we don’t need to worry about memory-intensive modalities like vision, so we can supervise loss for an action chunk at every prefix in the training example in parallel, just like we did for next-token prediction (Figure 4). A decoder with dense supervision along the last 128 positions in its 256 context length trains >4 times faster and performs >1 NSM better than a model trained with loss only at its final token position, even though the losses at prefixes are correlated.

While decoding a long action chunk at inference can be slower than flow matching, we can choose not to decode the entire chunk—only what is needed to support the replan horizon (see Real-time inference).

To sanity check that AR action chunking actually helps, I trained four 7.1M models that differ only in the decoder: one that predicts every frame and every controller group independently, one AR across controller groups within the same frame, one AR across frames, and one with both.

We can compare the difference in performance when varying planning frequencies, with no added delay before each plan starts. When replanning every 4–6 frames, AR across frames is most beneficial, adding 0.4–1.0 NSM (Figure 5). Another way to look at this result is that AR prediction across frames holds roughly the same performance while replanning less often. Replanning only once every six frames gives a lot of flexibility around inference and network latency.

Qualitatively, Video 4 shows that autoregressive action chunking results in smoother, more coherent actions across frames at long replan intervals, with fewer erratic, jerky controller movements.

#### Flow matching #

One disadvantage of AR action chunking is that an early mistake locks you in. Unlike text domains, high frequency closed control problems like Melee are brutally unforgiving of errant actions.

While in principle both paradigms should be able to factorize the joint distribution, I initially expected flow matching in practice to learn more coherent action chunks because it has multiple chances to revise earlier actions while looking at later ones. Unfortunately, even the analog inputs (main-stick, c-stick, triggers) are heavily clustered around a few hot spots due to notches. Sparse and relatively discrete datasets are a poor setting for continuous flow matching.

I tried naively flow matching on raw (x, y) values for the analog groups (main stick, c-stick, triggers) and one-hot button vectors and this totally failed.

The only flow matching variant that I got working was endpoint flow over discretized actions¹⁰.

Endpoint flow

Each controller group at each of ten future offsets is a vector over its classes, linearly interpolated from Gaussian noise at to the one-hot target at . Instead of regressing a velocity with MSE, the model predicts logits for every slot’s class with cross-entropy, so estimates the clean chunk given . Each solver step moves every slot part of the remaining way toward that guess (Figure 6)¹¹.

Even with the chance to revise the whole chunk at every step, endpoint flow still didn’t beat AR decoding: −0.28 vs. −0.10 NSM at 7M, and +0.06 vs. +0.40 NSM at 15M, with the same trunk, data, and compute.

Real-time inference

For a near-lagless playing experience, Slippi netplay uses rollback netcode to process inputs client-side before resolving discrepancies when packets arrive from the opponent’s computer. It’s standard to play with 2 frames of action delay as network buffer. This means given an observation at time , if we press a controller button at , it won’t register in the game until .

We need to account for this delay in order to run the model accurately in real-time over the internet. Let’s define a few terms:

• Inference delay : frames the model takes to predict an action chunk.
• Network delay : frames a controller input sits in Slippi’s queue.
• Total observation delay
• Replan interval : frames of each action chunk played open-loop before the next chunk replaces it.
• Prediction horizon : length of each action chunk. A chunk has to cover the delay plus the open-loop interval, so .

Generally, we request the next action chunk after the previous model inference finishes, so . (We could technically overlap model inferences and pay the overhead, but as we’ve seen, short replanning intervals are not crucial to performance.)

Here’s an interactive visualization of these constraints:

At test-time, I use real-time chunking¹². The first actions of every chunk are already locked in by earlier chunks, so I force those actions in the action head and only sample actions after them. Given that I control the replan interval, I can stop decoding the action chunk at exactly , earlier than it was trained for.

The model works well with observation delay despite not explicitly being trained for it because it has observation history and predicts action chunks autoregressively.

To find the optimal prediction horizon, I played one fixed checkpoint against the level 9 CPU on a grid of observation delays and replan intervals (Figure 8). I found that each additional frame of reaction delay hurts performance twice as much as each additional frame of open-loop action execution.

Scaling #

Now that we have a working recipe, it’s time to scale up!

Bigger models are usually smarter, but they’re also slower. All else equal, they have to contend with more control delay.

I had two central questions:

1. Open-loop scaling : What is the compute-optimal model size for the amount of data that I have?
2. Latency-aware closed-loop scaling : What model size plays best on fixed inference hardware, once inference delay is taken into account?

To answer these questions, I ran two series of experiments:

1. Holding compute constant (iso-FLOP), I trained models with varying parameter and dataset sizes, similar to Chinchilla .
2. Holding data constant (iso-data), I trained models of varying sizes and evaluated them at the fastest replanning frequency their size would allow on my RTX 3060¹³ .

Compute-optimal model scaling

I ran compute-optimal scaling experiments and found that compute-optimal model FLOPS should grow faster than data.

The usual rule of thumb is that training costs FLOPS. That assumes each token passes through every weight once, but my models don’t work that way—each token passes through the trunk once but through the decoder 36 times, once for each future action prediction¹⁴.

Following DeepSeek¹⁵, I measure model scale as training FLOPS per supervised position instead:

I trained a grid of model sizes on a range of data budgets, then fit to their validation NLL. The fit tracks the measured runs closely, with an RMS error of 0.004 (Figure 9a).

Slicing that surface at a fixed training budget gives an iso-FLOP curve (Figure 9b). The minimum of each curve is the compute-optimal model size for that budget.

Following those minima across budgets, the optimal model size and data grow roughly as

These scaling terms differ from DeepSeek’s findings for LLMs () and are consistent with what others have reported in behavior cloning.

Why? Perhaps extra model capacity is important for modeling the multimodality and noise of human demonstration data. Another way to interpret it is that data in this domain (especially from the same episodes) is highly correlated, and counts less than a fresh text token does.

This exponent could also be an artifact of how I scaled the model parameters faster in the trunk than the decoder. For example, adding just a bit of width to the decoder but reducing the trunk params considerably so that the total model params dropped from 15M → 11M resulted in an equally strong policy. I did not run separate experiments to find the optimal decoder vs. trunk size.

My dataset is finite, so the more useful view is: for supervised positions, the compute-optimal model grows roughly as , so the budget grows as .

I made a tactical YOLO run with a model size that looks about larger than optimal in Figure 9c. Its effective data seen is higher than suggested: it trained on more data, supervised sparser action chunks, and propagated loss at fewer prefixes per sequence, so its looks deceptively large and surprisingly small. It folded in many of the architectural tricks and learning objectives above that improve sample efficiency but were not included in smaller scaling runs. I was also confident that with inference optimizations, the larger model could run as fast as the smaller models.

It does share one metric with every scaling run: the validation loss four frames ahead. On that metric (Figure 9d), it keeps improving well past the compute of any run in the fit due to the aforementioned tricks.

#### Validation loss vs. closed-loop performance #

Does lower loss actually mean better play? Broadly. Across the scaling runs, each played at a 4-frame delay, lower validation NLL goes with more net stocks (Spearman ρ = −0.66 over 28 runs; Figure 9e). It’s too coarse to pick between checkpoints across runs, though.

Within a single run (Figure 9f), validation loss sometimes stalls and NSM with it. Once the loss starts falling again, NSM climbs steadily.

This finding is roughly consistent with my experience in robotics.

Latency-aware scaling

Open-loop validation loss isn’t the only metric we care about.

For the second question, I evaluated each model at a delay that naive inference (no KV cache) takes on an RTX 3060 at batch 32. Here, both delay and replan interval increase with model size. The 6-frame model executes actions predicted 7–12 frames ahead, whereas the models in Figure 5 execute actions predicted 1–6 frames ahead with no added delay.

I trained each model size across 4 data budgets (Figure 10).

It’s clear that the smallest model is capacity-limited and doesn’t benefit from more data. An inverted U shape shows up as expected: the largest models lose more from slower reactivity than they gain from greater representational capacity. The loss-based fit didn’t predict this ordering because the cost of observation delay is high, and outweighs the benefit lower open-loop validation loss.

Inference optimizations #

If a smaller, more reactive model is better than a bigger, slower model, then I want my flagship model as close to 60 fps as I can.

Due to the complex architecture, the model has a lot of small elementwise operations and kernel launches.

We used CUDA graphs to capture and replay the compiled inference kernels with less dispatch overhead. This reduced batch 1 p95 latency on an RTX 3060 for the 247M model from 11.5 to 6.5 ms. That’s under one 16.7 ms frame, so the flagship gets a big model’s capacity at a small model’s delay.

KV cache

Melee has very few long-range dependencies: on average, the attention heads in my models focused on positions 60 frames in the past. Notably, I did not train with parallel sliding window attention on entire replays. During training, I prioritized replay diversity by sampling short windows of 256 frames to fit more distinct replays per batch¹⁶ using plain causal self-attention.

However, at inference time, this poses an issue. If I want to use a rolling KV cache to cut down on inference latency, the cache would retain state that contains information on tokens that have fallen outside the window (Figure 11). The numerically correct way to match training is to fully recompute attention values on the full context window every inference.

Fortunately, the DeepSeek-V4.1-Flash paper emboldened me to try dropping old tokens from the KV cache anyway. In practice, play doesn’t suffer: the rolling cache scored 1.006 NSM vs. 0.979 NSM for full recomputation. Across 96 eval matches, the sampled tokens between full recomputation and rolling KV cache disagree in 0.84% of frames. I suspect that the model’s closed-loop generalization compensates for these numerical differences.

KV caching reduced median latency by 42.3% at batch 64 on the RTX PRO 6000, with prediction horizon 8 and CUDA graphs on.

Mode	Median	p95
Full recomputation	44.62 ms	44.78 ms
Rolling KV cache	25.74 ms	25.85 ms

Results #

Our final run trained on a single B200 for 16 hours.

With return conditioning set to 99th percentile, player conditioning set to Cody, playing as Fox, replanning every 2 frames, our 247M-parameter flagship model scored 1.7 NSM against the in-game level 9 CPU. This same configuration (with 3 frames of observation delay) scored 2200 Elo on the human ladder.

Against Phillip

We also benchmark against Phillip, currently the strongest Melee model in the world, which has been trained on the same dataset and undergone extensive self-play RL (up to 1 week wall-clock time). Evaluation results are across all character matchups over 96-192 games:

Opponent	NSM	Win Rate
Level-9 CPU	1.215	97.4%
Our BC baseline (Feb 2025)	0.883	88.5%
Phillip (base)	1.050	94.8%
Phillip (RL)	-0.661	25.0%

The clips below come from the 96 games against Phillip (RL), including several where HAL loses the exchange. HAL plays the first character named in each caption.

Qualitatively, our larger model with AR action chunking and return-conditioning plays in an aggressive yet coherent style, with fewer errant inputs that put its characters into disadvantageous states. AWR made the models play more strategically, claiming center stage control and holding edge guards longer and more consistently.

Next up for us is self-play RL! Our goal of creating the strongest Melee AI in the world is well within reach.

Thank you for reading!

Acknowledgements #

Thank you to Jeffrey Cheng, Lev Stambler, and Sung Jae Bae for feedback on this article.

A big thanks to Andrew Hinh and the Modal team for sponsoring the compute that made these experiments possible!

Appendix #

I share below both ideas that worked and ideas that I tried but couldn’t get to work. I didn’t always have time to prove why certain ideas performed better or worse, or that failed ideas hadn’t failed because of an implementation bug. Even so, I hope that simply reporting what I tried and found is interesting anyway.

Inputs and outputs

Last blog post, I made a few simplifying assumptions to the action space by discretizing analog controls and reducing concurrent button presses to the latest button. This can cause a lot of error accumulation in closed-loop.

First, I revisited the way we map controller groups (main stick, c-stick, triggers, buttons) to target labels for the model. I included the L and R digital shoulder buttons and represented all possible combinations of 8 buttons (A, B, X, Y, Z, L-digital, R-digital, D-pad right) as a one-hot with classes instead of only classes. I also added more joint (x, y) locations to the main stick discretization, which previously hurt performance but performs better as we scale up capacity (Figure 12, Video 5).

Together, these changes give a nice 0.29 NSM bump. Encoding the main stick as polar coordinates provided marginal improvement (within noise), and flow matching directly on continuous analog values was worse (see Flow matching). From this, I concluded main stick discretization was no longer a bottleneck.

All of the game state and player features for a single frame are concatenated into one input token as before. This time, I also added projectiles and Ice Climbers’ Nana¹⁷, with masking bits on frames when they don’t occur. Both appear in only a fraction of matchups, so they noticeably help there but otherwise don’t budge the bottom line much.

For past actions, adding a tiny learned embedding and a linear projection of the raw (x, y) values for each controller group, then concatenating them with the state features, was noticeably helpful.

Inside each frame

Going back inside each frame, I made further improvements to how we predict each controller group given we have information on past-predicted frames in the action chunk.

1. We can use AdaRMSNorm to strongly modulate past-action conditioning on subsequent heads. This works much better than the feature concatenation that I used in the previous blog post. Curiously, I found zero-init FiLM to work worse than ordinary Kaiming uniform init for this.
2. We can add a trunk skip: mapping straight into each controller group head’s logits. Though the decoder’s tokens already have , I hypothesize this offloads a lot of work and feature bandwidth to the trunk, which can heavily pattern match controller actions simply based on the game state history, while turning the tiny, latency-sensitive decoder heads into small residual writers that only have to edit the priors at each time offset based on locally sampled actions.
3. Finally, using a SiLU MLP head over a linear head helps a lot.

Essentially, at every turn, adding more expressivity and capacity to the decoder was disproportionately helpful to closed-loop performance.

Other ideas that didn’t work

I tried adding forms of cross-attention between the action expert’s tokens (future actions) and the trunk’s (history tokens). It didn’t help much (0.07 NSM), but I kept a single cross-attention layer. It seems that a single latent mostly contains enough bandwidth to predict a chunk of future actions.

Hand-engineered features like distance to opponent, distance to nearest ledge, hitlag left, etc. made model performance worse.

Button presses and other controller actions tend to be correlated, but fairly discrete and sparse (mostly zeros). I was inspired to try FAST, but since discrete cosine transform struggles with sparse binary signals, I tried run-length encoding actions over a 16-frame horizon before tokenizing them. A button sequence like AAANNNNNNXXXNNBB would turn into (A, 3), (None, 6), (X, 3), (None, 2), (B, 2). This didn’t pan out. The model was fairly non-reactive and would hold buttons for far too long. Performance hovered around −2 NSM.

I also tried “chording”, or jointly representing the complete controller state for one frame in a single vocabulary. Since , I scanned the training data and built a vocab of roughly 8000 classes that covered 99.9% of frames. Unfortunately, the model didn’t converge to coherent play.

I tried flattening the hierarchical autoregression by putting all of the controller actions across frames into one long sequence of tokens with the state. Basically, training a model to learn and so on. This increased sequence lengths roughly 5-fold for a 0.1 NSM gain, or at equal training budget, a −0.26 NSM change.

I also tried training a “state expert” head with flow matching alongside the action expert head to predict next-frame features of the game and player states. It seems this somewhat fought for representation capacity and led to a −0.05 NSM change in performance—within noise. Even as I scaled up the model size up and discretized future state targets in the action chunk, this was strictly worse than devoting all parameters to policy alone.

I also tried Explorative Modeling (XM), which generates candidates per training step and backpropagates loss for only the closest candidate. XM promises to teach the model to handle multimodal distributions by factoring the generation procedure, so I hoped that in lieu of using AR to factorize the action chunk, the XM model could generate all the controller parts at once. At , we saw net −0.8 NSM.

Footnotes #

2200 Elo within 5 hours. I do not condone running bots on Slippi without permission 2. Nair et al., 2018↩ 3. As opposed to open-loop evaluation or training, where the model’s loss is measured against a teacher (human) demonstration, closed-loop means the policy controls the game. Each action changes the next observation it receives, just like real deployment.↩ 4. Small mistakes take a cloned policy to states the demonstrations never visited, where it makes bigger mistakes. DAgger (Ross et al., 2011) formalizes this as compounding error due to covariate shift.↩ 5. Our behavior cloning policy from part 1 was excellent at returning to stage and recovering from unlikely inputs.↩ 6. FiLM (Perez et al., 2018) conditions a network by predicting a per-feature scale and shift. Adaptive normalization, such as adaLN inDiT (Peebles and Xie, 2023), applies the same idea after a normalization layer.↩ 7. Popularized by ACT (Zhao et al., 2023) andDiffusion Policy (Chi et al., 2023). There’s some countervailingevidence that action chunking works because it implicitly ensembles different policies with random delay.↩ 8. (Black et al., 2024) trains a flow matching action expert on top of a pre-trained VLM backbone.↩ 9. Term used interchangeably with “action output head.” ↩ 10. Closely related to discrete flow matching (Gat et al., 2024), which defines flows directly over categorical variables.↩ 11. , with . ↩ 12. Real-time chunking (Physical Intelligence, 2025)↩ 13. I hoped that a mid-range consumer GPU from five years ago would be representative of the hardware most players have. ↩ 14. For the scaling runs, I made my action chunks longer to cover inference delays for potentially very large models. ↩ 15. DeepSeek LLM (Bi et al., 2024) replaces parameter count with non-embedding FLOPS per token, , with , and finds the compute-optimal and from iso-FLOP curves. The first term is ; the second counts attention over the context. I leave the attention term out: over a 128-frame context it adds , about 7% for my smallest model and 2% for my largest.Kaplan et al. (2020) drop it for the same reason.↩ 16. Trajectories within the same replay tend to be heavily correlated. Sampling 4 windows (instead of 1 window) from each replay within a few batches of each other reduced performance by 0.73 NSM. ↩ 17. Ice Climbers are two characters: Popo, controlled by the player, and Nana, a partner that mostly repeats Popo’s inputs a few frames later but can be desynced in advanced play.↩

Cite this post #

Gu, Eric. “Scaling pre-training on Melee.” ericyuegu.com, October 2026. https://ericyuegu.com/hillclimbing-melee

@misc{gu2026melee,
  author = {Gu, Eric},
  title  = {Scaling pre-training on Melee},
  year   = {2026},
  month  = oct,
  url    = {https://ericyuegu.com/hillclimbing-melee}
}

source & further reading

ericyuegu.com — original article