I shrank my chess engine 70x by trying to compress it

A developer reduced a chess engine's size by 70x after retraining its architecture, boosting its Lichess rating from ~2294 to ~2588 Elo. The v2 model, with 122M parameters and 14 blocks, achieved 53.3% agreement with Stockfish and a value MSE of 0.0213, using the Lichess community evaluation database of 371M positions. The compression revealed the model was 70 times larger than needed, leading to a 70x size reduction.

03 Aug 2026 by flirp I shrank my chess engine 70x by trying to compress it Second in the series. Part one built a strong player out of a weak model and a good search, and warned you that what came later would supersede it. This is where that happens. The model gets an architecture worth searching with, then turns out to be 70 times larger than it needs to be, then goes on Lichess and plays like a maniac. The architecture that should have been there on day one Everything up to this point ran on a 39M-parameter ternary looped transformer, and three of those words were dead weight. The loops were flat from the first iteration. The ternary quantisation was a costume with no kernel to cash it in. And the 39M model had a zero train/val gap, which is the tell of a model too small for its data rather than one at risk of overfitting. So the plan was one retrain with five changes, each forced by a number. Ternary off, because there was no kernel on this path to collect the capacity tax. 39M to 122M parameters and 9 to 14 blocks, because a zero gap means capacity-limited, so buy some. Recurrence removed, because it was flat and dropping it ran 3x faster per step. And the standard modern attention stack on top: QK-norm, SwiGLU, geometric attention bias, plus a recall@4 hinge that pushes Stockfish's best move above the K-th logit, because cross-entropy never optimises the metric that actually gates search. | metric | v1 | v2 | |---|---|---| | val agreement | 49.4% | 53.3% | | value-mse | 0.0330 | 0.0213 | | vs SF-2250 | 56% ~2294 Elo | 88% ~2588 Elo | Roughly +294 Elo from a single retrain. It is the largest jump in the project, and the architecture change alone beat the entire data flywheel from part one. The exotic scaffolding came off and the model got dramatically stronger. That is the fifth time in this project that removing something clever has beaten adding something clever. One number set up the next problem: v2 finished with a 6.7-point train/val gap. It had flipped from capacity-limited to data-limited, which turned "get more data" into the priority. The data problem ends in thirteen minutes I had generated about 2.7M labelled positions with my own Stockfish over several days. Then someone pointed me at the Lichess community evaluation database: 371M positions, each analysed at depth 46 to 95, roughly four billion nodes apiece. That is on the order of 140,000 times the search behind my own labels, and every record carries five principal variations, which is my exact soft-target format plus the opponent replies a lookahead head would want. Converting the whole thing to my format took thirteen minutes. There is one trap in that data, and it is the kind that trains perfectly and plays terribly. Their centipawn score is White-relative. Mine is side-to-move. On black-to-move positions the sign is inverted, which I verified eight times out of eight against a local search. Negate when Black is to move, or half of every value target you train on quietly points the wrong way, and the model converges to a clean, confident, backwards evaluation. The value head is the weakness, and the metric lied again By now the question was not "is the model wrong" but "where, and which head". So I split the held-out positions by how much a deep evaluation disagrees with a shallow one, and scored the model on each half. The volatile half is where it falls apart. Value error more than doubles, from 0.113 to 0.270, while recall stays almost flat, and in the positions where shallow and deep search disagree on the best move, recall@4 collapses from 90% to 63%. The policy is distilled from shallow search, so wherever depth overturns the shallow read it is confidently on the wrong side. The weakness is tactical, it lives where depth matters most, and it is mostly the value head. That pointed at a knob nobody had touched: the policy-versus-value loss balance, set to 0.25 once and forgotten. So I ran three full retrains, identical except for that one weight, played at depth 5 against Stockfish limited to 2750: | value weight | result vs SF-2750 | approx Elo | |---|---|---| | 0.25 v4 | 54% | ~2779 | 0.50 v7 | 71% | ~2904 | | 1.00 v6 | 44% | ~2706 | v7 at 0.5 became the deployed model, and it had the worst validation scores of the three: the lowest agreement, 50.6% against 53.4 and 52.3, and the worst value-mse. It played the best chess by a wide margin. The metric did not merely fail to notice the improvement, it ranked the eventual winner dead last. Agreement measures a policy prior, and what you deploy is a searcher. A strong validation number is now a hypothesis I check, not a result I act on. v7 kept one visible flaw. Three of its draws were 300-move grinds where it reached a winning endgame and could not convert. That flaw later justified deleting most of the endgame from the training set. The compression detour A transformer's residual stream is a shared bus. Every layer reads and writes the same d-dimensional vector, and mine was 768 wide. The hypothesis was that a trained model arranges its features into that space loosely, and that a learned projection plus a little re-distillation could re-pack them into a narrower bus for free. I built it. Take the activations, PCA them, project the stream onto the top r directions at every layer, then heal with more distillation. The transformer trunk took a 6x squeeze for about four points of agreement. The convolutional input stem refused any linear code at all, which was interesting but not the point. The point was the control that kills the whole idea: train a narrow model from scratch at the same width, same recipe. If compression were exploiting loose packing, the squished-and-healed model should beat the from-scratch one. It lost. From-scratch scored 48.1% top-1, squished-and-healed scored 47.2%, and it had seen more total compute to get there. Width was not buying loose packing. A narrow model simply learns a narrow solution, directly, and slightly better. Compression was a longer road to a destination you can reach directly. Another clever idea lost, but it left a better question behind. Small models search deeper If a from-scratch narrow model matches a compressed one, the interesting question is not how far you can compress. It is how small the model can simply be. So I built real narrow models and started shrinking, from 147M down toward 2M, scoring each on held-out Stockfish agreement: | params | top-1 agreement | |---|---| | 147M | 51.6% | | 10.4M | 48.1% | | 5.3M | 46.1% | | 3.2M | 46.4% | | 2.1M | 44.6% | A gentle decline that stays nearly flat from 10M down to 3M, then starts to bend. By this metric, bigger is better and you keep the 147M model. This metric is the same trap as the value-weight sweep above, and this time I walked into it on purpose to show what is on the other side. The model does not pick moves in a vacuum. It is the policy-and-value prior inside an alpha-beta search, and what you ship is a player given a fixed number of seconds per move. A smaller model runs its forward pass faster, so in the same budget it searches deeper. So stop comparing accuracy and play games at equal time per move: | matchup, 1s per move | result | mean search depth | |---|---|---| | 10.4M vs 147M, 28 games | 10.4M wins 9-13-6 | 4.05 vs 3.2 | | 5.3M vs 10.4M, 30 games | 5.3M wins 14-7-9 | 4.67 vs 4.05 | The smaller model wins, both times. It gives up a couple of points of raw policy accuracy and buys back most of a ply of depth, and at these strengths depth is worth more than prior. Measured in games on a clock, 147M loses to 10.4M, which loses to 5.3M. And then it stops. Drop to 2.1M and the pattern breaks: against the 10.4M model at equal time it lost nine games out of ten. The reason is the same launch overhead from part one, now working against me. At these sizes the forward pass is mostly fixed kernel-launch cost, so the 2.1M model is barely faster than the 10.4M one, about 5.0ms versus 5.5ms per position, and it only searches 0.2 of a ply deeper instead of a full one. Meanwhile its policy is a few points worse and its value head noticeably so. A tiny depth edge cannot cover a real policy deficit. The floor is not an information-theoretic limit. It is the point where the model stops getting faster while it is still getting worse, and for this task on this hardware that point is around 5M parameters. One trick helped the small models specifically. Stockfish tablebases answer any position with five pieces or fewer perfectly, so I stopped training the small models on those positions at all. The model spends its shrinking capacity on the middlegame, which is the part no lookup covers and the part where it was weakest anyway. It also disposes of v7's 300-move endgame grinds: below six pieces, the tablebase plays, not the model. These brackets are about 30 games each, and 30 games resolves an Elo gap to roughly ±180. Any single one of these results sits inside the noise, and a rerun could flip the sign. What holds up is the direction, repeated as the model shrank by 28x, not any one scoreline. The next job is one bracket of 100-plus games on the best small model to turn the trend into an actual number. The trend is real and the error bar is the honest caveat on it, and part one already has a lesson about confusing the two. A footnote for anyone still reading from part one. Partway through this I found that a single forward pass on the small model was ninety percent kernel-launch overhead, and capturing it as a CUDA graph gave a beautiful 2.86x speedup. I got excited and wired it into the search. Then into training. Both delivered essentially nothing, because real workloads run batched and batching already hides the launches the microbenchmark was measuring. I wrote "measure what's broken, not what's slow" once and had to relearn it, from the other end, in the same month. The bot goes online and shows its character Held-out numbers are one thing. I put the 5.3M model on Lichess as a bot and watched it play real opponents on a real clock. Two things were obvious within an hour, and they are the same two the whole project keeps circling: it is dangerous in the middlegame and it is lost in the endgame. The middlegame: a tactical swindler. The model plays bold, materially reckless chess. In one game it gave up ten points of material and mated a 1763-rated bot. That sounds superhuman, and I nearly wrote it up that way, until I put Stockfish on the game at depth. The truth is better. The move that dropped the material was not a brilliancy. Stockfish says it turned a winning position +4.9 into a losing one -3.0 , and a stronger engine would have refuted it with a single check. The opponent did not find that check. It took the free rook instead and walked into a mating net. So it was a swindle, not a sacrifice, and that turns out to describe the whole style. Across a batch of its games only 2.9% of its moves are outright blunders by a depth-28 Stockfish, so it is not playing badly. But its edge is not soundness. Its edge is that it poses sharp, concrete problems fast, and opponents rated 1800 to 2000 cannot solve them in the time they have. A 3000-rated engine would punish the gambles. The players it actually meets cannot. At this level, aggressive-and-usually-sound beats slow-and-perfectly-sound. The endgame: it cannot finish. The same model that mates you from ten points down will then fail to win a position where it is up a queen. I watched it hold a forced mate in ten for a hundred and forty plies and draw by repetition. Stockfish confirmed the win was there the whole time. The bot simply could not find it, and the reason is mechanical. A mate in ten is twenty plies deep. In a seven-piece endgame the model was searching only three to five plies, because each position still costs a full network forward pass and the value head goes nearly flat once the board empties. Its value error is about four times worse in the endgame than the middlegame. A flat value head gives the search no gradient toward the mate, so it keeps the winning position and never converts it. The 50-move rule and threefold repetition finish the job. The fix is to search differently by phase. The middlegame and the endgame want opposite things. The middlegame wants a wide beam, because there are many plausible moves and the policy's top pick is right only about 80% of the time. The endgame wants a narrow beam and depth, because there are only a few sensible moves and the sole thing that finds a mate is looking further ahead. So the deployed search now counts the pieces on the board and switches: above ten pieces it searches wide, at ten or fewer it collapses the full-depth beam to the top three and spends the freed nodes on depth. Same engine, opposite settings, chosen by phase. The tablebase is the cleaner long-term answer, the same offload-to-a-lookup already used below five pieces, extended to six. But the phase-aware beam is free and it targets exactly what the games exposed. The model's honest Lichess bio: "Will sacrifice anything for an attack. Do not let me reach an endgame a piece up. I will find a way to draw it." Every Elo number above is wrong You have now read about four thousand words in which I quote ratings to four significant figures and lecture you about trusting metrics you have not checked. So it is worth telling you that I did not check mine. Here is how the ladder computes a rating, straight out of the harness: elo = level + 400 log10 score / 1 - score level is whatever I set Stockfish's UCI Elo to. That is the entire anchor. Every absolute number in this post, 2294 and 2588 and 2779 and 2904 and 2706, is a performance rating that assumes Stockfish really does play at its labelled strength. It got fifty milliseconds a move. UCI Elo is calibrated at sane time controls. Starve it to 50ms and it plays well below its label, so level is too high, so every rating built on it is inflated by an amount I cannot tell you, because I never measured it. Part one is worse. Most of those brackets ran at 30ms. There is a second problem, and I think it is the more embarrassing one. The model was not on a clock at all. Those runs set a fixed search depth of 5 and let Stockfish have its 50ms, so it is fixed-depth against fixed-time: two unrelated budgets pointed at each other and a rating printed at the end. It is not a time control. It is barely a match. And they were twelve games each. Earlier in this post I made a point of telling you that thirty games resolves an Elo gap to about ±180, then quoted 2904 off twelve games with no interval at all, in bold, in a table. So what actually survives? The comparisons do, which is most of the post. Every value-weight run used the same recipe against the same opponent, so 0.5 beating 0.25 beating 1.0 is a fair internal ranking even though the labels are fiction. The +294 Elo from the v1 to v2 retrain survives for a nicer reason: both were measured against the same 2250 setting, so level cancels in the subtraction and the gap does not depend on the anchor being right at all. And the result this post is actually about, small models searching deeper, never touched Stockfish. Those brackets are model against model at one second a side, so the calibration cannot reach them. What does not survive is the sentence a reader would most likely take away, which is that I built a 2900 Elo engine on a laptop. I built something that scores 71% against a crippled Stockfish under conditions I chose and did not validate. Those are different claims and only one of them is supported. The fix is a bracket at a real time control, on the clock for both sides, with enough games to put an interval on it. Until I run it, treat every four-digit number above as a project-internal score with a fancy costume on. Where this leaves things Every section here has been the Bitter Lesson in a different suit. The looped recurrence lost. The ternary weights lost. The compression scheme lost. Every time, the boring answer was standing behind the clever one: more data, cleaner targets, a wider search, and a smaller model that is allowed to look further. But the last result came with a second half the Bitter Lesson does not usually mention. "Smaller and deeper" is not free forever. It works while shrinking the model actually buys search depth, and it stops the moment the model is so small that its forward pass is all fixed overhead and cannot get any faster. Past that point the model keeps losing accuracy while gaining no depth, and a bigger model wins again. For this engine on this laptop that crossover sits near 5M parameters, roughly 30 times smaller than where the project started and about twice as strong per second of thinking. That floor is where two curves cross, not a wall. The lessons, updated Part one's list still stands. This part added four more: A benchmark can rank your winner last. v7 had the worst validation scores of its cohort and the best play. Score the system you ship, not the component you can measure cheaply. The microbenchmark lies at batch-1. A single-example speedup is a promise the batched workload never keeps. Measure the launch-overhead fraction of the real workload before you build the optimisation. Small plus search beats big, until it doesn't. At a fixed time budget a smaller model that searches a ply deeper beat a model 14 times its size. But the trick has a floor: once the model is small enough that the forward pass is all fixed overhead, shrinking buys no more depth while it keeps costing accuracy, and the bigger model wins again. Here that crossover was near 5M. Check the thing you are measuring against. I spent this whole post interrogating my validation metric and never once interrogated my opponent. A rating anchored on a number you assumed is not a measurement, it is that number with arithmetic done to it. Lesson 6 was sitting right there the entire time and I still managed to walk past it. Code and run logs: github.com/Oli-26/ChessLM https://github.com/Oli-26/ChessLM . Games: lichess.org/@/latentheatlm https://lichess.org/@/latentheatlm/all Get the next one New experiments, negative results included. No schedule, no spam, unsubscribe by replying. Comments Found a hole in this? Say so. Corrections and replications are the whole point, and a comment pointing at a mistake is worth more to me than a compliment.