{"slug": "both-of-the-wins-were-arithmetic-i-had-never-done", "title": "Both of the wins were arithmetic I had never done", "summary": "A developer's chess engine search gained 62% more nodes at fixed time and about +80 Elo after batching neural network evaluations, reducing batch-1 forwards from 59% to 11% and raising mean batch size from 4.9 to 8.2. The author, flirp, reports that cyclic learning rates, weight averaging, and puzzle data each measured zero improvement, while the largest gain came from a simple batching fix. The project's model is 10.6MB with 5M parameters, and the author notes that batching is a small-model lever specifically.", "body_md": "08 Aug 2026\n\nby flirp\n\n# Both of the wins were arithmetic I had never done\n\n*Third in the series. Part one built a strong player out of a weak model and\na good search. Part two shrank the model 70x, put it on\nLichess, and concluded the sweet spot was near 5M parameters. This part makes the search 2.4x faster,\nwhich demolishes that conclusion in the good direction, then spends a night on cyclic learning rates,\nweight averaging and puzzle data, measures all three at zero, and wins the day's biggest result from the\ndullest thing available.*\n\nThis week cost me two numbers, and both of them are divisions I could have done in my head at any point in the previous month.\n\n**59%** is the fraction of neural network forward passes in my search that evaluated exactly one\nposition, on a GPU that will evaluate sixteen for the same money.\n\n**2%** is the fraction of one epoch that every fine-tune this project has ever run actually saw.\n\nBetween those two facts sit about a dozen ideas I was excited about. Cyclic learning-rate restarts. Weight averaging. Puzzle training. Beam retuning. fp8. A Rust rewrite. Every one of them measured zero or worse. It was a very productive week for long division.\n\n## The latency curve is the whole argument\n\nAt batch one, a forward pass on this model costs 4.4 milliseconds. At batch sixteen, a forward pass costs 4.4 milliseconds. Turns out there is free lunch.\n\n| batch | 1 | 4 | 8 | 16 | 32 | 64 | 128 |\n|---|---|---|---|---|---|---|---|\n| ms/forward | 4.9 | 4.4 | 4.4 | 4.4 |\n5.3 | 7.2 | 13.1 |\n| ms/row | 4.9 | 1.1 | 0.55 | 0.28 | 0.17 | 0.11 | 0.10 |\n\nFlat to sixteen. The model is 10.6MB, so at these sizes it spends its time on kernel launches and fixed overhead rather than arithmetic. Fifteen of those sixteen slots were being paid for and thrown away, six times out of ten. I had been paying for sixteen seats and buying one ticket at a time.\n\nFor contrast, the same bench on the 147M model from part one: 5.7 / 7.0 / 9.6 / 16.3 / 27.8 / 51.8.\nCompute-bound from batch eight. There are no free slots on a big model, so **batching is a small-model\nlever specifically**, which is a second reason to be small on top of the one part two found.\n\nI did not spot this myself. It arrived as a question. The search is *backloaded*, one pass at the root\nthen branching, so could the batches be spread out more evenly? The measurement that followed took\ntwenty minutes and produced the largest single gain in the project's history, which says something\nuncomfortable about the eight hours I had spent worrying about weight size.\n\n## The fix is to stop asking one question at a time\n\nAlpha-beta is depth-first and sequential by design. You need node A's value to decide whether B is worth\nsearching at all, which is what a cutoff *is*. So the search dribbles out evaluation requests one at a\ntime and the GPU sits there.\n\nBut at any given node, the search is about to evaluate the whole beam anyway. So evaluate the beam's children in one forward before descending into them, and do the same for the capture frontier in quiescence. Children that a later cutoff prunes cost nothing, because their slot was free.\n\n| before | after | |\n|---|---|---|\n| batch-1 forwards | 59% | 11% |\n| mean batch | 4.9 | 8.2 |\n| nodes at fixed time | +62% |\n|\n| mean depth | 2.56 | 2.87 (+12%) |\n\n**80 games against the identical model with the identical config: 42W 24L 14D, 61.3%, about +80 Elo.**\nSame weights, same moves considered, same clock. The only thing that changed is *when* positions get\nevaluated.\n\n## Then the bottleneck moved, four times\n\nEvery fix promoted the next constraint, and the obvious next target was wrong more often than right.\n\nBatch-1 forwards went first. Underneath sat **114,582 dtype-copy kernels**, about 24 per search node,\ncasting fp32 weights to bf16 on every forward. The weights never change. Casting once at load:\n**+12% nodes**.\n\nUnderneath *that* sat launch overhead: **76 CUDA kernels per node** at 2.7µs each, which is more CPU\ntime issuing work than the GPU spent doing it. CUDA graphs record a fixed shape and replay it, and since\nwe were now deliberately padding batches, rounding to buckets (4/8/16/32/64/128) made them replayable.\n**76 kernels per node became 3.4. +37% nodes, +17% depth.**\n\nUnderneath *that*, nothing. GPU execution was finally the constraint. Compounded, the search does\nroughly **2.4x the nodes** it did two days ago at the same clock.\n\nFor every idea in that chain, several died. Widening the beam looked free now that batching had made\nwidth cheap, and the bench agreed enthusiastically at +119% nodes: **−64 Elo over 60 games**, because\nthe policy's recall@20 is already 97.1% and moves ranked past twenty mostly dilute move ordering.\nNarrowing the beam to spend the nodes on depth bought +0.5 ply and **−58 Elo**, so the tuned value of 20\nsurvived attacks from both sides and I have stopped poking it. Deeper quiescence won a five-way screen\nat +104 Elo over 24 games and was **+12** at 84. Speculative padding did nothing at all, because I gated\nit at `depth > 2`\n\nand the mean depth is 3.4. Top-K policy readback did nothing either: the cost was\nper-call synchronisation, not the 1858 floats I was shipping across to read twenty, a thing I\nmisdiagnosed twice in a row. And the Rust rewrite to escape .NET's garbage collector died after I\nmeasured GC pause time at **0.06% of search time**, making it the fastest Rust project I have ever\ncompleted.\n\n## Why fp8 did nothing, two parts late\n\nPart two reported, with a shrug where the explanation should have been, that fp8 quantisation gained\nnothing. Here is the explanation. 5.29M parameters in bf16 is 10.6MB, which streams from memory in\n**16 microseconds**, or 0.36% of a 4.4ms forward. Halving that saves 0.18%.\n\nA 9B model is 18GB of weights and streams in about 27ms, six times its fixed overhead, so it really is\nmemory-bound and quantisation really is the dominant lever. **The crossover on this GPU is around 1.5B\nparameters, and we are 280x below it.** The entire published playbook for LLM inference, quantize,\nstream fewer bytes, compress the KV cache, is aimed at the other side of a threshold we are nowhere\nnear. Below it the lever is *operations issued*, not *bytes moved*. I had been reading the manual for\nsomebody else's machine.\n\n## Part two's conclusion did not survive\n\nPart two established that at equal time the ordering was 147M < 10.4M < 5.29M. Smaller won, because shrinking bought search depth, and the floor sat near 5M where the forward became all fixed overhead. That floor is a function of per-forward cost. We just cut per-forward cost by a lot.\n\nRe-run at equal time, 40 games each, same recipe at every size:\n\n| comparison | score | Elo |\n|---|---|---|\n| 10.4M vs 5.29M | 65.0% | +108 |\n| 20M vs 5.29M | 65.0% | +108 |\n| 20M vs 10.4M | 51.2% | +9 |\n| 5.29M vs 147M | 56.2% | +43 |\n| 10.4M vs 147M | 53.8% | +26 |\n\nThe ordering is now **20M ≈ 10.4M > 5.29M > 147M**. The optimum moved up at least two size classes,\npurely from making the forward cheaper. The mechanism is unchanged, bigger still costs depth, but bigger\ncosts *less* depth than it did.\n\nI am not claiming more than that, because three of those five comparisons are mutually inconsistent.\nChaining 10.4M−5.29M (+108) with 5.29M−147M (+43) predicts +151 for 10.4M−147M, and the measured value\nis +26. A second chain misses by 80. Forty-game matches cannot resolve differences of this size, so the\nhonest statement is a **plateau from about 10M to 20M**, not a peak. I did briefly fit a parabola\nthrough three points and announce a maximum at 25M. Three points determine a parabola exactly, so the\nmaximum was a property of my choice of curve and nothing else.\n\n## Every Elo number in part two was inflated, again\n\nPart two's Elo figures came from a Stockfish ladder. That ladder gave Stockfish `--movetime 50`\n\nand let\nmy model search at fixed depth with **no time limit at all**. Both biases push the same way: a\ntime-starved opponent playing well below its nominal `UCI_Elo`\n\n, against a model thinking as long as it\nliked. I had scored my engine against a Stockfish that was being timed with an egg timer.\n\nFixed, at equal 1000ms per move for both sides, the ladder finally triangulates:\n\n| opponent | score | implied |\n|---|---|---|\n| UCI_Elo 2250 | 78% | ≈2465 |\n| UCI_Elo 2500 | 68% | ≈2627 |\n| UCI_Elo 2750 | 17% | ≈2481 |\n\nThree levels agreeing within 160 points across a 500-point span, which is what a calibrated ladder looks\nlike. So: **about 2500 on Stockfish's scale**, and that scale is not CCRL's or Lichess's.\n\nThe number I actually trust is hardware-independent and calibration-free. Against full-strength\nStockfish on a node budget: **89% at 1,000 nodes per move, 21% at 10,000, 4% at 100,000.** One second of\nmy GPU is worth roughly three to five thousand Stockfish nodes, which is, at Stockfish's throughput,\nabout three to five *milliseconds* of one CPU core. A second of laptop GPU, flat out, trades evenly with\nabout four milliseconds of Stockfish. The project was never about beating Stockfish.\n\n## The fine-tunes were sampling 2% of an epoch\n\nWith the search finally spending its time on arithmetic instead of overhead, the bottleneck moved one more time, out of the engine and into the weights. The second division was waiting there.\n\nThe fine-tune recipe had been stable for weeks: mine the model's own mistakes, mix them into the corpus, train 2000 steps at batch 512. It kept producing small gains, so I kept running it.\n\nThen I did the arithmetic I had never done. 2000 steps × 512 samples = **1.02M positions**, against a\ncorpus of **44 million**. Every fine-tune this project has ever run saw about **2% of one epoch**.\n\nIt gets better. While checking that, I found an error in my own notes. A comment in the DAgger script\nwarned that a previous run had oversampled its mined rows \"~465x\" and overfit. The real figure was ~58x.\nI had multiplied by the gradient-accumulation factor when `--batch`\n\nwas already the *effective* batch.\nEight times off, in a comment whose entire job was to stop future me from getting this wrong.\n\nSo the plan for the night wrote itself: run 100k steps, which is 51.2M samples, or **1.07 epochs**, the\nfirst time the model would see its whole corpus once with the mined data folded in.\n\n## First, the model needed something to learn from\n\nDAgger needs games. The harness generates them by playing the model against itself from an opening book,\nand there I hit something embarrassing: **the match command played at temperature 0**, always. Two\nidentical checkpoints from the same opening produce the same game, move for move. My \"3000 self-play\ngames\" would have been 800 distinct games and 2200 photocopies, and I would have reported the 3000 in a\ntable with a straight face.\n\nOne flag later, sampling the policy instead of taking the argmax, seeded per game so resumes stay reproducible, and 3000 games meant 3000 games. Two temperature streams: 0.5 for play close to what the deployed argmax would actually do, 0.8 for coverage.\n\nThen mine them two ways:\n\n| pass | what it keeps | yield |\n|---|---|---|\n| blunder | moves that threw away ≥150cp | 1,017 rows (0.7% of moves) |\n| recall | positions where Stockfish's best move is outside the policy's top-4 | 19,832 rows (17.6%) |\n\nThe second is the number that keeps this project honest. **One model move in six proposes a candidate\nset that does not contain the best move.** Search cannot fix that. A move that never enters the tree is\nnever searched at any depth. Everything in the first half of this post raised the ceiling on searching\nwell, and that 17.6% *is* the ceiling. It is also why mining recall failures outproduces mining blunders\nby twenty to one.\n\n## Puzzles: a superb evaluation, a terrible teacher\n\nSelf-play DAgger has a structural blind spot. It only ever labels positions the model *steers itself\ninto*, so a weakness it habitually avoids never enters the training set. Lichess publishes 6.1 million\npuzzles with ratings and themes attached: free ground truth, positions chosen by somebody other than me.\n\nAs an evaluation it was excellent from the first run. Accuracy fell monotonically with puzzle rating,\n100% at 400 and 10% at 2400+, which is the sanity check that the position and move alignment is right,\nand it gave a per-theme breakdown no aggregate metric could. `veryLong`\n\n34.4%, `sacrifice`\n\n37.1%,\n`quietMove`\n\n41.2%, `defensiveMove`\n\n46.2%.\n\nThen I trained on the failures. 32,224 of them, labeled with Stockfish, mixed at a third of the mined pool.\n\n| puzzles (held-out 20k) | games vs the model it came from | |\n|---|---|---|\n| before | 68.5% | |\n| after | 75.7% |\n−20 Elo (LLR −3.54) |\n\nPuzzle accuracy up **7.2 points**, first-move accuracy 78.3% → 85.2%, and the thing plays measurably\nworse chess. **A puzzle announces that a tactic exists.** Train on 32k of them and the policy learns to expect one everywhere, so in quiet\npositions it starts hunting for a brilliancy that is not there, like a detective who has decided every\ndeath is a murder. The theme table shows it. `sacrifice`\n\nand `mateIn4`\n\nimproved sharply while\n`quietMove`\n\nmoved 41.2% → 42.0% and stayed near the bottom.\n\nAcross three checkpoints that night, puzzle score was *anti-correlated* with playing strength. It is now\nan eval and nothing else.\n\n## The boring thing worked\n\nThe overnight run: 100k steps from the DAgger checkpoint, both mined rounds mirrored (139,432 rows) plus\n10k puzzle rows kept deliberately small, mixed at **0.08**, which is about 29 repeats per mined row and\nwell under the ~58 that had overfit before. Fourteen hours, no crashes, once I had stopped running two\nGPU jobs at once on a laptop that responds to that by switching itself off mid-sentence.\n\n**480 games pooled across two clean opening suites: 58.2%, +58 Elo, SPRT LLR +5.58, H1 accepted.**\n\nThat is the only decisive training result of the night, and it came from doing nothing more imaginative\nthan letting the model read the whole book once. Combined with the DAgger round before it (+27 Elo over\n720 games), roughly **+85 Elo in a day**, of which the clever half contributed the smaller share.\n\n## The clever half, measured\n\nI ran the 100k steps as four cycles with a cosine cooldown each, on the theory that restarting the learning rate would knock the model out of wherever it had settled and let it re-converge somewhere better. Snapshot Ensembles, SGDR, SWA: a well-studied family. Each cooled cycle is a finished, playable checkpoint, so it also promised four models for the price of one.\n\nEverything about it measured zero:\n\n| comparison | steps added | Elo | verdict |\n|---|---|---|---|\n| 5 × 4k fast cycles | 8k | +5 | null |\n| SWA soup of three cooled cycles | 0 | +1 | null |\n| 20k more steps, one long cooldown | 20k | +3 | null |\n\nThree different shapes of \"more\", all indistinguishable from the checkpoint they started from.\n\nThe explanation was one small tool away. I wrote a command to report the distance between two checkpoints in weight space:\n\n| pair | relative L2 | cosine |\n|---|---|---|\n| cycle1 ↔ cycle2 | 0.30% | 0.999998 |\n| cycle2 ↔ cycle3 | 0.28% | 0.999999 |\nv16 ↔ cycle3 (+58 Elo apart) |\n0.56% |\n0.999988 |\n\n**The restarts never move the model anywhere.** There is no basin to escape and no diversity for an\nensemble to average. I had told myself two stories, first the textbook one about escaping local minima,\nthen a more sophisticated one about sampling diverse points on a connected low-loss manifold, and the\nmeasurement refuted both, because both require the model to actually *go* somewhere.\n\nIt also explains the soup. Weight averaging works when snapshots **orbit** a basin: the mean lands in\nthe middle, at a flatter point than any of them. Mine do not orbit, they **march**. cycle1 → cycle2 →\ncycle3 is a slow drift in one direction, so their average is a point *behind* the leading edge, and an\naverage of a monotone trajectory cannot beat its endpoint.\n\nThe game records had been saying this all along, if I had looked. Genuinely different models drew 30.6%\nof their games. The cycle snapshots drew **41%** against each other.\n\n## What a clean measurement costs\n\nTwo methodology corrections, both from being caught out.\n\nThe first: I built a fresh opening suite to test on, and it came from the same file used to *generate*\nthe self-play games the model trained on. Testing a model on the openings whose continuations it had\nmemorised. Caught before it reached a conclusion, but only just.\n\nThe second is worse, because it makes every small result in this project suspect. The same two models, 240 games on each of two clean, balanced suites:\n\n| suite | score | Elo |\n|---|---|---|\n| holdout-bal120 | 47.9% | −14 |\n| openings-120 | 53.5% | +25 |\n| pooled 480 | 50.7% | +5 |\n\n**A 39-Elo swing from opening selection alone.** A single 240-game suite is worth roughly ±25 Elo of\nnoise, which is larger than most of the effects this project has ever chased. Everything now pools both\nsuites at a minimum of 480 games, and the +58 result is trustworthy partly because both suites agreed on\nit (+76 and +39).\n\nThat result also nearly went the other way for a stupid reason. I had declared the fast-cycle arm\n\"destructive\" on the strength of validation agreement dropping 3.1 points, and was about to bin it.\nPlaying it out gave +5 Elo, which is no difference at all. Validation mispredicted the winner **three\ntimes in one night**: the DAgger model that won +27 while losing on every metric, the puzzle model that\ngained 7.2 points of puzzle score and lost 20 Elo, and this one. At some point I have to stop calling\nthat a surprise.\n\n## Where this leaves it\n\nv17 is live on Lichess, the search does 2.4x the nodes it did at the start of the week, and the model is roughly 85 Elo stronger than the checkpoint it started from.\n\nThe corpus, though, is spent. Going from 50k to 75k steps was worth about 11 Elo. From 75k to 95k, +3. More passes over the same 44M positions have stopped paying, and neither cycling nor averaging nor a longer cooldown recovers it. The recall miss rate says the same thing from the other end: two full rounds of DAgger moved it from 17.6% to about 16%. The mistakes I am mining are not the ones that matter any more.\n\nSo the next lever is new information, not more optimisation of the old. The deployed bot logs every move\nits *search* played, along with the backed-up value, and search play is a different distribution from\nthe raw-policy self-play I have been mining for two rounds. Those are the positions deployment actually\nfaces.\n\n## The lessons, updated\n\nParts one and two still stand. This part adds seven:\n\n**Measure the shape of your workload before optimising it.** The same codebase is launch-bound at inference and compute-bound in training. Doubling GPU power moved training throughput 6%. Batching moved inference 140%. There is no way to know which regime you are in without a profiler, and the published optimisation playbook is written for the other one. Below ~1.5B parameters, weight streaming is a rounding error and quantization does nothing. What costs you is operations issued.**Fix one bottleneck and you have promoted the next.** Batch-1 forwards, then dtype copies, then kernel launches, then the GPU itself. Four constraints in one evening, each invisible until the one in front of it was gone. Expect the list not to end, and expect the next item to be somewhere you have already looked and dismissed.**A benchmark that measures nodes measures nodes.** Beam widening doubled node count and lost 64 Elo. Beam narrowing bought half a ply and lost 58. Validation mispredicted the winner three times in one night. Games decided every question this project has ever asked, and I keep asking the other things first because they answer faster.**Compute what fraction of an epoch you are training on.** Every fine-tune in this project ran 2000 steps against a 44M corpus, 2% of one pass, and I never checked. One full epoch was worth +58 Elo, more than every architectural and scheduling idea of the past month combined. The arithmetic takes ten seconds, and instead I wrote an eight-times-wrong version of it into a warning comment.**If your intervention is supposed to move the model, measure whether it moved the model.** Cyclic restarts, by the story I told myself, relocate the weights so they can re-converge somewhere better. They move them 0.3% at cosine 0.999998. And weight averaging needs snapshots that**orbit**, not snapshots that** march**: sequential cooldowns along one trajectory give you a line, and the mean of a line sits behind its endpoint. One 40-line diagnostic settled in a minute what three A/B runs had only hinted at.**A benchmark that announces the answer teaches the wrong lesson.** Puzzles come pre-labeled with \"there is a tactic here\". Training on them bought 7.2 points of puzzle accuracy and cost 20 Elo, because the model learned the announcement rather than the tactic. Superb as an eval, exactly because it measures something self-play cannot. Poison as training data, for the same reason.**One opening suite is one sample.** The same pair of models scored −14 Elo on one balanced suite and +25 on another. Anything below ~480 pooled games across independent suites is a coin flip wearing a decimal point, and half the small results in this series were probably read too confidently.\n\nCode and run logs: [github.com/Oli-26/ChessLM](https://github.com/Oli-26/ChessLM). Games:\n[lichess.org/@/latentheatlm](https://lichess.org/@/latentheatlm/all)\n\n### Get the next one\n\nNew experiments, negative results included. No schedule, no spam, unsubscribe by replying.\n\n### Comments\n\nFound a hole in this? Say so. Corrections and replications are the whole point, and a comment pointing at a mistake is worth more to me than a compliment.", "url": "https://wpnews.pro/news/both-of-the-wins-were-arithmetic-i-had-never-done", "canonical_source": "https://latentheat.dev/blog/chess-one-boring-epoch", "published_at": "2026-08-07 23:00:00+00:00", "updated_at": "2026-08-09 12:37:03.182946+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["flirp", "Lichess"], "alternates": {"html": "https://wpnews.pro/news/both-of-the-wins-were-arithmetic-i-had-never-done", "markdown": "https://wpnews.pro/news/both-of-the-wins-were-arithmetic-i-had-never-done.md", "text": "https://wpnews.pro/news/both-of-the-wins-were-arithmetic-i-had-never-done.txt", "jsonld": "https://wpnews.pro/news/both-of-the-wins-were-arithmetic-i-had-never-done.jsonld"}}