{"slug": "learning-megagem-from-self-play-to-price-discovery", "title": "Learning MegaGem, from self-play to price discovery", "summary": "Jane Street's three-player auction game MegaGem was used to train a 4B specialist model, with SFT raising Qwen3-4B-Instruct's benchmark rating from 551 to 1158, ahead of Claude Sonnet 4.6, Claude Opus 4.8, and GPT-5.5, and distillation from an analytic expert further raising it to 1275, the highest rating in the benchmark and roughly equal with Gemini 3.1 Pro. The released weights are available on Hugging Face, and the repository is on GitHub. The game combines elements of Figgie, Splendor, and poker, and the research explores whether SFT, self-play, or distillation produces robust sequential decisions across opponents.", "body_md": "We study MegaGem, a three-player, general-sum, imperfect-information auction game developed by Jane Street, and train a 4B specialist to play it.\n\nSFT raises Qwen3-4B-Instruct’s benchmark rating from 551 to 1158, ahead of Claude Sonnet 4.6, Claude Opus 4.8, and GPT-5.5. Distilling an analytic expert raises it further to 1275, the highest rating in the benchmark and roughly equal with Gemini 3.1 Pro.\n\nOur rollout infrastructure runs multi-agent evaluation, self-play, and data generation. Although relative-outcome self-play yields no measured transferable improvement, the resulting self-play data support a market model and a distributional bidder whose decisions we distill into our final model.\n\nThe released weights are available on Hugging Face: the [SFT checkpoint](https://huggingface.co/djdumpling/qwen3-4b-instruct-megagem-sft-step1200-v2) and the final [distilled checkpoint](https://huggingface.co/djdumpling/qwen3-4b-instruct-megagem-distill-para). The repository can be found [here](https://github.com/djdumpling/MegagemBench).\n\nThis blog would not be possible without the ongoing support of many people. I’m especially grateful to [Johannes Hagemann](https://x.com/johannes_hage?lang=en), [Sebastian Müller](https://x.com/omouamoua), [Christian Reetz](https://x.com/creet_z), and [Peyton Walters](https://x.com/peywalt) for their generous provisioning of credits for compute and access to model APIs. Additionally, thank you to early readers for providing invaluable feedback, including Sebastian Müller, Varun Srivastava, Nan Jiang, Austin Feng, Stephen Xia, Aaron Yu, and Lauren Shen.\n\n*Reading note:* To keep the main argument moving, several derivations, diagnostics, and examples are collapsed by default. Feel free to read linearly or ignore collapsed sections :).\n\nThe rules in brief: three players each start with 35 coins and a secret hand of five gems. Each round, the top auction card is flipped and all three players submit sealed bids simultaneously; the highest bid wins, with ties broken by turn order. The winner takes the auction’s payoff, then reveals one gem from their hand to a public Value Display. A public Value Chart maps that display to everyone’s gem values. The final score is\n\n$$ \\text{final score}=\\text{coins left}+\\text{collection value}+\\text{mission rewards}-\\text{loan repayments}+\\text{investment returns} $$The game ends when no treasure gems remain to be auctioned, typically after ~15 rounds.\n\nAt its core, MegaGem combines three familiar strategic games. In [Figgie](https://www.figgie.com/), each player observes only part of the supply and must infer fair value from what others are willing to pay and what is revealed over time. In [Splendor](https://en.wikipedia.org/wiki/Splendor_(game)), resources matter in combinations: public objectives create races, and players balance improving their own position, preserving flexibility, and blocking an opponent. In **poker**, hidden holdings and public action histories invite opponent modeling, while position and stack sizes determine which late-game outcomes can still be forced. Furthermore, MegaGem puts all three inside repeated sealed-bid auctions, where every bid trades the chance to win the current lot against the value of keeping cash available.\n\nThat combination can make MegaGem useful as a benchmark, because it gives post-training and multi-agent researchers a compact way to study whether SFT, self-play, or distillation produces robust sequential decisions across opponents, while giving agent builders a test of state tracking and reliable structured action.\n\nTo understand the game better, let’s walk through six decisions from different points in a game.\n\nThe 30-gem deck contains six gems of each color, and the terminal Value Display is the 15 gems dealt into players’ hands. From your five-card hand and the two face-up auction gems, 23 gems remain unseen and the opponents’ ten hand cards are drawn from them. Let $N_R$ and $N_Y$ be the terminal display counts for Red and Yellow, and $V$ the corresponding value dictated by the value chart A, which pays 4 points per displayed gem up to 5. Then, the EV is\n\n$$ \\begin{aligned} \\mathbb E[N_R] &= 2 + 10\\left(\\frac{3}{23}\\right)=\\frac{76}{23} \\implies \\mathbb E[V_R] = 4\\mathbb E[N_R]=\\frac{304}{23}\\approx13.22\\\\ \\mathbb E[N_Y] &= 10\\left(\\frac{5}{23}\\right)=\\frac{50}{23} \\implies \\mathbb E[V_Y] = 4\\mathbb E[N_Y]=\\frac{200}{23}\\approx8.70\\\\ \\mathbb E[V_{\\mathrm{lot}}] &= \\mathbb E[V_R]+\\mathbb E[V_Y] =\\frac{504}{23}\\approx21.91 \\end{aligned} $$\n\nWinning the Blue–Green auction changes your collection to Red–Red–Blue–Blue–Green, completing two of the missions for 25 points total. Factoring in the blue and green gems from the lot and their values on the display, we know that winning this lot is worth at least $25+2\\cdot4+2\\cdot4=41$. Because neither Alice nor Bob has more than 12 coins and you are first in tiebreak order, you can win the auction for 12 and earn at least $41-12=29$ in positive EV. Their lack of liquidity lets you take the auction much more cheaply.\n\nBecause you are last in tiebreak order, matching Alice’s bid is a losing action: to clear a predicted bid of $b$, you must bid $b+1$. Alice can bid at most 12 and Bob at most 6, so 13 is the minimum bid to guarantee the lot. However, this increase must still be weighed against the gem’s value and the option value of keeping that coin for the remaining auctions; it may make the lot no longer worth bidding on.\n\nBoth cards enter the Value Display at game end, so revealing Red or Purple does not change final gem values, but **it changes what other players are willing to pay before then**. Revealing Purple moves its public count from 2 to 3 and its current quote from 8 to 12. Bob already owns Yellow gems, so another Purple would also complete the 5-point Pendant mission. This reveal raises his visible reservation value for a future Purple from roughly $8+5=13$ to $12+5=17$. That can be useful if the goal is to induce Bob to spend his six coins, but it also makes a strategically important Purple harder to buy or deny, especially when you have only two coins and act last in ties.\n\nRevealing Red has the opposite advantage. Moving Red’s quote from 4 to 8 has a null effect on public perception because there are no more red gems to be auctioned off (all 6 are either in a collection or will be in the value display). It also keeps Purple quoted at 8 while your hidden Purple guarantees that its terminal supply is higher than the table can see. This usually makes Purple easier to snipe later (for example, to block Bob’s pendant). Revealing Red does advertise more of your lead (making you a bigger threat to other players who want to block you), but its direct price impact is confined to a market that has already closed.\n\nThe 20 borrowed coins are matched by a 20-point repayment at scoring, so the principal itself creates no terminal value. Winning the loan for 10 changes your available cash to $C_{\\mathrm{after}}=2+20-10=12$, but carrying those coins to the end would leave you 10 points worse off. For example, if you win and have 14 coins, you can guarantee winning the last two auctions by first paying 7 (beating Bob in the tiebreak) and paying 7 again (since you just won, you are last in the tiebreak and need to one-up Bob’s liquidity of 6).\n\nOf the five cards still hidden, two are Purple, which puts a Purple gem’s expected value at $0.1(12)+0.6(16)+0.3(20)=16.8$.\n\nWith 2 coins, under rational play by your opponents, you would never be able to win the last two auctions. But for the reasons explained above, the loan gives you an out to win these last two, and because the EV suggests a positive trade, it is worth winning the loan auction.\n\nBefore the final lot, terminal-value accounting puts you at 77, Alice at 75, and Bob at 63. The Purple gem is worth 16 on Chart A; it would also complete Bob’s 5-point Pendant mission, but Bob has only 1 coin. The score branches are:\n\n$$ \\begin{aligned} S_{\\mathrm{Bob}}(1) &= 63+16+5-1=83\\\\ S_{\\mathrm{Alice}}(4) &= 75+16-4=87\\\\ S_{\\mathrm{You}}(5) &= 77+16-5=88 \\end{aligned} $$Your tie priority beats Bob, so a bid of 1 covers his branch. Alice, however, acts before you so 5 is the smallest bid that covers both stacks. Any bid from 6 through 12 buys the same gem while needlessly giving away score. The asset is fundamentally worth 16, but the correct price is 5 because the opponents’ remaining cash sets the clearing threshold.\n\nSeveral well-known RL systems use self-play: [AlphaGo](https://www.nature.com/articles/nature16961) in Go, [AlphaStar](https://www.nature.com/articles/s41586-019-1724-z) in StarCraft II, and [Pluribus](https://doi.org/10.1126/science.aay2400) in poker. The broad recipe is to train an agent against copies or historical versions of itself (known as fictitious self-play) and reward successful play. Opponents therefore co-evolve with the learner, creating an automatic curriculum without a fixed external teacher. Self-play also works outside two-player zero-sum (2p0s) games, but it’s cleanest in that setting. Unfortunately, MegaGem removes those conveniences one by one.\n\nAnother complication is that the policy to be trained is an LLM, not a policy head that emits one move. Consider AlphaGo again, which uses search with policy and value networks to select a single legal board action. Now contrast this with temperature sampling in LLMs that operates across an autoregressive response, so response entropy accumulates very quickly. Now a game-level reward must assign credit through a long natural-language trajectory, and many distinct responses can collapse to the same parsed bid.\n\nChess, Go, and heads-up poker are standard examples. One player’s gain is the other’s loss, so a value function $V(s)$ can describe the expected outcome from a position for either player, with the sign reversed for the opponent. This is the same object visualized by the evaluation bar beside an online chess board.\n\nUnder pure opposition, you must assume the opponent chooses whatever hurts you most. You therefore choose the strategy with the best worst case, the maximin. Your opponent does the converse, minimizing the maximum value you can obtain, the minimax. [Von Neumann’s minimax theorem](https://en.wikipedia.org/wiki/Minimax_theorem) says these values are equal.\n\nIf neither player can improve by deviating alone, the joint policy is in Nash equilibrium. In a 2p0s game, its equilibrium strategies are also minimax. This supplies a clean progress measure: exploitability, or how much a worst-case opponent could gain by deviating. A lower exploitability means the strategy is becoming harder for any opponent to exploit.\n\nOrdinary policy-gradient self-play can still cycle (think of rock-paper-scissors) in the latest policy. Practical systems therefore often average policies or train with fictitious self-play, exploiters, and wider populations, and these mechanisms can stop the fresh policy from forgetting old weaknesses.\n\nThose conveniences disappear in five places:\n\nGeneral-sum games remove the requirement that the players’ payoffs add to a constant in every outcome.\n\nIf you’re interested in the cooperative side of multi-agent RL, Arturo wrote a great [hands-on account of training LLM agents to play Hanabi](https://nphard.io/2026/02/23/hanabi.html), including the communication conventions that emerge under self-play.\n\nGeneral-sum rewards give each player a different value for the same position. Multiple equilibria may exist, so different conventions or tradeoffs are common.\n\nMegaGem adds more players. Score changes are not constrained to sum to zero, and an action can redistribute advantage among players in ways that barely affect the actor. A losing player can decide which leader wins. Two players can pressure the leader without sharing a formal team reward.\n\nA single optimal bid also breaks down because best play is contingent on opponents. Against a player who always bids 0, bidding 1 wins cheaply; against a rational player, the same bid loses. If a kingmaking action barely changes the acting player’s own reward, a policy-gradient update receives almost no direct signal for which rival benefits.\n\nWe can measure deviation gains around a specified joint policy. No unique opponent-independent strategy supplies a universal reference point. In a general-sum game, the word “improvement” must name an opponent distribution, equilibrium-selection rule, or population game. The self-play reward describes the population that produced it.\n\nFirst, deals are stochastic. A strong hand can raise terminal score without reflecting a better decision. Same-deal comparisons remove much of this variance.\n\nSecond, MegaGem has imperfect information. A player observes an information state consisting of its private hand and the public history; the hidden state remains unknown. Formally, the observation defines a set of hidden worlds still consistent with what the player has seen. New bids and reveals update the player’s belief over those worlds. A poker four-bet leaves the cards hidden while shifting the belief toward stronger hands.\n\nThis makes MegaGem a partially observable Markov decision process (POMDP) because the relevant value depends on the full belief state, not just the current board. The effective state space is therefore much larger and harder to estimate over.\n\nBecause bids are simultaneous, action selection also requires a distribution over clearing prices. For a candidate bid $b$, we need $P(\\text{win}\\mid b)$, the probability that the bid beats both opponents. Every bid trades the surplus retained after paying against the probability of winning.\n\nAlso, terminal gem values are fixed by the initial deal but revealed endogenously through play. Every private gem eventually enters the public Value Display, so the final display does not change. The order of reveals changes what each player believes before the remaining auctions. A reveal can therefore move perceived value even when it does not move terminal value.\n\nFor the common-value component of a lot, winning also carries information. If both opponents decline to pay your price, you may have overvalued the asset, the auction version of the winner’s curse. Private mission value weakens that inference because the same lot can rationally be worth different amounts to different seats.\n\nUsing Prime Intellect’s [verifiers](https://github.com/PrimeIntellect-ai/verifiers), we build the [RL environment](https://app.primeintellect.ai/dashboard/environments/djdumpling/megagem) around LLM clients playing one another. The implementation described here uses the legacy v0 API, but the newer, v1 API natively supports multi-agent RL environments such as this one!\n\nTo reduce context cost, a game runs as chained single-turn interactions. On every round, each player receives all information available to that seat, including the public state, their private hand, and the public history of the last five auctions (to maintain a sort of multi-turnness). The model returns a structured response and the environment parses the bid, resolves the winner, and prompts that player again if a gem reveal is required. Each game emits a trajectory containing the prompts, responses, parse and legality telemetry, bids, public state, final scores, and the seed needed to reconstruct the deal.\n\nThe environment emits complete game trajectories without defining the RL reward. A separate offline scorer reconstructs per-turn credit, and the update masks every opponent token so only the trainable seat receives gradients.\n\nExperimental design determines which triples play, on which deals, and in which seats. Estimation derives Elo-scaled Plackett-Luce ratings and confidence intervals. The benchmark uses three measurements that should not be conflated:\n\nRatings come from a balanced 624-game schedule fitted with a lightly regularized Plackett-Luce model, with 95% confidence intervals from bootstrapped deal clusters. To compare two models A and B directly, we simply determine how often A beats B in games where both appear, which removes more deck variance because it is a paired statistic.\n\nFor balanced rating estimation, we use a Steiner triple system, which places every pair of models in exactly one triplet. With 13 models, this produces 26 triplets, each playing 8 seeds with the models rotating through all 3 starting hands: 624 games, or 144 games per model. We then fit the resulting three-player rankings with a lightly regularized Plackett-Luce model.\n\nGiven a strict finish $i\\succ j\\succ k$ and positive rating $\\gamma$, the Plackett-Luce model assigns this ordering probability\n\n$$ P(i\\succ j\\succ k) =\\frac{\\gamma_i}{\\gamma_i+\\gamma_j+\\gamma_k}\\cdot \\frac{\\gamma_j}{\\gamma_j+\\gamma_k} $$[MM iteration](https://projecteuclid.org/journals/annals-of-statistics/volume-32/issue-1/MM-algorithms-for-generalized-Bradley-Terry-models/10.1214/aos/1079120141.full) fits a lightly regularized Plackett-Luce model. The model factors each final ranking into rank selections. In game $g$, let $Z_{g,r}$ be the summed strength of the models still eligible at ranking position $r$; for the ordering above, $Z_{g,1}=\\gamma_i+\\gamma_j+\\gamma_k$ and $Z_{g,2}=\\gamma_j+\\gamma_k$, the two denominators in $P$. We add 0.1 virtual wins and losses against a strength-1 anchor to stabilize finite samples. Let $W_i = 0.1 + \\#\\{\\text{games in which } i \\text{ does not finish last}\\}$. Initialize $\\gamma_i=1$ and repeat until convergence:\n\nFinally, take $\\theta_i = \\ln\\gamma_i$, center them on the field mean, and convert to the conventional Elo scale at $\\frac{400}{\\ln 10} \\approx 173.7$ points per logit. For example, a rating of 1275 is 275 points above the fitted center.\n\nTo quantify uncertainty, we bootstrap *matchup-on-deck clusters*, each of which contains the three seat rotations of one triple on one deal. Resampling whole clusters preserves the correlation introduced by a shared deck. We report 95% confidence intervals.\n\nThe observed cyclic component is indistinguishable from finite-sample noise. No statistically resolved triple forms a directed cycle, and the fitted rankings are stable.\n\nThe base Qwen3-4B-Instruct is sufficiently coherent to parse but strategically awful. It spends almost all its coins, completes few missions, and hallucinates a gem outside its private hand on roughly 2–5% of reveal turns.\n\nWe collect teacher games from a 70/30 mixture of Gemini 3 Flash and Claude Opus 4.6. Flash makes it affordable to collect broad coverage, while Opus contributes longer rationales.\n\n| teacher | full-field mean score | self-play mean score | visible rationale length |\n|---|---|---|---|\n| Gemini 3 Pro | 82.5 | 68.7 (-14.3) | 207 words |\n| Gemini 3 Flash | 80.4 | 69.4 (-11.0) | 230 words |\n| Claude Opus 4.5 | 77.3 | 73.1 (-4.2) | 327 words |\n| Claude Opus 4.6 | — | — | ≈421 words |\n\nWe choose the teachers on two criteria:\n\nFor production collection, we keep Gemini 3 Flash and replace the screened Claude Opus 4.5 with its newer Claude Opus 4.6 successor. We choose the successor based on model family; the screen itself tests only Claude Opus 4.5.\n\nThe training set contains 6,153 examples from 150 teacher games, with a further 10 games held out for validation.\n\nKeeping only the winner’s traces would waste data and select the luckiest copy of an otherwise identical policy. It would also remove the lower liquidity and slower tempo states that Qwen3-4B-Instruct needs to handle. So instead, we consider the top two models’ traces from each game while removing unparseable or illegal turns (only 10 training turns fail those validity gates). The final split contains:\n\n| split | teacher games | examples | bid turns | reveal turns |\n|---|---|---|---|---|\n| train | 150 | 6,153 | 4,990 | 1,163 |\n| validation | 10 | 420 | 342 | 78 |\n\nWe train for 1,200 optimizer steps ($\\sim 3$ epochs), and retain merged checkpoints from steps 1,000 and 1,200.\n\nDespite the growing train–validation gap, the loss curve shows no sign of overfitting over the evaluated range.\n\nOn a 6-model validation panel, step 1,200 finishes first in 64.7% of games, including 50% of those against two Claude Opus 4.6 copies. We select it.\n\nCheckpoint selection uses a fixed 6-model panel on the 10 validation deals. Qwen faces two copies of each opponent and occupies all three seats, for 180 panel games, plus 10 self-play games.\n\n| checkpoint | panel first-place rate | vs two Claude Opus 4.6 copies | parseable | illegal reveal | self-play mean score |\n|---|---|---|---|---|---|\n| base Instruct | 2.8% | 0.0% | 99.6% | 5.3% | 59.77 |\n| SFT step 1,000 | 61.7% | 16.7% | 100% | 0.9% | 64.27 |\n| SFT step 1,200 | 64.7% | 50.0% | 100% | 0% | 61.63 |\n\nThe self-play mean is only a sanity check, since all three copies co-evolve, but it does show that the model hasn’t degenerated.\n\nAcross a reconstructed audit set of 7 mainline GRPO configurations, the two runs played against the SFT checkpoint remained flat. Several others improve against a fixed scripted heuristic, demonstrating opponent-specific learning. Because five configurations did not record the same stationary anchor, those gains don’t measure transfer.\n\nBecause first-place rate among three evolving, co-adapting copies remains approximately $\\frac1{3}$ regardless of how much the policy evolves, population-based reward isn’t sufficient. Instead, we need an anchor, and the initial SFT policy is the perfect choice.\n\nThe evidence uses two instruments, which should not be conflated:\n\nThe figure separates these two measurements. Several policies learn to exploit the scripted heuristic, but neither directly anchored run improves against the frozen SFT policy.\n\nNone of these results rules out a richer league design. In the directly anchored SFT-anchor mixed-pool run and the later weak-KL ablation, opponent mixing and weaker KL likewise fail to produce a positive anchor trend.\n\nThe configurations use the same DAPO-style clipped objective; the labels below describe what changed in the training pool or schedule. **Snapshots** are lagged copies of the learner, and **SFT anchor** is the frozen step-0 policy. The two snapshots-plus-Flash rows are separate replications of the same basic pool. The frontier-API pool substitutes Gemini 3.1 Pro, Claude Opus 4.6, and Claude Sonnet 4.5; the 400-step run doubles the usual training horizon; seat rotation moves the learner through all three seats; and the final mixed pool combines the SFT anchor, lagged snapshots, and a decaying scripted-heuristic share. The figure abbreviates that final pool as “heterogeneous.”\n\n| GRPO configuration | scripted-heuristic $\\Delta$ own score vs SFT (95% CI) |\n|---|---|\n| snapshots + Flash (replication 1) | +4.64 [2.68, 6.57] |\n| snapshots only (400 steps) | +16.48 [14.26, 18.93] |\n| snapshots only (replication) | +4.36 [2.89, 5.88] |\n| snapshots only + seat rotation | +4.04 [−3.88, 11.88] |\n| snapshots + Pro/Opus/Sonnet | −2.07 [−3.89, −0.31] |\n| snapshots + Flash (replication 2) | −1.16 [−3.22, 0.89] |\n| SFT anchor + snapshots + heuristic | +0.65 [−1.14, 2.43] |\n\nThe heuristic contrasts use 60 matched seed groups with eight samples each, except the 24-game seat-rotation diagnostic. The optimizer can learn a narrow exploit.\n\nIn each run, one seat is trainable while the other two come from some mixture of frozen SFT, lagged snapshots, a scripted heuristic, and external models. For each seed, seat assignment, and opponent table, the trainable policy plays the same deal $K=8$ times with different sampled responses. Each fresh rollout batch supports six optimizer steps.\n\nThe terminal reward is a squashed competitive margin: each player’s score minus the average of the other two plus a bonus for finishing first. Illegal actions are penalized, and a lightly weighted shaping channel assigns intermediate credit between rounds.\n\nFor player $i$, playing against $j$ and $k$, define competitive margin as $m_i=s_i-\\frac{s_j+s_k}{2}$. The terminal reward is\n\n$$ R_i^{\\text{terminal}} =\\tanh\\left(\\frac{m_i}{19.6}\\right)+0.1 \\cdot \\mathbf 1[i\\text{ wins}] $$We calibrate the scale 19.6 so the median absolute margin maps to 0.5 via $\\tanh\\left(\\frac{10.75}{19.6}\\right) \\approx 0.5$. This compresses extreme wins and losses while preserving their order via monotonicity. We add a shaping channel (weighted by 0.01) that assigns intermediate credit from reconstructed public-score changes between rounds. Then, a terminal correction aligns its cumulative accounting with the final margin. Illegal actions receive $-0.5$.\n\nLet $r_u$ denote the terminal, legality, or shaping reward assigned at turn $u$. For each trainable turn $q$, define the return $G_q=\\sum_{u\\ge q}r_u$ and subtract an exponential-moving-average baseline $b_q$ indexed by game phase and seat role. Let $\\mathcal G$ contain all trainable turns from the eight sibling games. Then, we define the advantage here as\n\n$$ A_q=\\frac{(G_q-b_q)- \\operatorname{mean}_{v\\in\\mathcal G}(G_v-b_v)} {\\operatorname{std}_{v\\in\\mathcal G}(G_v-b_v)} $$Only turns from the trainable seat receive gradients.\n\nWe sample at temperature 1.0 and top-$p=0.95$, then update a rank-32 LoRA with learning rate $10^{-5}$ and KL coefficient 0.01.\n\nThe SFT-anchor mixed-pool run completes 100 optimizer steps with 192 games per rollout generation. Taken together, the diagnostics below rule out a dead adapter, broken reward ordering, or format collapse as explanations for the null effect.\n\n| diagnostic | SFT-anchor mixed-pool run |\n|---|---|\n| mean KL from reference | 0.0037 |\n| maximum KL | 0.0061 |\n| mean PPO ratio clip fraction | 0.0012 |\n| advantage variance | 0.81 |\n| zero-standard-deviation groups | 0 |\n| advantage↔terminal-margin Spearman | 0.902 |\n| training parse failures | 5 / 66,649 |\n| final-eval illegal actions | 0 / 480 |\n\nA second problem is assigning those wins to individual decisions.\n\nGRPO is easy to engineer in this setup. Its statistical fit is weak for this task. It avoids training a separate critic, yet this setup is far from the typical single-turn GRPO task. A game lasts roughly 15–25 rounds of interleaved bids, reveals, loans, and investments. Most reward arrives at the end, so an early bid and a late reveal can inherit nearly the same verdict even though either may have changed every state that follows. The same-deal comparison removes hand luck. It leaves within-trajectory credit unresolved.\n\nWe don’t claim that GRPO cannot work on long-horizon or multi-turn tasks. Dense verifiable rewards, process supervision, much larger groups, or environments in which one decision dominates the outcome can still make trajectory-level comparison useful.\n\nFor each fixed board $b$, let $\\bar Y_b$ and $s_b^2$ be the sample mean and within-board variance of its rollout returns. Because the observed variance of board means includes finite-sample noise, we estimate the true between-board variance as\n\n$$ \\hat\\sigma^2_{\\text{board}} =\\max\\left(0, \\operatorname{Var}_b(\\bar Y_b)-\\frac{\\mathbb E_b[s_b^2]}{K} \\right) $$with within-board variance $\\hat\\sigma^2_{\\text{trajectory}}=\\mathbb E_b[s_b^2]$. In the probe, 99.8% of the estimated variance lies among trajectories from the same board, while only 0.2% lies between corrected board means. The within-board term contains both action effects and continuation noise; it isn’t a clean action signal. A state-value critic must learn the much smaller between-board component from labels dominated by within-board variation.\n\nUnder relative-margin self-play, all three copies benefit from the same favorable deal, which suppresses most common board advantage. The fixed-heuristic probe allows repeated outcomes to recover more of the position’s value.\n\nThe probe argues against learning $V(I)$ from these particular relative-margin labels but doesn’t rule out state values in general. However, an action-conditioned value $Q(I,a)$ remains viable because it could still distinguish bids within the same state.\n\nThe first plausible fix is a [piKL](https://arxiv.org/abs/2210.05492)-style target, inspired by Meta’s work on Diplomacy, a seven-player game outside the 2p0s setting. It builds a search distribution that favors bids with higher action value $Q(I,a)$ while paying a KL penalty for moving too far from the SFT policy. Here, $Q(I,a)$ means the expected outcome after taking bid $a$ from information state $I$ and playing the game to completion.\n\nWe estimate $Q$ by simulation. The acting player cannot see the opponents’ hands or the undealt deck order, so we sample several *worlds* $w$: complete assignments of the hidden information that remain consistent with the player’s hand and the public history. In each world, we force candidate bid $a$, let the SFT continuation policy $\\pi$ finish the game in the acting seat while policy $\\omega$ controls the opponents, and read off the terminal score margin:\n\nHere, $\\xi$ is the continuation of the game generated by $\\pi$ and $\\omega$. We score every candidate on the same sampled worlds to reduce variance. The estimate $Q$ still depends on $\\omega$ because changing the continuation opponent can reorder bids even when $(I,a)$ stays fixed.\n\npiKL combines these action values with sampled action frequencies $\\hat\\tau$ from the SFT policy, upweighting high-$Q$ bids while remaining KLed to the SFT policy:\n\n$$ \\pi_{\\text{piKL}}(a\\mid I)\\;\\propto\\;\\hat\\tau(a\\mid I)\\,\\exp\\!\\left(\\frac{Q(I,a)}{\\lambda_{\\mathrm{KL}}}\\right) $$The temperature $\\lambda_{\\mathrm{KL}}$ controls the strength of the anchor. As $\\lambda_{\\mathrm{KL}}\\to\\infty$, the $Q$ tilt vanishes and the distribution returns to the SFT frequencies $\\hat\\tau$; as $\\lambda_{\\mathrm{KL}}\\to0$, it concentrates on the sampled candidate with the largest $Q(I,a)$. The $\\lambda_{\\mathrm{KL}}=0$ arm in the table implements that limiting argmax directly.\n\nWe label decisions with this $Q$, build piKL targets at several $\\lambda_{\\mathrm{KL}}$ values, and distill them with LoRA SFT. Each candidate then occupies every seat against two frozen SFT copies over the same 120 games, 40 per seat.\n\n| $\\lambda_{\\mathrm{KL}}$ | first-place rate | mean margin (95% CI) | paired Δ margin vs $\\lambda=\\infty$ (95% CI) |\n|---|---|---|---|\n| $\\infty$ | 0.425 | +1.23 [−2.21, 4.68] | — |\n| 1 | 0.417 | +2.73 [−1.06, 6.51] | +1.49 [−3.37, 6.35] |\n| 0.3 | 0.317 | −1.38 [−4.72, 1.95] | −2.62 [−7.12, 1.89] |\n| 0.1 | 0.292 | −1.79 [−5.06, 1.48] | −3.02 [−7.55, 1.51] |\n| 0 | 0.350 | −0.25 [−3.64, 3.13] | −1.49 [−6.37, 3.40] |\n\nMean-margin arm intervals use the normal approximation; paired intervals use a $t$ interval over matched games.\n\nAt 40 bid nodes with up to six candidates each, changing only the continuation opponent from self-play to base Qwen reduces median action-rank correlation with the self-play ordering from 1.0 to 0.43, while top-action agreement falls from 100% to 27.5%. The sign of each candidate’s advantage over the reference bid still agrees 72.5% of the time. The probe is too small, and the alternative opponent too weak, to establish the mechanism by itself. It does show that fine $Q$ rankings can depend strongly on the continuation policy.\n\nA second five-arm sweep replaces the self-play continuation with Gemini 3 Flash. Relative to its $\\lambda=\\infty$ control, the $\\lambda=1$ and argmax contrasts are +6.10 [0.38, 11.82] and +5.81 [0.70, 10.93]. These comparisons are exploratory, uncorrected for multiplicity, and unreplicated.\n\nOn a separate 43-node offline proxy, aggregating $Q$ over five fixed continuation types also never beats the single-self-play-$Q$ baseline. This diagnostic uses an offline proxy and provides no live policy comparison.\n\nGRPO’s group return cannot identify which bid changes the outcome. PPO could instead update from an action-specific advantage supplied by a learned critic. Before paying for an actor update, we test whether such a critic can rank candidate bids reliably enough to be useful.\n\nThe actor produces a bid using $I$ before the opponents’ bids are known. Once the auction resolves, those bids become public. During training, a critic can use this newly revealed information even though the actor never receives it at inference (centralized training with decentralized execution). We can fix the observed opponent bids $\\mathbf b_{-i}^{\\mathrm{obs}}$ to determine what happens when only our bid changes.\n\n$$ Q^{\\mathrm{obs}}_{\\pi,\\omega}(I,a) =Q_{\\pi,\\omega}\\!\\left( I,a\\mid\\mathbf b_{-i}=\\mathbf b_{-i}^{\\mathrm{obs}} \\right) $$Most raw $Q$ variation reflects whether the state itself is favorable. To isolate differences among bids, we center each target over the retained candidate set $C(I)$:\n\n$$ \\widetilde Q_C(I,a)=Q^{\\mathrm{obs}}_{\\pi,\\omega}(I,a) -\\frac{1}{|C(I)|}\\sum_{a'\\in C(I)} Q^{\\mathrm{obs}}_{\\pi,\\omega}(I,a') $$This target centers candidates with equal weight. A PPO update would instead require an advantage weighted by the actor policy.\n\nAt each treasure-bid state reached by the actor, we sample $K=16$ bids and compress them to 6 diverse candidates. The selection retains the played bid while approximately preserving both the policy’s sampled mass and the numerical range of bids. We score every candidate on the same sampled worlds and random seeds, and hold the opponents’ observed bids fixed. After the forced bid, a deterministic frozen SFT policy completes the game. The labels therefore apply to that frozen continuation policy; an updated PPO actor would induce a different distribution.\n\nFor 32 games, we evaluate each candidate with $M=64$ hidden-world samples to get 379 bid states and ~1.5k candidate rows. Because smaller estimates reuse prefixes of the same 64 worlds, we can compare 4, 8, 16, or 32 worlds without further rollouts. “Pair-sign agreement” measures whether actions separated by a $Q$ gap of at least 0.05 and 95% paired sign confidence retain their ordering while regret measures how much reference $Q$ is lost by selecting the lower-sample-budget argmax.\n\n| worlds | pair-sign agreement with $M=64$ | mean regret vs $M=64$ | 90th-percentile regret vs $M=64$ | all-state argmax agreement |\n|---|---|---|---|---|\n| 16 | 95.45% | 0.0207 | 0.0770 | 55.3% |\n| 32 | 99.77% | 0.0105 | 0.0370 | 71.1% |\n| 64 | 100% | 0 | 0 | 100% |\n\nSince the later training set retains confidently separated pairs, we adopt 32 worlds at half the rollout cost.\n\nAcross all 379 calibration states, the mean spread from worst to best retained action is 0.2149 in $\\tanh$-margin utility, and the estimated best action improves 0.0931 over the SFT-weighted candidate mean. That second number is an in-sample rollout-oracle bound, not realizable policy uplift. The average gap between the top two is only 0.0599, and the SFT policy’s played action is already best 36.1% of the time.\n\nMost candidate-$Q$ variation is irrelevant to choosing a bid: 84.8% lies between states, while only 15.2% separates actions within a state. To be clear, this split measures paired rollout-$Q$ estimates grouped by decision state while the earlier 99.8% figure measures raw whole-trajectory variation across replicas grouped by initial deal. PPO can use only the within-state component because a state-wide offset raises or lowers every candidate together.\n\nAcross 4 split seeds, 4-fold cross-validation fits small structured models to multi-action states from the $M=64$ reference labels. In the table, “uplift” is offline $Q$ improvement over the SFT candidate mixture.\n\nEach non-control row is a gradient-boosted tree ensemble fitted to the candidate-$Q$ labels. The centralized model receives the observed opponent bids, while the actor-visible model does not. Two negative controls either shuffle the targets or drop the candidate bid, leaving a model that can predict only common state value. “Filtered pair accuracy” scores only the confidently separated action pairs defined above.\n\n| model | filtered pair accuracy | full-set argmax | mean $Q$ uplift vs SFT |\n|---|---|---|---|\n| centralized context | 74.8–75.9% | 33.4–35.9% | +0.0268 to +0.0315 |\n| actor-visible state only | 72.3–74.6% | 32.2–35.6% | +0.0272 to +0.0314 |\n| shuffled-target control | 44.3–51.9% | 23.8–29.4% | −0.0309 to −0.0062 |\n| no-action control | all ties | 23.2% | −0.0680 |\n\nWe reuse three confidence gates from calibration: gap $\\ge0.05$, 95% sign confidence, and at most 6 pairs per state. The SFT-initialized critic’s Bradley–Terry loss ranks the better bid above the worse one, while a Huber regression term trains the numeric centered value at weight 0.25.\n\n| critic | filtered pair accuracy | full-set argmax (75 groups) | mean grouped regret | $R^2$ on centered advantage |\n|---|---|---|---|---|\n| non-leaky linear-bid baseline | — | 40.0% | — | — |\n| 1 epoch, seed 0 | 62.9% | 24.0% | 0.0989 | −2.31 |\n| 1 epoch, seed 1 | 59.5% | 25.3% | 0.1096 | −107.54 |\n| 1 epoch, seed 2 | 64.8% | 34.7% | 0.1051 | −17.64 |\n| 3 epochs, seed 0 | 66.2% | 25.3% | 0.1011 | −16.56 |\n\nEven with the preregistered gates, every neural run misses the 55% argmax threshold and the 40% linear-bid baseline.\n\nThe next meaningful rung uses 128 games and roughly 1,485 decision states. It costs an estimated 328 H200-hours, with $\\pm40\\%$ uncertainty. Reaching 1,024 games and then evaluating on the sequestered 128-game final test approaches 3,000 H200-hours.\n\nPPO could in principle improve beyond a fixed expert, but it must first compress expensive rollouts into small critic differences and then refresh them as the actor’s state distribution moves.\n\nWe therefore stop after the critic diagnostics: no production label corpus and no actor update.\n\nA corpus of policy-generated games contains two kinds of information:\n\nGRPO uses the first at trajectory level, while the action critic tries to refine it to individual decisions. We model the second as a conditional price law $\\hat F$. This law makes one sealed-bid auction inside the multiplayer game tractable. Given an estimated lot value and a distribution over opponent bids, we can score every legal bid by expected surplus.\n\nA point model treats one predicted opponent bid as certain. A distribution instead asks what fraction of the predicted bid mass each candidate beats:\n\n$$ U_{\\hat F}(b)=(\\hat V-b)\\hat P_{\\hat F}(\\text{win}\\mid b) $$One final search variant keeps the terminal rollouts but replaces the opponents’ current bids with deterministic predictions from a fitted market model. The best arm changes paired margin by only +0.58 [−5.6, +6.8], therefore unresolved.\n\nA separate diagnostic asks whether a much better point predictor would solve the problem. A gradient-boosted tree trained on ~3.2k logged Gemini 3 Flash bids reaches an out-of-fold mean absolute error (MAE) of 1.10 coins with essentially zero bias (all variance). However, it still predicts the wrong side of the actual win/loss threshold on 26.1% of auctions. 55% of auctions resolve within one coin, where Flash’s own sampling noise straddles the boundary even when the conditional mean is accurate.\n\nPrediction error now shifts probability mass across outcomes. We develop the first version on Gemini 3 Flash logs, then test whether the SFT policy’s own bids can fit the same price law.\n\nThe acting player knows three useful quantities for color $c$:\n\nThere are six gems of each color in the game. Because the player’s remaining hand eventually enters the display, a live lower bound on the final count is $L_c=\\text{display}_c+\\text{own-hand}_c$. Collected gems never enter the display, so an upper bound is $U_c=6-\\text{collection}_c$. An 11-feature classifier uses these bounds, reveal progress, and round number to predict the final count $n_c$, one of 7 values from 0 through 6. It never sees opponents’ hidden hands. The prediction also remains independent of the Value Chart until inference, when we convert the count distribution to points.\n\nWe split games 70/10/20 for fitting, calibration, and held-out testing and train using a regularized MLP for at most 300 iterations.\n\nFor a lot with colors $g_1,\\ldots,g_r$, the head outputs the 7-way count distribution $\\hat P(n_{g_i}=k\\mid I)$. We map that distribution through the active chart to get each gem’s raw expected value, then apply an isotonic calibration map $f$ before summing the gems. This flexible one-dimensional correction curve can fix systematic bias while preserving monotonicity:\n\n$$ \\hat V_{\\text{lot}} = \\sum_{i=1}^r f\\!\\left(\\sum_{k=0}^{6} \\hat P(n_{g_i}=k\\mid I)\\operatorname{Chart}(k)\\right) + \\operatorname{MissionBonus}(\\text{collection},\\text{lot}) $$Here $\\operatorname{Chart}(k)$ is the eventual **per-gem** value when $k$ gems of that color appear in the final display. The mission term is computed exactly. Thus $\\hat P$ is the classifier output, while $\\hat V_{\\text{lot}}$ is the calibrated value supplied to the selector.\n\nA histogram gradient-boosted regressor predicts each opponent seat’s conditional mean bid $\\hat\\mu_s(x)$ from 16 public features (e.g. coin count, lot size and displayed value, recent clearing prices, and tiebreak position). We fit it on ~3.6k Gemini 3 Flash bid rows, cross-validated in folds grouped by paired seed. The grouped out-of-fold MAE is 1.230 coins with bias +0.013; 69.4% of predictions land within one coin.\n\nBut we can go further. Instead, let $B_s$ be opponent seat $s$’s actual bid in the current auction. We retain every out-of-fold residual $\\epsilon=B_s-\\operatorname{round}(\\hat\\mu_s(x))$ and pool them into an empirical noise distribution. At inference, we shift those historical errors by the current predicted mean, round to integers, and clamp to the opponent’s legal budget. With $N$ stored residuals and current budget $c_s$,\n\n$$ \\hat P(B_s=z\\mid x)=\\frac{1}{N}\\sum_{r=1}^N\\mathbf 1\\left[z=\\operatorname{clip}\\big(\\operatorname{round}(\\hat\\mu_s(x)+\\epsilon_r),0,c_s\\big)\\right] $$For clarity, suppose the current mean prediction is 8 and the residual law puts 20% of its mass at −1, 50% at 0, and 30% at +1. The implied opponent bids are 7, 8, and 9 with probabilities 0.2, 0.5, and 0.3. Bidding 9 is no longer treated as a certain win: it beats 70% of that distribution and wins or loses the 30% tie mass according to public tiebreak order.\n\nWith exact tiebreak handling, the probability that candidate bid $b$ beats opponent $s$’s unknown bid $B_s$ is\n\n$$ p_s(b)=\\hat P(B_sp$ but losing the tiebreak makes the cheapest winning bid $p+1$ instead of $p$.\n\nTo isolate the price model, every counterfactual bidder receives realized terminal gem value $V$ and no mission bonus. Later, our selector replaces that privileged value with $\\hat V$ plus exact mission value.\n\nWe compute regret differences within each seed cluster. The oracle beats the logged SFT bids by 1.304 coins per decision, while retaining residual uncertainty reduces regret by another 0.456 coins per decision relative to the point version using the *same fitted mean*.\n\nOur selector first samples an ordinary SFT response, then calculates the EV curve for every legal bid and takes the argmax to get $b^*$. To prevent weak analytic preferences from overwriting coherent LLM behavior, it deviates only when the estimated improvement in immediate expected value is at least one coin:\n\n$$ U(b^*)-U(b_{\\mathrm{SFT}})=(\\hat{V}-b^*)\\hat{P}(\\text{win} \\mid b^*, x) - (\\hat{V}-b_{\\mathrm{SFT}})\\hat{P}(\\text{win} \\mid b_{\\mathrm{SFT}}, x)\\ge1 $$We compare three strategies: selector off, selector on, and selector on with history repair so future turns see a consistent record of changed actions. All three play the same 150 seeds against two Gemini 3 Flash opponents, with Qwen fixed in seat 0. We compare each treatment with selector-off on the same seed.\n\nThe selector fails live: raw paired margin changes by −5.01 points per game with 95% CI [−10.04, +0.02]. The large offline regret reduction does not translate into better full-game performance.\n\nWe introduce a control variate to reduce auction-resolution noise: a decision should not receive full credit because an opponent happens to sample below its usual price, or full blame because the opponent samples above it.\n\nFor each treasure decision $d$, let $y_d=1$ if the bid wins and $y_d=0$ otherwise, let $p_d$ be the model’s predicted win probability, and let $V_d-b_d$ be the lot’s estimated surplus. Define the decision-level resolution surprise as $x_d=(y_d-p_d)(V_d-b_d)$. This converts auction luck into score units: a surprising outcome matters more when the lot’s surplus is larger. For arm $a$, game $g$, and seat $k$, sum the surprises over that seat’s decisions:\n\n$$ X_{a,g,k}=\\sum_{d\\in\\mathcal D_{a,g,k}}x_d $$For the adjustment to be unbiased, the model must satisfy the stronger, value-weighted calibration condition\n\n$$ \\mathbb E[x_d\\mid I_d] =\\mathbb E[(y_d-p_d)(V_d-b_d)\\mid I_d] =0, $$where $I_d$ is the information available at the decision. Ordinary calibration of $p_d$ alone isn’t sufficient because the surplus $V_d-b_d$ varies across decisions. For each matched seed, define the seatwise luck contrast $Z_{g,k}=X_{\\mathrm{on},g,k}-X_{\\mathrm{off},g,k}$ and the paired margin difference $D_g=\\text{margin}_{\\mathrm{on},g}-\\text{margin}_{\\mathrm{off},g}$. We regress\n\n$$ D_g=\\tau+\\sum_{k=0}^{2}\\beta_k Z_{g,k}+\\epsilon_g. $$The intercept $\\tau$ is the adjusted treatment effect. In a paired pilot, this regression explains 42% of the paired variance, a 1.72× reduction. No direct validation of the stronger value-weighted calibration condition is available, so later gates report the raw paired effect as the primary evidence and the adjusted estimate as model-assisted corroboration.\n\nAdjustment removes about 3 points of adverse resolution luck, yielding −1.88 [−6.57, +2.81]. The adjusted history-repair arm is harmful at −5.16 [−9.90, −0.41]. Neither adjusted estimate overturns the raw result.\n\nThe table shows why:\n\n| paired component | treatment minus control |\n|---|---|\n| treasure spend | +6.81 |\n| final gem value | +5.17 |\n| mission rewards | +3.17 |\n| loan repayments | +3.73 |\n| investment returns | −7.13 |\n| own final score | −3.61 |\n\nImproving one-auction expected value harmed the full game because myopic spending depleted liquidity. Extra loan repayment and foregone investment returns together cost 10.9 points per game, overwhelming the gains from gems. The intervention also raised early clearing prices by 1.48 coins and lowered late prices by 1.04 after liquidity was depleted.\n\nWhat’s most suspect is the calibration error $\\hat V$, which overstates realized lot value on this distribution, so perhaps the gate selects lots worth less than their price. But lots won on deviation and non-deviation decisions carry similar value bias. The one-coin gate isn’t uniquely selecting overvalued lots and cannot explain the loss.\n\nThe table points to a more basic omission: myopic EV ignores the option value of a coin.\n\nAt face value, a coin contributes one point to final score. During the game, its real value is the best thing you can still do with it. You can buy the current lot, wait for another auction where the coin goes further, sink it into an investment, or use it to deny an opponent a loan. You have the right but not the obligation to spend the coin, which makes liquidity option-like.\n\nThe timing effect cuts both ways. Early in a game, holding a coin preserves many possible uses across the remaining deck. Late in a game, if valuable lots remain and the other players are nearly broke, the same coin can carry unusual market power because few rivals can contest it. The relevant quantity is the continuation score lost when that coin is spent in the current state, which can exceed its face value.\n\nLet $J(I_t,c)$ be expected continuation score from information state $I_t$ with $c$ coins, holding the rest of the state fixed. The marginal continuation value of the last coin is\n\n$$ \\lambda(I_t,c)=J(I_t,c)-J(I_t,c-1) $$Rather than solve that dynamic program, we hold the premium constant: $\\lambda_{\\text{coin}}$ approximates the value of a coin above the one point already charged when cash leaves the final score. The corrected objective subtracts a fixed calibration offset $\\delta$ from predicted lot value and charges that constant shadow premium for the future flexibility lost with every paid coin:\n\n$$ U_{\\delta, \\lambda_{\\text{coin}}}(b) =\\left[(\\hat V-\\delta)-(1+\\lambda_{\\text{coin}})b\\right] \\hat P(\\text{win}\\mid b) $$The previous table shows why the premium must be positive: the invest and loan auctions move by roughly $\\frac{3.73+7.13}{6.81}\\approx1.60$ points for each extra coin of treasure spending.\n\nWe therefore sweep the liquidity premium $\\lambda_{\\text{coin}}$ and the calibration offset $\\delta$ at gate 1, plus three gate-only arms. The margin-only best is $\\lambda=1,\\delta=0$ at +26.42 points. But every $\\lambda\\ge1$ arm enters a proxy-rescue regime where the selector overbids, and it gains by vetoing proposals that the live LLM does not make.\n\nWe therefore choose $\\lambda=0.5,\\delta=2$. During evaluation, it yielded +22.62 with 4.33 deviations per game.\n\nWe measure the combined fix on 150 paired seeds against Gemini 3 Flash opponents.\n\n| metric | control | paced selector | Δ (treatment − control) |\n|---|---|---|---|\n| first-place rate | 36.0% | 59.3% | +23.3 percentage points |\n| mean margin | −7.17 | +4.99 | +12.16 [7.61, 16.71] |\n\nThe model-assisted adjustment is +12.62 [8.28, 16.96]. Pacing also preserves the extra asset value while removing the financing damage:\n\n| paired component | paced selector minus control |\n|---|---|\n| treasure spend | −0.53 |\n| final gem value | +5.44 |\n| mission rewards | +2.77 |\n| loan repayments | +1.00 |\n| investment returns | −0.25 |\n| own final score | +8.41 |\n\nTreasure count changes by only +0.02 from control. The financing penalties collapse, and the early/late clearing-price cascade disappears.\n\nThis result applies to a gated treasure-bid intervention, not a complete game policy. The SFT model still handles treasure bids rejected by the gate, reveals, loans, and investments.\n\nThe price law above is fitted on Gemini 3 Flash bids, making it an opponent-specific component built from deployment-policy data. We refit the same model family on roughly 1,800 SFT-policy bids, keeping $\\lambda_{\\text{coin}}=0.5$, $\\delta=2$, and gate 1.0. This removes Flash bids from model fitting, although the three decision-rule constants still come from Flash-centered diagnostics.\n\nOn the Flash replay corpus, the SFT-fitted law retains 91% of the oracle-value offline regret improvement over the logged SFT bids: regret falls from 4.12 to 2.94, as opposed to 2.82. This shows similar decision utility on these logged auctions but not convergence between the policies.\n\nAcross 150 new paired seeds, the control’s first-place rate is 32.7% and the SFT-fitted selector reaches 50.7%. Raw paired margin improves by +8.47 [3.99, 12.94], while control-variate-adjusted margin improves by +10.19 [6.14, 14.24] with $t=4.93$.\n\nThe SFT-fitted price law reproduces the main fix, but at a smaller magnitude by retaining 81% of the Flash-fitted selector’s +12.62 points. Relative to control, the early/late market cascade does not reappear.\n\nExIt alternates between a policy that visits states and an expert that improves the action taken in those states. If the current policy $\\pi$ induces a state distribution $I\\sim d_\\pi$, the expert supplies\n\n$$ a^*(I)=\\arg\\max_{a\\in\\mathcal A(I)}U(a\\mid I) $$and the resulting dataset $\\mathcal D_E$ trains a new policy by imitation:\n\n$$ \\theta'=\\arg\\min_\\theta -\\mathbb E_{(I,a^*)\\sim\\mathcal D_E} \\left[\\log\\pi_\\theta(a^*\\mid I)\\right] $$The new policy can generate another state distribution, which the expert can label again, giving the loop $\\pi_k\\rightarrow d_{\\pi_k}\\rightarrow\\mathcal D_E\\rightarrow\\pi_{k+1}$. Unlike ordinary SFT from a fixed teacher corpus, the supervision is concentrated on states the current policy actually reaches and on actions the expert would change.\n\nHere, we run one such iteration. The current SFT policy generates the games, and the paced analytic selector is the expert: its utility $U$ combines predicted lot value, the opponent bid distribution, and the shadow price of liquidity. We then distill its selected bids into the model weights.\n\nWe play 400 fresh mixed-policy games. The analytic selector controls seat 0 using the SFT-fitted price law and gate, while the other two seats remain plain SFT policies. This holds the opponent policies fixed, although the selector’s bids can still change later budgets and market history.\n\nThe run produces ~4.7k bid decisions, including about 1,200 selector deviations. It also supplies another opponent-type check: the selector-augmented SFT policy finishes first against two SFT clones in 48.7% [43.9, 53.6] of games, above the $\\frac1{3}$ baseline.\n\nThe SFT export contains three row types: expert deviations, pass-throughs where the selector agrees, and loan and investment anchors.\n\nPass-through and financing anchors come from all three seats. Only seat 0 contains selector deviations, so all-seat anchors reduce the risk that the adapter learns a `player_id = 0`\n\nshortcut.\n\nIn one game, round 9, seat 0 has 22 coins and a public collection of three Green and one Yellow gems. The current auction offers two more Green gems, enough to complete a mission, and the ordinary SFT policy samples a bid of 22.\n\nThe frozen expert values the lot at 30.59 coins after the calibration correction, including the 10-point mission bonus it would complete, and predicts opponent mean bids of 10 and 11. A bid of 22 wins with certainty, but once each coin’s option value is priced in, its paced utility is negative. A lower bid can preserve most of the win probability at a far better price:\n\n$$ \\begin{align*} U(22)&= [30.59-1.5(22)]\\cdot 1.0000=-2.41 \\\\ U(13)&= [30.59-1.5(13)]\\cdot 0.8825=+9.79 \\end{align*} $$The final 12 percentage points of win probability cost too much, so the expert shades the bid to 13. The training target reads:\n\nThis treasure is worth about 31 to me net of bias. 22 would overpay once each coin’s option value is priced in (budget 22); shading to 13 keeps the win probability at ~88% and earns more surplus per coin. Expected surplus favors 13.\n\n`{\"bid\": 13}`\n\nThat prose comes from a template filled with four logged quantities: $\\hat V$, both bids, win probability, and remaining budget. Every deviation target initially has the same synthetic voice.\n\nThe log preserves neither the SFT policy’s free-form rationale nor a student response. This row therefore illustrates the supervision target.\n\nWe hold out 15% of games by seed. The primary metric is EV closure: the share of the expert’s utility improvement over the recorded SFT bid that the student recovers on held-out deviation states, ranging from 0 (no gain) and 1 (matches the expert).\n\n$$ \\operatorname{closure} =\\frac{U(b_{\\text{student}})-U(b_{\\mathrm{SFT}})} {U(b_{\\text{expert}})-U(b_{\\mathrm{SFT}})} $$\n\nClosure measures how much of the expert’s utility improvement the student recovers on held-out deviation states, where the denominator is positive by construction. A value of 0 means no utility gain over the recorded SFT bid; 1 means equal utility to the expert, not necessarily the same bid; values above 1 are possible. The “SFT reference” below is a fresh stochastic response to the same prompts, so it need not reproduce the recorded $b_{\\mathrm{SFT}}$ and can score above zero.\n\nThe deviation:pass-through:anchor mixture is $2:2:1$, and we repeat each deviation three times to emphasize the new supervision over rows that merely clone existing SFT behavior. An earlier development build, used to choose that recipe, reaches 0.674 closure and 0.883 direction agreement. Direction agreement is the fraction of student bids that move from the recorded SFT bid in the expert’s direction. We then freeze the recipe and rebuild the corpus from the final selector run. On this final corpus, EV closure is 0.760 versus 0.316 for a fresh SFT response; direction agreement is 0.878 versus 0.523; parse validity is 1.00; and financing MAE is 1.751 versus 1.937.\n\nRemoving the selector entirely, the distilled weights improve raw paired margin by +7.97 [2.70, 13.25] over SFT. Against Gemini 3 Flash, the control wins 29.3% with mean margin −12.83, whereas the distilled weights win 40.7% with mean margin −4.85.\n\nThe analytic branch’s first measured held-out improvement beyond SFT is weights-only. Two implementation steps reduce the effect relative to the Flash-fitted analytic selector:\n\nParaphrasing the synthetic targets restores the SFT model’s response style without a detectable loss in game strength. The compact templates reduce median response length to 254 characters, versus roughly 1,016 for the SFT policy; a style-matched corpus has the SFT model paraphrase each deviation target in its own voice, restoring the original length.\n\nThe LoRA trained on paraphrased responses retains held-out closure of 0.718 and direction agreement of 0.849. In evaluation, raw paired margin improves by +8.31 [3.21, 13.42], and the adjusted estimate is +7.95 [3.16, 12.74]. Against the template LoRA, the adjusted difference is +0.02 [−5.06, +5.10], but it still retains 78% of the SFT-fitted selector’s adjusted live effect. We therefore use this LoRA for final evaluation.\n\nWe fit a new price law on roughly 7,000 bids from 200 games of distilled-model self-play. Its mean prediction differs from the SFT law by 0.753 coins, but the two laws’ best-response regrets differ by only 0.068 coins per decision.\n\nTo measure the remaining imitation gap, we add a selector fitted to that new price law on top of the distilled model. Relative to the SFT control, raw paired margin improves by +11.99 [6.41, 17.56], and the adjusted estimate is +10.94 [6.00, 15.88]. Relative to weights alone, the selector adds +3.95 [−0.83, 8.74]. It now changes only 15.9% of treasure decisions, versus 27.8% on the original SFT policy.\n\nThe optional selector produces +9.44 [4.55, 14.33] margin, with descriptive per-seat effects of +7.26, +10.94, and +10.12 across the 3 seats. Against two Claude Sonnet 4.5 opponents, it achieved +25.12 [13.78, 36.46] and against Gemini 3.1 Pro, +6.10 [−4.67, 16.87], directionally positive but underpowered.\n\nTwo limitations qualify these optional-selector results:\n\nThe table below consolidates the intervention ladder. Every value is a treatment-minus-control component change. Because the columns come from separate experiments, cross-column comparisons are descriptive. Its weights-only ledger comes from the compact-template LoRA; the figure that follows uses the style-matched LoRA selected for final evaluation.\n\n| paired component | myopic selector | paced selector | SFT-fitted | distilled LoRA | distilled LoRA + selector |\n|---|---|---|---|---|---|\n| treasure spend | +6.81 | −0.53 | −0.44 | −0.63 | +0.51 |\n| final gem value | +5.17 | +5.44 | +4.08 | +1.89 | +6.37 |\n| mission rewards | +3.17 | +2.77 | +2.40 | +2.23 | +3.07 |\n| loan repayments | +3.73 | +1.00 | +1.87 | +0.93 | +2.20 |\n| investment returns | −7.13 | −0.25 | −1.01 | −0.57 | −1.64 |\n| own final score | −3.61 | +8.41 | +5.62 | +4.33 | +7.33 |\n\nThe incremental live interval includes 0, and offline gates find the residual labels only weakly learnable, so another round of ExIt isn’t justified.\n\nAcross separate experiments, adjusted point estimates leave a 2.99-point gap; the direct within-run selector increment is +3.95 [−0.83, 8.74]. Applying the two measured distillations’ 73–78% retention range to that gap suggests only ~2.2 points from another round, but this estimate is just an extrapolation.\n\nThe final benchmark evaluates the style-matched distilled 4B weights alone, with no analytic selector active at inference.\n\nIn the 13-model Steiner-triple benchmark, direct comparisons resolve wins over 10 of 12 opponents. Against SFT, Distilled finishes higher in 75% of shared games [58%, 88%], an internal check outside the development evaluations.\n\nTo sharpen the ordering of the top three, we evaluate Distilled, Gemini 3.1 Pro, and Gemini 3 Flash together on 150 seeds with all three seat rotations. The resulting 450 games comprise 150 independent deal clusters; the intervals resample the three rotations together.\n\n| pair | fraction A finishes above B | 95% CI | mean score difference |\n|---|---|---|---|\n| Distilled vs Gemini 3 Flash | 0.56 | [0.52, 0.60] | +5.2 |\n| Distilled vs Gemini 3.1 Pro | 0.53 | [0.48, 0.57] | +0.8 |\n| Gemini 3.1 Pro vs Gemini 3 Flash | 0.59 | [0.54, 0.65] | +4.4 |\n\nThe specialist beats Gemini 3 Flash and is statistically unresolved against Gemini 3.1 Pro.\n\nLet’s consider the raw final-evaluation trajectories produced by the [distilled model](https://huggingface.co/djdumpling/qwen3-4b-instruct-megagem-distill-para). Below are 3 examples that illustrate smart decisions made by the model and reflect the types of reasoning distilled by ExIt.\n\nHere, the model plays seat 0 against GPT-5.5 and Gemini 3.1 Pro. By round 8, it has 14 coins and a collection of Yellow and Purple. A Blue gem would complete the 5-point Blue–Yellow mission, and revealing the Blue in its private hand would move Blue from 4 to 8 points per gem. One opponent has only 6 coins, while the model has tiebreak priority over the other.\n\nThe model writes:\n\nWinning this Blue gem would immediately complete the “specific_2_by” mission (Blue + Yellow) for 5 coins and put me one step closer to the “specific_3_bgp” mission (Blue + Green + Purple) for 10 coins.\n\nA bid of 6 ensures I beat Player 1’s maximum possible bid. If Player 2 bids 6, I win on tiebreak. If Player 2 bids higher, they are spending a significant portion of their capital early. Given the mission rewards and the value of the gem, 6 is a strong but efficient bid that preserves my coins for future rounds.\n\n`{\"bid\": 6}`\n\nThe bids are 6, 6, and 5. The model wins the tie, claims the mission, and reveals Blue. Ignoring any later increase in Blue’s price, the immediately visible surplus is already at least\n\n$$ 8\\ \\text{(gem value)}+5\\ \\text{(mission)}-6\\ \\text{(price)}=7 $$Recognizing both the value/surplus of Blue while accounting for covering price and coin preservation is what allows it to eventually win the game 93–89–59.\n\nThe same game produces a cleaner test of state tracking in rounds 13 and 14. Both opponents have exhausted their cash. In round 13, the model is first in tiebreak order, so it recognizes that a zero bid wins the Blue gem:\n\nPlayer 1 has 0 coins, Player 2 has 0 coins. Neither can bid above 0!\n\nI’m first in tiebreak order, so if we all bid 0, I win.\n\nThis is a free win at 0 coins! Since both opponents have 0 coins, they can only bid 0. I have tiebreak priority, so bidding 0 guarantees I win the Blue gem for free.\n\n`{\"bid\": 0}`\n\nAll three players bid 0, and the model wins. Winning moves it to the back of the tiebreak order. On the next auction, a two-gem Yellow–Blue lot, it updates rather than repeating the previous action:\n\nPlayer 2 > Player 1 > Me. So if everyone bids 0, Player 2 wins. I need to bid at least 1 to guarantee winning.\n\nBidding 1 is clearly optimal—it guarantees the win while costing almost nothing.\n\n`{\"bid\": 1}`\n\nThe other players again bid 0, so the model buys both gems for 1, and these two purchases contribute to the same 93-point win above.\n\nIn another game, the model plays seat 1 against Gemini 3.1 Pro and GPT-5.5. In round 13 it has 14 coins, while GPT-5.5 has 7 and acts earlier in ties. Purple would complete the 5-point Red–Purple mission, so the model chooses the exact guaranteed price:\n\nPlayer 2 has 7 coins and is ahead of me in the tiebreak order. To guarantee a win against Player 2, I must bid 8.\n\nSpending 8 now for a gem that is worth at least 12–16 coins (plus the 5-coin mission) is a net positive, and I still retain 6 coins for future rounds.\n\n`{\"bid\": 8}`\n\nThe bids are 0, 8, and 7, so the model wins Purple and the mission. It now has only 6 coins. The next card is a 10-coin loan, and its rationale connects that financing decision to the remaining mission:\n\nWinning this loan would provide immediate liquidity (10 - bid) to compete for the remaining 4 auctionable gems, which is crucial because I need a Yellow gem to complete the “specific_3_rgy” mission (10 coins) and a second Purple for the “same_2_purple” mission (5 coins). However, bidding too high on a loan is counterproductive as it increases my end-game debt without adding net value.\n\n`{\"bid\": 5}`\n\nThe loan bids are 2, 5, and 4. Winning raises the model’s available cash to $6+(10-5)=11$. The next auction is another Purple. GPT-5.5 still has 7 coins and the better tiebreak position, so the covering price is again $7+1=8$:\n\nPlayer 2 has 7 coins max. They might bid up to 7. I should bid 8 to guarantee winning over Player 2’s maximum.\n\nGiven the expected surplus-maximizing bid is 8, and this is worth ~17+ to me, bidding 8 is clearly optimal.\n\n`{\"bid\": 8}`\n\nThe bids are again 0, 8, and 7. The model wins its second Purple and the 5-point mission. It finishes first, 72–66–49, with\n\n$$ 2\\ \\text{coins}+56\\ \\text{gem value}+10\\ \\text{missions} -30\\ \\text{loan repayments}+34\\ \\text{investment returns}=72 $$This sequence is not enough either: an exact covering bid creates a liquidity shortage which a subsequent loan repairs to fund a second exact covering bid that wins the mission.\n\nStationary comparators matter: frozen-SFT anchors, fixed-opponent gates, and non-co-adapting panels answer different questions. Runs that improve against a scripted heuristic and runs anchored to frozen SFT are not measuring the same thing.\n\nThree quantities matter in each bid. $\\hat V$ estimates the asset, $\\hat F$ estimates the market, and rollout $Q$ estimates a full continuation against specified policies. The selector combines value and price into a local action.\n\nRegret replay identifies better current-auction purchases within the transitions it holds fixed. Its myopic policy creates a liquidity cascade in live games. Pricing the future use of cash turns an unresolved negative point estimate into a +12.16-point live gain.\n\nThe analytic expert also supplies a useful distillation target. Each label shows how a candidate bid changes price paid, win probability, and remaining budget, giving supervised training a decision-level advantage to imitate.\n\nIn the runs where transfer is directly tracked, relative outcome rewards did not produce measured transfer across the tested opponent populations. Policy-generated bid logs provide the useful learning signal by exposing price distributions, making liquidity cost explicit, producing distillable targets.", "url": "https://wpnews.pro/news/learning-megagem-from-self-play-to-price-discovery", "canonical_source": "https://djdumpling.github.io/2026/08/09/megagem.html", "published_at": "2026-08-09 04:00:00+00:00", "updated_at": "2026-08-10 03:39:47.743389+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-products"], "entities": ["Jane Street", "Qwen3-4B-Instruct", "Claude Sonnet 4.6", "Claude Opus 4.8", "GPT-5.5", "Gemini 3.1 Pro", "Hugging Face", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/learning-megagem-from-self-play-to-price-discovery", "markdown": "https://wpnews.pro/news/learning-megagem-from-self-play-to-price-discovery.md", "text": "https://wpnews.pro/news/learning-megagem-from-self-play-to-price-discovery.txt", "jsonld": "https://wpnews.pro/news/learning-megagem-from-self-play-to-price-discovery.jsonld"}}