cd /news/artificial-intelligence/how-krafton-turned-a-29b-moe-model-i… · home › topics › artificial-intelligence › article
[ARTICLE · art-171728] src=mindstudio.ai ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

How Krafton Turned a 29B MoE Model Into a 1.5B Dense LLM

Krafton converted its 29 billion parameter mixture-of-experts model Raon-1 Mini into a 1.5 billion parameter dense model, Raon-1 Nano, using a method detailed in a NeurIPS paper that earned an oral presentation. The technique scores the 128 experts per MoE layer with a diversity-aware scoring method, groups and merges similar experts into dense feed-forward layers, then distills the original model's behavior into the new network; Krafton reports the scoring approach beat pruning-based baselines by 6.3 points on average while training 1.6 times faster. Both models were trained from scratch by the South Korean PUBG maker, and the likely use case is on-device NPCs and AI teammates in Krafton's games, where Raon-1 Mini leads on instruction following and omniscience benchmarks despite lower MMLU Pro scores.

recommendation

When compressing a MoE model to dense, score experts with diversity-aware grouping and merge similar experts before distilling; Krafton measured a 6.3-point average gain over pruning baselines at 1.6x faster training.

more in playbook: Fine-tuning and data
by read9 min views1 publishedNov 18, 2026
How Krafton Turned a 29B MoE Model Into a 1.5B Dense LLM
Image: Mindstudio (auto-discovered)

Krafton's NeurIPS paper details a new way to convert a mixture-of-experts model into a dense model, shrinking 29B parameters to 1.5B.

What did Krafton actually build? #

Krafton, the South Korean company behind PUBG, trained a 29 billion parameter mixture-of-experts (MoE) model called Raon-1 Mini, then converted it into a dense 1.5 billion parameter model called Raon-1 Nano. The conversion method, described in a NeurIPS paper, scores and merges experts from the MoE model into a standard dense feed-forward network, then distills knowledge from the original model into the new one. This is different from typical model compression, which usually shrinks a dense model into a smaller dense model, or compresses an MoE model into a smaller MoE model. Krafton’s approach crosses architectures entirely: MoE in, dense out.

TL;DR #

  • Krafton converted its own 29 billion parameter MoE model , Raon-1 Mini, into a1.5 billion parameter dense model , Raon-1 Nano, using a technique described in a NeurIPS paper this year.
  • The method scores each expert in the MoE layer, groups similar experts together, merges each group into one dense feed-forward layer, then distills the original model’s behavior into the new dense network.
  • Krafton’s own diversity-aware scoring method for picking which experts to keep beat simple pruning-based baselines by6.3 points on average on their benchmarks, while training1.6 times faster .
  • Both parent and child models were trained from scratch by Krafton, not fine-tuned from an existing open model like Qwen, which means the company controls exactly what knowledge went in and could be stripped out.
  • Raon-1 Mini scores relatively low on general knowledge benchmarks like MMLU Pro but leads on instruction following and onomniscience benchmarks that reward a model for admitting it doesn’t know something , suggesting the model is tuned to defer to outside context rather than recall facts.
  • The likely use case is on-device NPCs and AI teammates in Krafton’s games, where a model needs to follow instructions, track game state, and stay in character, but doesn’t need broad world knowledge.
  • The paper earned an oral presentation at NeurIPS , a distinction reserved for a small share of accepted papers that the review committee judges worth a dedicated talk.

Enterprises use Remy.

Why convert an MoE model into a dense one at all? #

Mixture-of-experts models are efficient at training and inference time because only a fraction of their parameters activate for any given token. Raon-1 Mini, for example, has 29 billion total parameters but only about 4.3 billion active per token, spread across 128 experts, with 6 selected per token plus one shared expert. That efficiency is great for cloud serving, where you can batch requests and route tokens dynamically.

It’s a poor fit for edge deployment. A game running on a player’s GPU can’t easily take advantage of sparse expert routing the way a data center server can, and shipping 29 billion parameters worth of weights, even if only a fraction activate per token, is still a memory and burden on a machine that’s also trying to render a game. A dense model of equivalent active compute is simpler to deploy, quantize, and reason about on consumer hardware.

Most prior distillation work has stayed within one architecture family: dense-to-dense (Llama 3.2’s 1B model was pruned from an 8B model and trained on outputs from both the 8B and 70B versions), or MoE-to-smaller-MoE. Krafton’s paper targets the less common MoE-to-dense direction.

How does the MoE-to-dense conversion actually work? #

The paper lays out a four-step process:

  1. Score every expert. Krafton developed a “diversity-aware scoring” method to evaluate which of the 128 experts in each MoE layer are contributing something distinct, rather than scoring them purely on individual performance.
  2. Select and group experts. Experts are grouped based on their scores, aiming to preserve a spread of different capabilities rather than just keeping the “best” individually-scored experts, which might overlap heavily in what they do.
  3. Merge each group into a dense layer. Each group of experts gets fused into a single standard feed-forward layer, collapsing the sparse, routed structure into one dense network.
  4. Distill from the parent. The resulting dense model is trained to match the outputs of the original MoE model.

According to the transcript’s account of the paper, Krafton tested 350 different configurations of this pipeline and found that the expert-scoring method mattered most to the final result. Their diversity-aware scoring approach outperformed a straightforward pruning baseline by 6.3 points on average across benchmarks, while also cutting training time by a factor of 1.6.

It’s worth noting that converting 128 experts down into a 1.5B dense model is a roughly 20x parameter reduction, which is a large jump for any distillation method. But because only 4.3 billion of the original 29 billion parameters activate per token, the effective compute gap between parent and child is smaller than the raw parameter counts suggest. The Nano model isn’t just a shrunken version of a 29B model, it’s closer to a distillation from a dynamically-reconfiguring 4B model that changes which experts fire on every token.

Rapidlybuild enterprise-grade, production-ready software.

Already in:✓ SOC 2✓ HIPAA✓ Audit log

The paper’s output isn’t the finished Nano model either. After the architectural conversion, Krafton ran roughly 3 trillion more tokens of training, followed by supervised fine-tuning, on-policy distillation from the Mini model, and reinforcement learning. The conversion technique gets you a strong starting point, not a finished product.

Why strip out knowledge instead of compressing everything evenly? #

A common theory in interpretability research is that a model’s factual knowledge lives largely in its feed-forward layers, while attention layers handle more general reasoning and in-context processing. In an MoE model, the feed-forward layers are the experts. When Krafton’s method merges and discards most experts during conversion, it’s plausible that a disproportionate share of memorized facts gets shed, while the attention mechanism (which carries over largely intact) preserves more of the model’s ability to work through a problem using whatever’s in its context window.

Krafton doesn’t make this claim directly in the paper, and the paper doesn’t test for it. But the benchmark pattern is suggestive. Raon-1 Mini, the parent model, scores last among five comparison models on MMLU Pro (a broad general-knowledge test), coming in at 76.5 against 85.9 for a Qwen 3 model in the 30B range. Yet it leads on instruction-following benchmarks and tops the artificial analysis omniscience benchmark, which specifically rewards a model for recognizing when it doesn’t know something rather than guessing.

That combination looks less like a weak model and more like a model that was trained to behave a specific way: don’t bluff, don’t try to recall obscure facts, work with what’s given. For a model meant to be injected with game-specific context at inference time rather than asked general trivia, that tradeoff makes sense.

What is this model actually for? #

Krafton builds games, not general-purpose assistants, and the company has already shipped AI-driven NPCs. Its recent life-simulation game included on-device characters referred to as Smart Zoy, and the company has also built an AI teammate feature for PUBG that talks to players and responds to in-game calls, a concept Nvidia demoed in early 2025 that has since gone into production.

Both use cases share the same technical constraints: the model has to run locally on the player’s machine, sharing GPU resources with the game itself, under tight latency budgets, and it needs to track a lot of transient context (inventory, map state, squad position, prior player instructions) rather than broad world knowledge. A smaller model that’s good at following instructions and using supplied context, but doesn’t waste parameters memorizing facts about world history or geography, fits that job description closely. Krafton’s own chief AI officer has described Raon-1 Nano as a better base model for the company’s edge AI applications, without confirming outright that it powers any specific shipped feature.

Is this relevant outside of game development? #

Nothing gets lost.

312 apps across 41 teams · all in one place

The constraints Krafton is optimizing for, low latency, limited memory, long sessions, heavy reliance on context rather than stored knowledge, are the same constraints facing anyone building a small local agent, a customer-facing kiosk bot, or an on-device assistant. The MoE-to-dense distillation technique itself is a general contribution to model compression research, independent of gaming. And because both Raon-1 Mini and Raon-1 Nano were trained from scratch by Krafton rather than fine-tuned from an existing open model, the technique is reproducible by other teams that have their own MoE model and want a dense, edge-deployable counterpart.

The models are released under the Apache 2.0 license. Krafton initially gated Raon-1 Mini under a more restrictive license and ungated it the following day after community feedback.

Frequently Asked Questions #

What is Raon-1 Nano?

Raon-1 Nano is a 1.5 billion parameter dense language model released by Krafton, the South Korean game studio behind PUBG. It was produced by converting a larger 29 billion parameter mixture-of-experts model, Raon-1 Mini, into a dense architecture and then distilling it further.

How is this different from normal model distillation?

Most distillation goes from a large dense model to a smaller dense model, or from a large MoE model to a smaller MoE model. Krafton’s method converts an MoE model’s architecture into a dense one by scoring, grouping, and merging experts into standard feed-forward layers, then distilling behavior from the original model.

Why would a game company build its own LLM instead of using an existing one?

Running cloud-hosted LLMs for every player interaction is expensive at scale. Krafton wants small models that run locally on the player’s own GPU for features like AI teammates and NPCs, and training its own model from scratch lets it control exactly what knowledge is included, which matters for a model that mainly needs to follow instructions and track game state rather than recall general facts.

Did Krafton confirm this model powers PUBG Ally or other NPC features?

Not explicitly. Krafton’s chief AI officer described Raon-1 Nano as a better base model for the company’s edge AI applications, but the company has not stated outright that any specific shipped feature runs on this exact model.

Is the MoE-to-dense technique useful outside of gaming?

Yes. The underlying method for converting a mixture-of-experts model into a dense model is a general model compression technique, and the paper describing it earned an oral presentation at NeurIPS this year. Anyone building small, edge-deployable language models for latency-sensitive, context-heavy tasks faces similar tradeoffs.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @krafton 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-krafton-turned-a…] indexed:0 read:9min 2026-11-18 · —