LatentMoE: NVIDIA’s Latent Mixture of Experts Explained Through Equations, Architecture, Code and…

NVIDIA published LatentMoE in January 2026, a mixture-of-experts architecture that performs expert arithmetic in a space four times narrower than the model's width and spends the saving on a larger expert pool. On a 95B-parameter transformer trained on 300 billion tokens, the latent layer raised the MMLU-Pro score from 29.26 to 34.91, a gain of 5.65 points, with the same active parameters and the same bytes of expert weights per token as the baseline; on a 73B hybrid Mamba-Transformer trained on a trillion tokens the gain was 4.57 points. NVIDIA shipped the design in Nemotron 3 Super in March with 512 experts of width 1,024 inside a model of width 4,096, followed by Nemotron 3 Ultra in June with the same design.

Every mixture-of-experts design since 2021 has been judged on the same number, how much accuracy you get per floating-point operation, which is another way of asking how well the GPU is being used. It’s a fair thing to measure. Sparsity is the one trick that separates parameter count from compute cost, so counting the result in FLOPs feels natural. Switch Transformer, GLaM, Mixtral, DeepSeekMoE and Qwen3 all make a version of the same claim, the quality of a much larger dense model at a fraction of the arithmetic. Then we serve one of these models, and we find that on the hardware we actually pay for, almost nothing about serving a mixture of experts is decided by arithmetic. It’s decided by bytes, which is to say by memory. The bytes of expert weights a token pulls out of memory, and the bytes it sends across the interconnect to reach its experts. In January 2026, NVIDIA published LatentMoE, an architecture built on that observation. It fits in one sentence. Do the expert arithmetic in a space four times narrower than the model’s, and spend the saving on a larger pool of experts. It shipped fast, too. Nemotron 3 Super, released in March, carries 512 experts of width 1,024 inside a model of width 4,096, and Nemotron 3 Ultra followed in June with the same design. The numbers set the scale. On a 95B-parameter transformer trained on 300 billion tokens, the latent layer lifts the MMLU-Pro score from 29.26 to 34.91, 5.65 points, with the same active parameters and the same bytes of expert weights per token as the baseline. On a 73B hybrid Mamba-Transformer model trained on a trillion tokens, the gain is 4.57 points. So in this article we explore and visualise the layer in order. First the dense feed-forward layer and the mixture of experts that replaced it, with the shapes, the equations and the code. Then the latent space, which is two projections and one number, the compression factor. Then the LatentMoE layer itself, with its code. Only after that do we go back and count the bytes, because the reason the experts had to move only really lands once you’ve seen where they moved to. A conclusion closes with what to take from the design and where its limits are. A note on the figures- Each one shows a transformation, the tensors going in, the operation, and the tensor coming out, labelled with the symbols the equations use and with the equation the figure draws set above it, in the notation of the paper. To keep the shapes concrete, every figure uses one running example, the 95B configuration NVIDIA trained, which is also the width Nemotron 3 Super ships at. That means a model dimension d of 4096, a latent dimension ℓ of 1024, an expert intermediate size m of 2688, and a baseline layer of N = 128 experts with K = 6 active. Scaling by α = 4 gives the latent layer N′ = 512 experts with K′ = 24 active, and Nemotron 3 Super uses K′ = 22. Every weight in this layer is shared by every token, so the figures draw one token, and only the dispatch figure draws several, because that’s the one place the token count matters for the workflow. A token is a vector of 4096 numbers, drawn as a column. The numbers inside the vectors and along the edges of the matrices are coordinate indices, 0 up to 4095, so you can follow one coordinate through a product. The one exception is the router figure, where the scores and the softmax weights are example values. In the text we write W down, W up and W r for the down-projection, up-projection and router weight matrices, and E i x; d for an expert that works at dimension d. Where a figure compares layers it labels them ℓ-MoE acc and ℓ-MoE eff, NVIDIA's names for the 24-of-512 layer K = 24 and the 6-of-512 layer K = 6 . Every byte count in this article is worked out from the configuration above. A transformer block has two halves. Attention mixes information between tokens, and the feed-forward layer works on each token alone. The feed-forward layer is two matrices with a nonlinearity between them. With x the token, W 1 and W 2 the two matrices and σ the nonlinearity, the layer computes The squared ReLU is the activation the 95B model uses. With a gated activation such as SwiGLU there’s a third matrix beside W 1, and everything below carries over with three matrices in place of two. In Figure 1, x is the input vector, dimension 4096. W 1 is the first weight matrix, 2688 rows of 4096, and row k of it dotted with x gives coordinate k of h, the hidden vector of dimension 2688. The squared ReLU is applied to each of those 2688 numbers. W 2 is the second weight matrix, 4096 rows of 2688, and it takes h back to an output vector of dimension 4096. Two matrices of 11.01 million parameters each, 22.0 million in all, and every one of them multiplies every token that passes through. A mixture of experts replaces that one feed-forward layer with N of them, the experts , and adds a router that decides which ones each token uses. Only the K experts with the highest router scores run for a given token. The total parameter count grows with N while the arithmetic per token stays flat. That’s the entire bargain, and it’s why the pattern has been the standard way to buy capacity for five years. If you’re not sure how a sparse feed-forward block is wired, my earlier articles cover it in the same style as this piece. The Architecture of Mistral’s Sparse Mixture of Experts https://pub.towardsai.net/the-architecture-of-mistrals-sparse-mixture-of-experts-s-%EF%B8%8F-e-89627636e1e3 covers the router, the top-k selection and the block as a whole. Revolutionizing AI with DeepSeekMoE https://pub.towardsai.net/revolutionizing-ai-with-deepseekmoe-fine-grained-expert-and-shared-expert-isolation-%EF%B8%8F-0b6e51155449 explains why finer experts and an isolated shared expert help, an argument this article comes back to. Breaking Down Mistral 7B https://medium.com/towards-artificial-intelligence/breaking-down-mistral-7b-3176f119f2eb covers the dense model the sparse one was built from. They’re two years old, and the mechanisms haven’t moved. Figure 2 The token arrives from the previous layer, passes through self-attention, and is added back to itself through the residual connection. From there it goes three ways at once. To the router, to the all-to-all dispatch that carries it to its experts, and to the shared expert SE. The router produces one score per expert, the small bar chart, and the dashed lines carry the two winning scores to the multipliers above E2 and E3. Those two experts run, their output vectors get scaled by their scores and combined, and the result meets the shared expert’s output and the residual connection. The router is the first of the two mechanisms in that picture, and it’s just a matrix-vector product followed by a softmax In Figure 3, W r is the router weight matrix, one 4096-wide row per expert, 128 rows in all. Row 3 dotted with x gives score 3 2.3 in the example , and the same matrix-vector product gives all 128 scores at once. The softmax turns those scores into weights that sum to 1. Then the top-K step keeps the 6 largest weights, K = 6 experts per token, together with the experts they belong to. In the example those are experts 127, 3, 41, 88, 17 and 64, in that order. The token vector itself hasn’t changed. The router has only decided where it goes and how much each expert’s output will count. The second mechanism is the experts themselves, and the layer’s output is their weighted sum plus the shared experts Each routed expert E i x; d has exactly the shape of the dense layer in Figure 1, two weight matrices of 2688 × 4096 and 4096 × 2688, and the notation x; d says it takes an input vector of dimension d and returns an output of dimension d. The S shared experts E j have the same shape but no router weight p i in front of them. Every token passes through them, which is what lets the model learn the knowledge every token needs in one place. Figure 4 is Equation 3 in blocks. The token x is copied at the dot. One copy goes up to the router, which produces the 128 scores and picks the 6 largest, and the dashed line drops that selection onto the strip of experts, where the six highlighted experts are the ones that will run. The other copy goes to those six experts and, along the bottom path, to the two shared experts, which run for every token. The weighted output vectors are added at the end into y, dimension 4096. Every expert in Figure 4 holds the 22.0 million parameters of the dense layer Figure 1 , so the routed pool is 128 × 22.0 million, 2.82 billion parameters in one layer, and a token touches 6 × 22.0 million of them, 132 million. Hold on to both numbers. The first is what the layer knows, the second is what one token pays for, and LatentMoE is about the ratio between them. The code below is the example MoE layer. We pass the expert width in as an argument, so the same Expert class serves the routed experts and the shared experts here, and the latent experts later. The routing decision comes first, the K weights are normalised to sum to 1 the values in Figure 3 are the raw softmax weights, before that step , and the dispatch uses a loop over slots so it reads the way Equation 3 does. python import torch, torch.nn as nn, torch.nn.functional as Fclass Expert nn.Module : """Squared-ReLU FFN of width width and intermediate size m.""" def init self, width, m : super . init self.up = nn.Linear width, m, bias=False self.down = nn.Linear m, width, bias=False def forward self, x : return self.down F.relu self.up x 2 class MoE nn.Module : """y = sum {i in top-k} p i E i x + sum j E j x , every expert at width d.""" def init self, d=4096, m=2688, n experts=128, top k=6, n shared=2 : super . init self.top k = top k self.router = nn.Linear d, n experts, bias=False one row per expert self.experts = nn.ModuleList Expert d, m for in range n experts self.shared = nn.ModuleList Expert d, m for in range n shared def forward self, x : x: tokens, d scores = self.router x one score per expert w, idx = torch.topk scores.softmax -1 , self.top k, dim=-1 w = w / w.sum -1, keepdim=True the K weights sum to 1 y = torch.zeros like x for slot in range self.top k : e ids, gate = idx :, slot , w :, slot:slot + 1 for e in e ids.unique : sel = e ids == e y sel += gate sel self.experts e x sel for sh in self.shared: every token, no router y = y + sh x return y Two things in that code are the ones LatentMoE will change. The experts are built at dimension d, so each holds two d × m weight matrices, and the token is dispatched to them at dimension d, so each of the K trips carries a vector of d numbers. Everything else, the router, the softmax, the shared experts and the sum, stays as it is. The two numbers from the MoE section, 2.82 billion held and 132 million activated, are both proportional to the model dimension d. So is the dimension of the vector a token travels as when it’s dispatched to its experts. LatentMoE starts from the observation that d is just the dimension the experts happen to be written in, and asks whether they need to be that wide at all. The question should feel familiar, because it’s the one DeepSeek asked about the key-value cache in Multi-Head Latent Attention. There, the answer was to compress keys and values into a latent vector a fraction of the model dimension, cache the small object, and project back up when it was needed. LatentMoE makes the same move one block to the right. Compress the token vector into a latent space, do the expensive and repeated work there, and project back. In attention the expensive repeated thing was storing the cache. Here it’s shipping the token to hundreds of expert weight matrices and multiplying it against them. Two shared linear maps do the compressing and the restoring. W down, an ℓ × d weight matrix, takes the token from the model dimension d to a latent dimension ℓ, and W up, a d × ℓ weight matrix, brings the result back. The ratio between the two dimensions is the compression factor and α = 4 throughout this article, so ℓ is 1024 when d is 4096. Why 4 and not 8 is a question for the end. If you have seen an autoencoder drawn, you already know what this looks like. Figure 5 is the encoder, bottleneck, decoder shape of a variational autoencoder, and the analogy is worth a moment because it’s where the word latent comes from. On the left, x enters at dimension 4096. The first wedge is W down, and it squeezes the token down to z, a vector of dimension 1024. Everything expensive happens inside the box in the middle, at that dimension. The second wedge is W up, and it widens the experts' summed output u back to 4096. Two differences from a real autoencoder. Nothing is sampled, both wedges are plain linear maps with no nonlinearity, and the bottleneck isn’t a place to store a compressed copy of the token. It’s a place to do work on it cheaply. The down-projection is a single matrix-vector product, Figure 6 upper half , x is the input token vector, dimension 4096, and W down is the down-projection weight matrix, 1024 rows of 4096. The product of the two is z, the latent vector of dimension 1024 that the experts take as their input. The highlighted row shows one of those products, row 1 of W down against all of x, giving z 1. Compression here means a learned linear mixture of all 4096 coordinates, not a choice of which coordinates to keep. The lower half is the reverse step at the end of the layer. u is the experts' summed output, dimension 1024, and W up is the up-projection weight matrix, 4096 rows of 1024, so W up · u is a vector of dimension 4096 again and can be added to whatever else runs at the full dimension The weights p′ i are the router's, exactly as in Equation 2. The prime just marks that the latent layer's router scores a larger pool of experts, which we get to in the next section. W down holds 4096 × 1024 = 4.19 million parameters, and there's exactly one of it in the layer, shared by every token and every expert. W up is another 4.19 million. Together the two are 8.4 million parameters against a routed expert pool of 2.82 billion in the same layer, an overhead of 0.3 per cent. That's the figure everything the architecture gains has to be weighed against. With the two projections in hand, the latent layer is the standard layer Figure 2 with the routed path moved inside them. Between the Standard MoE Figure 2 and LatentMoE Figure 7 , three things have changed. The token now passes through the latent down-projection before the all-to-all dispatch, so it travels to its experts at the latent dimension. The expert pool has grown from four to eight and the selection from two to four, the figure’s own example at α = 2. And the latent up-projection sits after the all-to-all combine, so the experts’ summed output is restored to the model dimension before it meets the shared expert and the residual connection. Two things haven’t changed. The router still reads the token before the down-projection, and the shared expert SE still reads it at the model dimension. Figure 8 is the same LatentMoE layer with the shapes on it. The token x is copied into three paths. The top path is the router, which reads x at its full dimension of 4096 and returns 512 scores, and the yellow block keeps the 24 largest, the top 24 experts for that token. The middle path is the latent one. W down takes the token to z, the latent vector of dimension 1024, and the selected experts work on z inside the shaded band. Their weighted outputs are summed at dimension 1024, and W up brings the sum back to 4096. The bottom path is the shared experts, which read x at the full dimension and never enter the projections. The two outputs are added, and the output y has dimension 4096. Everything that made the standard MoE layer expensive is inside the latent band. Everything outside it, the router and the shared experts, is the part NVIDIA judged too cheap to touch. The router did grow, from 128 rows to 512, and unlike the expert pool it’s read in full for every token, but 512 × 4096 is 2.1 million parameters against 2.82 billion. Keeping it at the full dimension means the compression never touches the quality of the routing decision. Only the payload is compressed, never the choice of where to send it. The experts inside the band are where the saving comes from, and there’s one detail in them that’s easy to read past. Figure 9 A standard expert on the left is the dense feed-forward layer of Figure 1, a 2688 × 4096 weight matrix, a squared ReLU, and a 4096 × 2688 weight matrix back, 22.0 million parameters. The latent expert on the right does the same to the 1024-dimensional z, and its two weight matrices are 2688 × 1024 and 1024 × 2688, 2.75 million each, 5.5 million per expert. A quarter of the parameters, and a quarter of the bytes to read. The hidden vector h has dimension 2688 on both sides. Only the input and output dimensions changed. A classical result on one-hidden-layer networks says the capacity a token gets from the layer is proportional to the total width of the nonlinearity it passes through. In a mixture of experts that's the number of active experts times their intermediate dimension so cutting m to save bytes would cut capacity directly, and so would cutting K. Compressing the input and output dimensions costs nothing on that count. The latent expert holds a quarter of the parameters and its nonlinearity is exactly as wide as before. Now the whole layer in one equation. Compressing the routed path by α makes each expert α times cheaper, and LatentMoE spends that saving on more of them. The expert pool grows to N′ = αN and the number of active experts to K′ = αK, so the running example goes from 6 of 128 to 24 of 512 experts, the configuration from NVIDIA’s paper. Let’s compare it against the standard MoE Equation 3 . Only three things moved. W down went inside the experts' argument, W up wrapped their sum, and the top-k now picks 24 out of 512. The token activates four times as many experts, each of them four times narrower, so active parameters, arithmetic and expert bytes read all land back where the baseline was. The all-to-all volume does too. A token makes one round trip per selected expert, so the traffic is K′ times the latent dimension, and 24 × 1024 is 6 × 4096. The layer moves the baseline’s bytes. What it gets for them is the question. There’s a second way to spend the saving, which keeps K at 6 and banks the reduction as lower inference cost instead. It’s a different trade, and we come back to it when we count the bytes. The question is why 24 narrow experts should beat 6 wide ones when the arithmetic is the same. The answer is a counting argument, and it’s the argument the DeepSeekMoE article linked above was about. A layer with N = 128 and K = 6 can route a token to any of C 128, 6 subsets of experts, about 5.4 billion distinct combinations. That number is the real capacity of a sparse layer, because it’s the number of different committees of specialists the router can convene. Scaling both N and K by α moves it according to C 512, 24 is about 10^41 against C 128, 6 at about 10^9.7. That's thirty-one orders of magnitude more ways to combine experts at the same parameter count and the same operation count. The dashed line in Figure 10 is the bound Equation 9 promises, C 128, 6 to the fourth power, or 10^38.9, and the real count clears it by two more orders. DeepSeekMoE arrived at the same place by splitting experts into finer ones. LatentMoE arrives there by shrinking them. One constraint on all of this is easy to lose. Compressing to ℓ without also scaling the expert count degrades validation loss noticeably, and compressing and scaling together recovers it. Compression on its own is a regression, and compression plus reinvestment is the architecture, so they’re one move rather than two. The same ablations sweep the compression ratio on the 16B model, out to a trillion tokens. At α = 4 the validation loss sits on top of the baseline’s, and at α = 8 it ends a few thousandths higher, marginally worse but nowhere near a cliff. So α = 4 was adopted as the ratio with no cost at all, and that choice was re-verified at 95B. It’s the ratio in this article’s examples we have used. The code module below builds the routed path in latent space and the shared path at the full dimension, reusing the Expert class from the standard layer above. Compared with the MoE class, the two projections and the two scaled counts are new, and the experts are built at dimension ell instead of d. The routing decision is still taken from x, before any compression. The variant argument is there for the other way of spending the saving, mentioned above, and defaults to the layer this section described. php class LatentMoE nn.Module : """variant='acc' - N'=aN, K'=aK same cost, more accuracy variant='eff' - N'=aN, K'=K same accuracy, less cost """ def init self, d=4096, ell=1024, m=2688, n experts=128, top k=6, n shared=2, variant='acc' : super . init alpha = d // ell self.n prime = alpha n experts self.k prime = alpha top k if variant == 'acc' else top k self.w down = nn.Linear d, ell, bias=False d - l, shared self.w up = nn.Linear ell, d, bias=False l - d, shared self.router = nn.Linear d, self.n prime, bias=False routes in FULL d self.experts = nn.ModuleList Expert ell, m for in range self.n prime self.shared = nn.ModuleList Expert d, m for in range n shared def forward self, x : x: tokens, d scores = self.router x decision from x, not z w, idx = torch.topk scores.softmax -1 , self.k prime, dim=-1 w = w / w.sum -1, keepdim=True z = self.w down x the only d-wide op out = torch.zeros like z for slot in range self.k prime : e ids, gate = idx :, slot , w :, slot:slot + 1 for e in e ids.unique : sel = e ids == e out sel += gate sel self.experts e z sel y = self.w up out for sh in self.shared: shared path stays at d y = y + sh x return y The baseline holds 2,862.6 million parameters per layer and the latent layer 2,871.0 million, and the 8.4 million between them is exactly the two projections, 0.3 per cent. Per token, the baseline activates 176.2 million parameters counting the shared experts and the latent layer 184.5 million, again the projections. The model-level counts put the two within half a per cent of each other. In exchange the latent layer has 10^31 times more expert combinations to choose from. Everything above we have seen is what the layer does. This section is about why it was worth doing, and the answer is two byte counts. Two serving regimes matter in practice, and neither is limited by arithmetic. Each is limited by a different kind of byte movement, and we’ll take them one at a time. The interactive case first. One chat session, an effective batch size of one, one token at a time. The router Figure 3 selects 6 experts out of 128 for that token, and each expert’s two weight matrices have to come out of high-bandwidth memory into the compute units, to be used exactly once, in a single matrix-vector product. Each stack Figure 11 is one of the two weight matrices of the six selected experts, 22.0 million weights per expert, 22 MB in FP8, so six experts are 132 MB read from memory for one token in one layer. At 8 TB/s that read takes 16.5 microseconds, and for almost all of it the compute units sit idle, because what comes back is one vector of dimension 4096, 8 KB. The way to make this precise is arithmetic intensity , the floating-point operations performed per byte moved. For one expert that serves t exp tokens in a step, counting the multiply-adds against the weights plus the activations in and out A GB200 can do about 10 PFLOP/s of FP4 arithmetic but can only pull about 8 TB/s out of memory, so the chip’s balance point, the ridge of the roofline, sits at 1250 operations per byte. An expert that does fewer operations than that for every byte of weight it reads is waiting on memory. One that does more is waiting on compute. What decides which side an expert lands on is t exp, the number of tokens it serves in one step. Each weight is read from memory once and then used 2 · t exp times, so the more tokens share the read, the more operations each byte pays for. On the weights alone an expert needs about 625 tokens per step to reach the ridge, and more once the activations flowing in and out are counted. An interactive deployment doesn’t come close. A handful of concurrent requests spread over 128 experts leaves each expert single-digit tokens per step, nearly two orders of magnitude below the ridge. A couple of hundred tokens per expert, where latency-critical serving usually sits, is still short of it. So at interactive batch sizes the expert layer is memory-bound, and the time a token spends in it is set by the bytes of expert weight it pulls out of memory, two d × m weight matrices for each of its K experts. In this regime what matters is accuracy per parameter, not accuracy per operation. Now increase the batch. Every expert has enough tokens to share the read, the memory problem dissolves, and a different one takes its place. Under expert parallelism the experts are spread across GPUs, so each token vector has to be sent to the GPUs holding its selected experts and the results sent back. That exchange is an all-to-all collective, run twice per expert layer, and the data it moves depends on how many numbers each token carries when it travels. In Figure 12, five tokens leave one GPU for the GPUs that hold their experts, one round trip per selected expert. On the left each token travels as 4096 numbers, 2 KB out in FP4 and 8 KB back in BF16. On the right, inside the latent layer, it travels as 1024 numbers, a quarter of each. The awkward thing about this second bottleneck is that a bigger batch doesn’t help. Both the traffic and the compute grow with the number of tokens, so their ratio stays put. Over NVLink high-speed communication tech allows multiple GPUs to communicate directly with each other, from NVIDIA the expert layer spends several times longer moving tokens between GPUs than computing on them, at any batch size, and the only way down is fewer bytes per token, which means fewer numbers per token, which means a smaller dimension. So both variants come down to bytes, and both byte counts are proportional to d. The active expert count K and the intermediate dimension m are in there too, K in both counts and m in the weights, but Equation 7 says cutting either one cuts the layer’s capacity. That leaves d, the dimension the experts happen to be written in, and that is the dimension LatentMoE shrinks. Which brings us to the second way of spending the compression. Shrinking the routed path by α makes each expert α times cheaper. The layer in the previous section spent that on four times as many active experts, 24 instead of 6, and kept the bytes where they were. The other option keeps the top-k at 6, still grows the pool to 512, and pockets the saving. It’s the layer of Equation 8 with K′ = K in place of K′ = αK . Total parameters stay roughly the same, active parameters per token fall by a factor of α, and the result is the baseline’s accuracy at a fraction of the inference cost. Figure 13 puts the three layers side by side. All three hold the same routed pool, 2.82 billion parameters, since 512 experts of 5.5 million is 128 experts of 22.0 million. The 24-of-512 layer reads 132 MB of expert weight per token and sends 60 KB over the interconnect, the baseline’s numbers exactly, so the same bytes move, split into four times as many pieces, and what the split buys is the routing combinations. Nemotron 3 Super, at K′ = 22, sits just under it at 121 MB and 55 KB. The 6-of-512 layer reads 33 MB and sends 15 KB, a quarter on both counts, at 85.5 million active parameters per token against the baseline’s 176.2 million. Across the 32 expert layers of the 95B model that’s 1.1 GB of expert weights per token against 4.2 GB, 0.13 milliseconds of streaming at 8 TB/s against 0.53 for every token generated. So the two layers deliver different things. The 24-of-512 layer doesn’t make decoding cheaper, it makes the same decoding cost more accurate. The 6-of-512 layer is the one to reach for when the bandwidth bill is what hurts. What the extra combinations buy shows up in the benchmarks. On the 95B transformer at 300 billion tokens, the running example of this article, the comparison reads as follows. The 24-of-512 ℓ-MoE acc layer wins at equal cost, 5.65 points on MMLU-Pro on the 95B transformer, and on the hybrid model at a trillion tokens 4.57 on MMLU-Pro, 3.19 on Code and 2.01 on MMLU, at 8.02 billion active parameters against the baseline’s 8.09 billion. The 6-of-512 layer nearly holds the line at 73 per cent of the active parameters, ahead on MMLU-Pro, MMLU and Code and behind on Math and Commonsense. The baseline’s hyperparameters were reused unchanged for every LatentMoE run, so the gaps are if anything conservative. The layer isn’t faster in isolation. It removes the need to buy accuracy with parameters that then have to be streamed through memory on every token. Nemotron 3 Super, published on 10 March 2026, is the layer at production scale. 120.6 billion parameters with 12.7 billion active, 88 layers of Mamba-2 and mixture-of-experts blocks with periodic global attention, and an expert layer that is the running example with one change, d = 4096, ℓ = 1024, 512 routed experts of intermediate dimension 2688 with a top-k of 22, plus a shared expert of intermediate dimension 5376, pretrained in NVFP4 on 25 trillion tokens with a context length of up to one million. NVIDIA reports up to 2.2 and 7.5 times higher inference throughput than GPT-OSS-120B and Qwen3.5–122B at comparable or better accuracy. That bundles LatentMoE with the Mamba hybrid, NVFP4 pretraining and multi-token-prediction decoding, so it isn’t a clean measurement of this one change, but it is the workload the change was designed to serve. Nemotron 3 Ultra followed on 4 June 2026, at 550 billion parameters with 55 billion active, on the same architecture. The habit to take away is small and general. Before believing a floating-point-operation count about a layer, work out its arithmetic intensity. Accuracy per operation is what mixture-of-experts research optimises, accuracy per byte moved is what a serving bill optimises, and at interactive batch sizes the two diverge by two orders of magnitude. If you’re designing a sparse layer, the model dimension is the first variable to look at, because it sets the expert weight bytes and the width every token travels on the interconnect at the same time. Reduce it, and then decide on purpose what to do with the saving. Spend it on a larger expert pool when you’re training something new and want the accuracy, or bank it when the deployment is bandwidth-bound today. The reinvestment isn’t optional in either case, since compressing without scaling the pool loses accuracy instead of saving cost. Keep the router at the full dimension, and keep m where it was. The limitation to carry forward is the compression ratio. α = 4 was validated at 16B and 95B and adopted, and α = 8 was run at 16B and came out only marginally worse, so there’s no cliff at 8, at least at that scale. The open question is why 4 and not 8, and whether the small gap at 16B widens at 95B or over a longer token horizon. It’s also worth noticing what this design and Manifold-Constrained Hyper-Connections , the subject of the previous article in this series, have in common, because it reads as the characteristic move of 2026. Both take a component nobody was touching, ask what it costs in bytes rather than in operations, and change its shape rather than its function. NVIDIA calls the two complementary and says they could be stacked, and leaves that experiment to future work. mHC: Manifold-Constrained Hyper-Connections. Explained Through Equations, Architecture, Code https://pub.towardsai.net/mhc-manifold-constrained-hyper-connections-explained-through-equations-architecture-code-eb516948d166 A personal note to close. This is the second article since my return to writing after a gap of two years, and it arrives while I’m exploring new roles, in India or remote, on the training and applied side of the AI. If the piece was useful and you know a team that would fit, a referral or a message would mean a lot. 🔗 LinkedIn: jaiganesan-n 🌏❤️ 👏 If you got something out of this, a few claps help more people find it. LatentMoE: NVIDIA’s Latent Mixture of Experts Explained Through Equations, Architecture, Code and… https://pub.towardsai.net/latentmoe-nvidias-latent-mixture-of-experts-explained-through-equations-architecture-code-and-7671f8a47e27 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.