“Sparse” sounds like one switch. With Mistral Large 4, it helps to draw two boxes.
One chooses which expert networks process a token. The other chooses which context positions receive attention. Those boxes have different inputs, different outputs and different jobs. Counting selected experts tells me almost nothing about how many earlier positions a query can attend to.
I built a tiny executable illustration to keep those choices separate. It selects experts 1 and 3 on one axis, and positions 1, 3 and 4 on the other. The numbers below are deliberately authored toy scores, not activations extracted from Mistral. 🧭
Original illustration. Each panel has its own score vector; equal index numbers across panels have no shared meaning.
Think of the expert list as a directory of processing functions. The token's representation reaches a router, and the router chooses functions to apply. Their outputs are combined into the token's next representation.
The context list is a directory of positions in a sequence. An attention mechanism decides which available positions contribute information to the current query.
An expert index is therefore an address in a collection of networks. A position index is an address in a sequence. Expert 3 is not “the network for the third word.” Position 3 is not “the third expert.” That accidental association is particularly easy to make when both explanations use the phrase top-k.
Top-k only says: choose the k highest-scoring candidates. It does not tell me what the candidates are.
The Mistral Large 4 launch describes both expert sparsity and indexed sparse attention.[1] To make the distinction concrete, I inspected its model configuration at an immutable revision rather than relying on the word “sparse” in a launch headline.[2]
The text configuration contains these fields:
{
"n_routed_experts": 128,
"num_experts_per_tok": 4,
"n_shared_experts": 1,
"index_topk": 2048,
"first_k_dense_replace": 1
}
The first three describe the expert side: four routed experts out of 128, alongside a shared expert. index_topk belongs to position selection. The initial dense-layer setting is another reason to avoid describing every layer as an identical sparse block.[2]
That is the source-linked architecture claim. It is enough for this explainer; I have not loaded the full model, measured its speed or verified its quality.
Now for a version small enough to inspect without a data center.
This function returns indices, sorted by descending score. A secondary index sort makes ties deterministic.
def top_k(scores, k):
return sorted(range(len(scores)), key=lambda i: (-scores[i], i))[:k]
expert_scores = [0.2, 2.0, -0.5, 1.0]
position_scores = [0.1, 1.3, -0.2, 0.8, 0.4, -0.6]
experts = top_k(expert_scores, 2)
positions = top_k(position_scores, 3)
print(experts, positions)
The executed result:
[1, 3] [1, 3, 4]
The expert selector returns two function addresses. The position selector returns three sequence addresses. I can change every position score and keep the expert result unchanged because this toy gives them independent inputs.
That independence is a teaching device. In a real model, both mechanisms operate on learned representations inside the same network. I am separating the meanings of their outputs, not claiming the trained mechanisms cannot influence each other.
The complete runnable illustration uses only Python's standard library. Save it as toy.py and run python3 toy.py.
There is a second distinction hiding behind the first one. Choosing candidates does not yet tell me their contribution weights.
In the toy, I normalize the selected expert scores with a softmax. For scores 2 and 1, subtracting the maximum before exponentiation gives weights proportional to exp(0) and exp(-1).
import math
def selected_softmax(scores, selected):
peak = max(scores[i] for i in selected)
weights = [math.exp(scores[i] - peak) for i in selected]
total = sum(weights)
return [w / total for w in weights]
The resulting weights are approximately 0.7311 and 0.2689, summing to one. If the selected expert outputs are the toy scalars 20 and 40, their weighted sum is 25.3788.
Those scalars make the arithmetic visible. Real expert outputs are vectors, and this is not a reimplementation of Mistral's router, shared expert, indexer or attention computation. In particular, the position-ranking scores in my figure are not attention probabilities. Selecting a context subset and computing attention over it are separate steps.
The script checks known selected indices, normalization and the weighted-sum result. It also changes the position scores and checks that the unchanged expert input still produces the same expert selection. These checks establish the toy's behavior, not model quality.
Four routed experts out of 128 does not establish a 32× end-to-end speedup. That fraction leaves out the shared expert, attention, dense components, routing, memory movement and serving overhead. It also says nothing about hardware utilization.
Likewise, the attention index limit does not mean a model stores only that many context tokens. A selection bound and a cache-storage requirement answer different questions. And a large context limit does not guarantee successful retrieval of every fact within that context.
My practical reading order is now simple: identify the candidate set, find the selection rule, then ask how selected values are combined. Only after that does a performance claim become interpretable.
The useful question to take into another architecture diagram is: sparse over what? Functions, positions, weights and stored states are different things. The diagram gets much easier once every arrow has an address type.
Figure and toy code are original. Hand-written scores illustrate two selection axes; no full-model inference or benchmark is presented.
AI was used to draft the article.