{"slug": "mistral-large-4-makes-two-different-sparse-choices", "title": "Mistral Large 4 makes two different sparse choices", "summary": "A developer published a runnable Python illustration separating the two distinct forms of sparsity in Mistral Large 4: mixture-of-experts routing, which selects which expert networks process a token, and indexed sparse attention, which selects which earlier sequence positions a query can attend to. Inspecting the model configuration at an immutable revision, the developer found 128 routed experts with 4 selected per token, 1 shared expert, an index_topk of 2048 for position selection, and a first_k_dense_replace of 1, and noted the toy scores are authored rather than extracted from the model. The writeup stresses that top-k only describes picking the k highest-scoring candidates, not what those candidates are, and that the two selectors have independent inputs and outputs.", "body_md": "“Sparse” sounds like one switch. With Mistral Large 4, it helps to draw two boxes.\n\nOne chooses **which expert networks process a token**. The other chooses **which context positions receive attention**. Those boxes have different inputs, different outputs and different jobs. Counting selected experts tells me almost nothing about how many earlier positions a query can attend to.\n\nI built a tiny executable illustration to keep those choices separate. It selects experts 1 and 3 on one axis, and positions 1, 3 and 4 on the other. The numbers below are deliberately authored toy scores, not activations extracted from Mistral. 🧭\n\n*Original illustration. Each panel has its own score vector; equal index numbers across panels have no shared meaning.*\n\nThink of the expert list as a directory of processing functions. The token's representation reaches a router, and the router chooses functions to apply. Their outputs are combined into the token's next representation.\n\nThe context list is a directory of positions in a sequence. An attention mechanism decides which available positions contribute information to the current query.\n\nAn expert index is therefore an address in a collection of networks. A position index is an address in a sequence. Expert 3 is not “the network for the third word.” Position 3 is not “the third expert.” That accidental association is particularly easy to make when both explanations use the phrase *top-k*.\n\nTop-k only says: choose the k highest-scoring candidates. It does not tell me what the candidates are.\n\nThe Mistral Large 4 launch describes both expert sparsity and indexed sparse attention.[1] To make the distinction concrete, I inspected its model configuration at an immutable revision rather than relying on the word “sparse” in a launch headline.[2]\n\nThe text configuration contains these fields:\n\n```\n{\n  \"n_routed_experts\": 128,\n  \"num_experts_per_tok\": 4,\n  \"n_shared_experts\": 1,\n  \"index_topk\": 2048,\n  \"first_k_dense_replace\": 1\n}\n```\n\nThe first three describe the expert side: four routed experts out of 128, alongside a shared expert. `index_topk` belongs to position selection. The initial dense-layer setting is another reason to avoid describing every layer as an identical sparse block.[2]\n\nThat is the source-linked architecture claim. It is enough for this explainer; I have not loaded the full model, measured its speed or verified its quality.\n\nNow for a version small enough to inspect without a data center.\n\nThis function returns indices, sorted by descending score. A secondary index sort makes ties deterministic.\n\n``` python\ndef top_k(scores, k):\n    return sorted(range(len(scores)), key=lambda i: (-scores[i], i))[:k]\n\nexpert_scores = [0.2, 2.0, -0.5, 1.0]\nposition_scores = [0.1, 1.3, -0.2, 0.8, 0.4, -0.6]\n\nexperts = top_k(expert_scores, 2)\npositions = top_k(position_scores, 3)\nprint(experts, positions)\n```\n\nThe executed result:\n\n```\n[1, 3] [1, 3, 4]\n```\n\nThe expert selector returns two function addresses. The position selector returns three sequence addresses. I can change every position score and keep the expert result unchanged because this toy gives them independent inputs.\n\nThat independence is a teaching device. In a real model, both mechanisms operate on learned representations inside the same network. I am separating the meanings of their outputs, not claiming the trained mechanisms cannot influence each other.\n\nThe complete [runnable illustration](https://panditshivam.github.io/media/mistral-two-sparsities/toy.py) uses only Python's standard library. Save it as `toy.py` and run `python3 toy.py`.\n\nThere is a second distinction hiding behind the first one. Choosing candidates does not yet tell me their contribution weights.\n\nIn the toy, I normalize the selected expert scores with a softmax. For scores 2 and 1, subtracting the maximum before exponentiation gives weights proportional to `exp(0)` and `exp(-1)`.\n\n``` python\nimport math\n\ndef selected_softmax(scores, selected):\n    peak = max(scores[i] for i in selected)\n    weights = [math.exp(scores[i] - peak) for i in selected]\n    total = sum(weights)\n    return [w / total for w in weights]\n```\n\nThe resulting weights are approximately **0.7311 and 0.2689**, summing to one. If the selected expert outputs are the toy scalars 20 and 40, their weighted sum is **25.3788**.\n\nThose scalars make the arithmetic visible. Real expert outputs are vectors, and this is not a reimplementation of Mistral's router, shared expert, indexer or attention computation. In particular, the position-ranking scores in my figure are not attention probabilities. Selecting a context subset and computing attention over it are separate steps.\n\nThe script checks known selected indices, normalization and the weighted-sum result. It also changes the position scores and checks that the unchanged expert input still produces the same expert selection. These checks establish the toy's behavior, not model quality.\n\nFour routed experts out of 128 does **not** establish a 32× end-to-end speedup. That fraction leaves out the shared expert, attention, dense components, routing, memory movement and serving overhead. It also says nothing about hardware utilization.\n\nLikewise, the attention index limit does not mean a model stores only that many context tokens. A selection bound and a cache-storage requirement answer different questions. And a large context limit does not guarantee successful retrieval of every fact within that context.\n\nMy practical reading order is now simple: identify the candidate set, find the selection rule, then ask how selected values are combined. Only after that does a performance claim become interpretable.\n\nThe useful question to take into another architecture diagram is: **sparse over what?** Functions, positions, weights and stored states are different things. The diagram gets much easier once every arrow has an address type.\n\nFigure and toy code are original. Hand-written scores illustrate two selection axes; no full-model inference or benchmark is presented.\n\nAI was used to draft the article.", "url": "https://wpnews.pro/news/mistral-large-4-makes-two-different-sparse-choices", "canonical_source": "https://dev.to/panditshivam/mistral-large-4-makes-two-different-sparse-choices-40ce", "published_at": "2026-10-30 14:44:11+00:00", "updated_at": "2026-10-30 14:51:17.732357+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence"], "entities": ["Mistral", "Mistral Large 4"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/mistral-large-4-makes-two-different-sparse-choices", "markdown": "https://wpnews.pro/news/mistral-large-4-makes-two-different-sparse-choices.md", "text": "https://wpnews.pro/news/mistral-large-4-makes-two-different-sparse-choices.txt", "jsonld": "https://wpnews.pro/news/mistral-large-4-makes-two-different-sparse-choices.jsonld"}}