{"slug": "inkling-a-new-open-weight-975b-moe-with-a-few-surprises", "title": "Inkling: A New Open-Weight 975B Moe with a Few Surprises", "summary": "Thinking Machines Lab released Inkling, an open-weight 975B-parameter sparse Mixture-of-Experts model with 41B active parameters and a 1M-token context window. The model outperforms GLM-5.2 on IFBench and SimpleQA Verified but trails on reasoning and coding-agent benchmarks, and features unusual architectural choices including small convolution layers, an additional RMSNorm after the embedding layer, and a learned input-dependent relative-position bias instead of RoPE.", "body_md": "# Inkling: A New Open-Weight 975B MoE with a Few Surprises\n\nInteresting surprise open-weight LLM drop from Thinking Machines Lab yesterday. Their nearly 1T-parameter [Inkling model](https://thinkingmachines.ai/news/introducing-inkling/) looks pretty solid on the reported benchmarks.\n\nCompared with GLM-5.2, Inkling performs better on evaluations such as IFBench (79.8% versus 73.3%) and SimpleQA Verified (43.9% versus 38.1%). It performs worse on several reasoning and coding-agent benchmarks, including HLE without tools (29.7% versus 40.1%), SWE-Bench Pro Public (54.3% versus 62.1%), and Terminal-Bench 2.1 (63.8% versus 82.7%).\n\nThese are release-time numbers, and some rows combine externally reported values with internal harness results. I would therefore not over-interpret small differences. Overall, Inkling looks like a good all-rounder that is intended for further fine-tuning and specialization. That direction makes sense given that Thinking Machines Lab also develops Tinker, its model customization and fine-tuning platform.\n\nArchitecture-wise, Inkling is a 975B-parameter sparse Mixture-of-Experts model with 41B active parameters and a [context window](/glossary/#context-length) of up to 1M tokens.\n\nSome side-by-side observations:\n\n- Inkling has about 231B more total parameters than GLM-5.2 (975B versus 744B), although their active footprints are almost identical at 41B and 40B parameters.\n- Inkling is less sparse than Kimi K2.5. It activates 4.2% of its parameters per token, compared with 3.2% for Kimi K2.5 (41B of 975B versus 32B of 1T).\n- Inkling uses a regular Transformer decoder rather than the hybrid Mamba-Transformer approach used by Nemotron 3 Ultra.\n\nI am curious about token throughput. The larger active footprint than Kimi K2.5 and the use of conventional GQA rather than [MLA](/glossary/#mla) or a recurrent hybrid stack suggest that raw decoding speed may not be Inkling’s main advantage. However, throughput also depends heavily on quantization, expert parallelism, attention kernels, batching, and hardware. I have not seen directly comparable provider measurements yet.\n\nThe overall design follows the recent large-[MoE](/glossary/#moe) trend, but the architecture has a few interesting surprises:\n\n-\n**Small convolution layers in several places.** Each decoder layer applies short kernel-4 convolutions after the key and value projections and on the attention and MLP branch outputs. My intuition is that these provide cheap local token mixing and an explicit short-range inductive bias alongside attention. -\n**An additional RMSNorm directly after the token** This is separate from the pre-attention[embedding layer](/glossary/#token-embeddings).[RMSNorm](/glossary/#rmsnorm)inside every transformer block. At first glance, it looks almost redundant, but it is explicitly enabled in the[configuration](https://huggingface.co/thinkingmachines/Inkling/blob/main/config.json)and present in the[Transformers implementation](https://github.com/huggingface/transformers/blob/main/src/transformers/models/inkling/modeling_inkling.py). Whether it materially helps would require an ablation. -\n**A learned, input-dependent relative-position bias instead of**[RoPE](/glossary/#rope).[Thinking Machines says](https://thinkingmachines.ai/news/introducing-inkling/#architecture)that the relative-position approach “performs better and extrapolates better to longer sequences” than RoPE.\n\nRegarding the last point, my intuition is that the sliding-window-heavy architecture helps. Of the 66 decoder layers, 55 use local attention with a small 512-token window. A learned relative-position bias may provide enough [positional information](/glossary/#positional-encoding) within these windows.\n\nIn the 11 global layers, the released implementation applies the learned bias only over the preceding 1,024 tokens. Attention beyond that range is effectively content-based with respect to this positional bias. This is somewhat similar to the intuition behind [NoPE](/glossary/#nope), which other architectures use in selected global-attention layers.\n\nOverall, Inkling is an interesting variation on the DeepSeek-V3-style MoE recipe and a solid release. Some may find it underwhelming because it does not lead every benchmark. I find the broad and mixed benchmark profile refreshingly honest. It may also indicate that the model is less benchmark-specialized than several recent releases.\n\nAnd, as always, it is good to see another strong open-weight [base model](/glossary/#base-model) available for fine-tuning and independent study.\n\n## Read Next\n\n[200,000 Subscribers Short note celebrating Ahead of AI reaching 200,000 subscribers.](/blog/2026/ahead-of-ai-reached-200000-subscribers.html)\n\n[GPT 5.6 Has 72 Possible Configurations. What's A Good Default? Short note on how GPT 5.6 model and effort choices map onto training-time and inference-time scaling, producing 72 configurations.](/blog/2026/gpt-5-6-configurations.html)\n\n[Build a Reasoning Model From Scratch Is Out Short note announcing the release of Build a Reasoning Model From Scratch and linking the publisher and Amazon pages.](/blog/2026/build-a-reasoning-model-from-scratch-is-out.html)", "url": "https://wpnews.pro/news/inkling-a-new-open-weight-975b-moe-with-a-few-surprises", "canonical_source": "https://sebastianraschka.com/blog/2026/inkling-architecture-benchmark-notes.html", "published_at": "2026-07-16 14:03:05+00:00", "updated_at": "2026-07-16 14:25:50.209672+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-products", "artificial-intelligence"], "entities": ["Thinking Machines Lab", "Inkling", "GLM-5.2", "Kimi K2.5", "Nemotron 3 Ultra", "Tinker"], "alternates": {"html": "https://wpnews.pro/news/inkling-a-new-open-weight-975b-moe-with-a-few-surprises", "markdown": "https://wpnews.pro/news/inkling-a-new-open-weight-975b-moe-with-a-few-surprises.md", "text": "https://wpnews.pro/news/inkling-a-new-open-weight-975b-moe-with-a-few-surprises.txt", "jsonld": "https://wpnews.pro/news/inkling-a-new-open-weight-975b-moe-with-a-few-surprises.jsonld"}}