What Hardware Runs Inkling? A 975B Model That Fits on One Box (Unlike Kimi K3)

Thinking Machines' open Inkling model, with 975 billion total parameters and 41 billion active, fits on a single high-memory box at 2-bit or 3-bit quantization, unlike the datacenter-scale Kimi K3. Real GGUF file sizes from Unsloth confirm a 2-bit Inkling at ~317GB and a 3-bit at ~433GB, enabling single-box deployment on systems like 4x RTX 3090 with 384GB system RAM, though llama.cpp support is still pending.

Our Inkling spotlight https://vettedconsumer.com/inkling-mira-muratis-first-open-model-is-a-975b-moe-you-can-actually-run/ ended on a claim that deserves the full receipts: Thinking Machines' first open model is big, 975 billion parameters, and yet a single high-memory box can hold it. That is the exact opposite of where our Kimi K3 hardware piece https://vettedconsumer.com/what-hardware-runs-kimi-k3-the-2-8t-options-ranked-and-when-to-just-rent/ landed two weeks ago, where nothing short of a datacenter fit. This is the ranked answer for Inkling, and this time the numbers are measured, not projected. That difference matters. K3's weights were not out, so every size we gave was scaled from an older model. Inkling's weights are out, and Unsloth has already published a full GGUF ladder https://huggingface.co/unsloth/inkling-GGUF?ref=vettedconsumer.com with real file sizes. So the fit math below is confirmed. The one thing still missing is owner tokens-per-second, because the model is three days old and llama.cpp support is an unmerged draft PR. We will flag that clearly where it matters. The two numbers that decide everything A Mixture-of-Experts model carries two parameter counts that do different jobs. Total parameters decide whether it fits in your memory. Active parameters decide how fast it decodes , because generation is memory-bandwidth-bound and each token only reads the experts that fire. Our MoE explainer https://vettedconsumer.com/mixture-of-experts-moe-explained-why-active-parameters-decide-what-runs-on-your-machine/ walks through the mechanism. For Inkling, both numbers are officially published, which is itself a step up from K3: 975B total, 41B active 6 of 256 routed experts plus 2 shared, per token . The 975B sets your memory floor. The 41B active is what makes a big model tolerably quick once it fits, roughly the same active footprint as a 40B dense model doing the actual token-by-token work. The usual trap still applies: "only 41B active" does not mean Inkling runs on a 41B-class card. The router picks different experts every token, so the whole model has to sit in memory you can reach quickly, or you pay a brutal streaming penalty. Total sets what fits; active sets the speed once it does. The measured footprint Because the weights shipped, these are real file sizes from Unsloth's GGUF ladder, not estimates. Inkling's 975B total lands just under Kimi K2's 1T, and the sizes track accordingly: | Quant | Measured size | Accuracy retained Unsloth | |---|---|---| | UD-IQ1 S 1-bit | ~270GB | ~74% | | UD-Q2 K XL 2-bit | ~317GB | ~81% | | UD-Q3 K XL 3-bit | ~433GB | ~89% | | UD-Q4 K XL 4-bit | ~587GB | ~94% | | Q8 0 8-bit | ~857GB | near-lossless | | BF16 full | ~1.89TB | reference | Weights only. Add roughly 10 to 15% runtime overhead plus KV cache. Thinking Machines also ships a native NVFP4 checkpoint at ~592GB, a 4-bit-class build meant for Blackwell GPUs. The accuracy-retained column is Unsloth's own benchmark of its dynamic quants, so read it as their measurement, not ours. The line that changes the story: a 2-bit Inkling is about 317GB , and a 3-bit is about 433GB . Both sit under 512GB. That is what makes a single box viable. What actually holds it, ranked The rule is unchanged: your fast memory has to exceed the quant size above, plus overhead and KV cache. Here is where the common machines land. | Hardware usable memory | Runs Inkling? | |---|---| | RTX 5090 32GB alone | No as a standalone needs heavy CPU offload; VRAM holds a fraction | | Strix Halo 128GB unified | No, even 1-bit ~270GB is about 2x too big | | 2x RTX 3090 48GB + 256GB system RAM | 1-bit / 2-bit via heavy RAM offload, slow but it runs | | 4x RTX 3090 96GB + 384GB system RAM | 2-bit to 3-bit via offload, the cheapest real path | | Yes, 2-bit ~317GB or 3-bit ~433GB in one box | | 2x H100 80GB 160GB | No on GPU alone 1-bit needs ~270GB ; fits only with big host-RAM offload | | 4x H200 141GB 564GB or an 8x node | Yes, up to 4-bit ~587GB fits an 8x node with KV room | The single 512GB Mac line is the reversal from Kimi K3: one Mac Studio cannot hold K3 at any quant, but it runs Inkling at 2-to-3-bit comfortably. Speed: what we can say, and what we can't We are not going to quote an owner tokens-per-second number, because none has been credibly measured yet. Inkling shipped three days ago, llama.cpp support is still an unmerged draft PR, and the r/LocalLLaMA thread is full of intent rather than logged runs. Anyone posting a consumer tok/s figure right now is on a patched build or estimating. What the architecture tells us is directional. Decode speed is memory bandwidth divided by the bytes read per token, and the bytes read are set by the 41B active count, not the 975B total. As we cover in Bandwidth, Not TFLOPS https://vettedconsumer.com/bandwidth-not-tflops-what-sets-your-local-llm-speed-and-why-the-newest-card-isnt-always-fastest/ , that is why the box you pick matters more than raw compute: Mac Studio M3 Ultra 819 GB/s : holds a 2-to-3-bit build in one unit, but that bandwidth over a 41B active load points to a modest single-digit-to-low-double-digit tok/s. Usable for chat, not fast. 3090 offload rigs: once the model spills into system RAM a few hundred GB/s, far slower than VRAM , speed drops further. Expect the low single digits. The appeal here is price, not pace. Multi-GPU H200 nodes ~4.8 TB/s per card : an order of magnitude faster, and the only path to genuinely snappy generation, but it is a rented-or-datacenter cost, not a desktop one. We will replace these directional notes with measured owner rates the moment llama.cpp support merges and real logs appear. The optimal box, by goal Best single-box self-host: a Mac Studio M3 Ultra 512GB. This is the headline. One machine, no cluster, no rack, holds a 2-bit ~317GB or even 3-bit ~433GB Inkling with room for context. At roughly $9,500 to $10,000 it is a serious purchase, but it is the cleanest "I own a frontier open model outright" answer that exists right now, and it is a machine that stays useful for every smaller model too. Our M3 Ultra review https://vettedconsumer.com/mac-studio-m3-ultra-the-local-ai-workhorse-buy-now-or-wait-for-m5/ covers the buy-now-or-wait question. Cheapest that genuinely runs it: a 4x RTX 3090 rig with 384GB of system RAM. Used 3090s plus a lot of DDR gets you a 2-bit Inkling through heavy offload for well under the price of a Mac, in the rough $4,000 to $6,000 range depending on used-GPU prices which have roughly doubled in 2026 https://vettedconsumer.com/gpu-price-index/ . The tradeoff is speed: offload to system RAM is slow, so this is the path for people who value owning it cheaply over running it quickly. Smartest for most people: rent or use the API. If you want Inkling fast, or only reach for it occasionally, an H200/B200 node by the hour beats a five-figure box that sits idle. Thinking Machines also hosts it, and independent trackers clock the hosted endpoint around 70 tokens per second, far past anything a home box will do at these sizes. Our cost calculator https://vettedconsumer.com/cost-calculator/ is built to price exactly this buy-vs-rent-vs-API tradeoff. The verdict Inkling is the first frontier-class open model in a while that a determined enthusiast can run on a single box they own. A 512GB Mac Studio holds a 2-to-3-bit build outright; a 4x 3090 rig does it cheaper but slower; and if you want speed, renting a GPU node or hitting the API still wins on cost-per-token. That is a genuinely different conclusion from Kimi K3, where the realistic answer was "almost nobody runs this locally." The one caveat: no owner tok/s numbers exist yet, so if raw speed is your deciding factor, wait a week for llama.cpp support to merge and real logs to land, and we will update this piece with measured rates when they do. And keep an eye on Inkling-Small 276B / 12B active , the sibling that will bring this same capability down to a single 3090 or a 128GB mini-PC. Sources and how we researched this - Specs 975B total, 41B active, 256+2 experts, 1M context, NVFP4 checkpoint : Thinking Machines Lab's Inkling model card https://thinkingmachines.ai/model-card/inkling/?ref=vettedconsumer.com and Hugging Face repo https://huggingface.co/thinkingmachines/Inkling?ref=vettedconsumer.com . - Measured quant sizes and accuracy retention: Unsloth's Inkling GGUF ladder https://huggingface.co/unsloth/inkling-GGUF?ref=vettedconsumer.com real file sizes, not projections . llama.cpp support is an unmerged draft PR https://github.com/ggml-org/llama.cpp/pull/25731?ref=vettedconsumer.com as of writing. - Hosted throughput reference: Artificial Analysis https://artificialanalysis.ai/models/inkling?ref=vettedconsumer.com ~70 tok/s on the hosted endpoint, a datacenter figure . - Hardware memory and bandwidth figures are vendor specs Apple M3 Ultra 819 GB/s, NVIDIA H200 ~4.8 TB/s, RTX 3090/5090 . Used-GPU pricing from our GPU price index https://vettedconsumer.com/gpu-price-index/ . - We have not run Inkling first-hand. Fit sizes are community-measured; speeds are directional from bandwidth math, not owner-measured. No consumer tok/s numbers exist yet. Related: Inkling: the 975B spotlight part 1 · What hardware runs Kimi K3 · Mac Studio M3 Ultra for local AI · Why active parameters decide speed