{"slug": "show-hn-run-full-kimi-k3-with-29-gb-of-ram", "title": "Show HN: Run Full Kimi K3 with 29 GB of RAM", "summary": "A developer has demonstrated running the full 2.78-trillion-parameter Kimi K3 model on a consumer 64 GB MacBook Pro using a custom inference engine called WASTE, achieving 0.32–0.34 tokens per second with a minimum RAM requirement of 29.05 GB. The engine streams experts from disk and uses RAM as a bounded cache, marking the first known instance of a trillion-scale model streaming from NVMe on consumer hardware without distillation or pruning.", "body_md": "**Kimi K3 — 2.78 trillion parameters — running on a consumer laptop.**\n\n``` bash\n$ waste run ~/models/k3.waste 'What is the capital of Italy?'\nwaste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB)\nThe capital of Italy is **Rome**.\n[16 tokens, 49.31 s, 0.32 tok/s | experts 3357 hit / 20195 miss = 14%]\n```\n\nWASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.\n\nIts current proof point is the complete open-weights Kimi K3 model: 2.78 trillion parameters, converted into a 982 GiB container and running on a 64 GB MacBook Pro at 0.32–0.34 tokens per second. **This is not a distilled, pruned, or reduced variant**.\n\n| Model | Container | Minimum RAM | Tested speed |\n|---|---|---|---|\nKimi K3 2.78T |\n982 GiB | 29.05 GiB | 0.32–0.34 tok/s |\nKimi-Linear 48B |\n19 GiB | 1.86 GiB | 8.92 tok/s |\n\nWASTE was written for that one model and that one constraint: **K3 does\nnot fit in the RAM of current mainstream consumer systems.** It is 1.42 TB\nas published and 982 GB after conversion. But a mixture of experts\nactivates about 4% of itself per token, so almost all of that weight is\nidle at any instant — and idle weight does not need to be in memory, it\nneeds to be *reachable in time*. WASTE keeps it on disk in a layout where\none expert costs exactly one read, streams what each token actually\nneeds, and spends every remaining byte of RAM on the part that repeats.\n\nThe engine is correct: every layer is validated against a PyTorch reference, the final logits agree to 3.6e-06, and the vision tower matches its own oracle to 2.3e-06. It is also slow — a third of a token per second, fifty seconds for the sentence above.\n\nBoth of those matter, and the second one should not be read as a disclaimer. We are not aware of another published demonstration of a model this size streaming from disk on a consumer machine: we found none for trillion-scale NVMe streaming, and the best-documented 671B-class recipes assume a server with a terabyte of DDR5. That is a report of what our search turned up rather than a survey — this repository carries no bibliography and no comparison table, so read it as an invitation to send a counter-example, not as a result. The interesting part is not the speed, it is that the whole thing is in the reachable range on a single consumer machine — and that from here the question is engineering rather than feasibility. Half of a decode step is already disk I/O running near the drive's measured ceiling, so the levers are known: read fewer bytes per token, and keep more of them in RAM.\n\nWhat that opens up, concretely: a frontier-scale model that answers with no network, no per-token invoice, and nothing leaving the machine — which is the difference between \"you may not send that data to an API\" and \"run it here\". The format and the engine are not K3-specific in any deep way; K3 is simply the hardest case that exists today, and a model that streams at 2.78T streams comfortably at 48B.\n\nEvery number in this document was measured on the commit it is published\nwith, and the ones that were wrong are recorded as wrong in\n[docs/LEARNED.md](/sqliteai/waste/blob/main/docs/LEARNED.md) rather than quietly corrected.\n\nEvery token answered by a cloud service is paid for twice: once on the invoice, and once in the electricity of a datacenter running a model that would fit — barely, awkwardly, but genuinely — on hardware already sitting on a desk. WASTE means to be the first concrete step toward ending that waste of tokens. The acronym came second.\n\ndisk, for the model |\n982 GB for the converted container — plan a terabyte |\n| disk, to convert it | another 1.42 TB of staging for the published shards, freed afterwards |\nRAM |\n29.05 GB minimum to open K3 at 4K context; 64 GB for the numbers here |\n| storage speed | the container must be on internal NVMe — see below |\n| build | a C11 compiler and `make` . No BLAS, no CUDA, no Python at run time |\n\nSizes here are powers of two, the way `df`\n\nand the engine both report\nthem: the container is 982 GiB, which a disk vendor would call 1.05 TB.\n\nThe RAM floor is what the engine refuses to start below, and it is almost entirely the 27.28 GB resident trunk. Useful throughput starts higher: on a 64 GB machine the engine gives itself a 46 GB budget, of which 17.56 GB is expert cache, and that is the top of the measured curve. A 32 GB machine can technically open the model and will page badly; treat 64 GB as the real requirement.\n\n**Storage speed is not a detail.** A token reads 17 GB of experts. On the\ninternal SSD that is 12.78 GB/s and the model streams; over a USB\nenclosure it is 0.94 GB/s and the same token takes thirteen seconds.\nConvert onto internal NVMe, and use the external disk for the download\nonly.\n\nIf a terabyte is not available, the same engine and the same format run\n`Kimi-Linear-48B-A3B-Instruct`\n\nfrom a **19 GB** container with a\n**1.86 GB** floor, at 8.92 tok/s. That is the good path for trying WASTE\nout before committing a disk to K3.\n\n**Self-contained.** One`libwaste.a`\n\n, one`waste`\n\nbinary, nothing at run time beyond libc and pthreads.**Zero dependencies.** No BLAS, no ONNX, no Python in the inference path, nothing to install. The Python under`tools/`\n\nconverts models and validates the engine; it never runs alongside it.**Fully embeddable.** Twenty-six public functions in[src/waste.h](/sqliteai/waste/blob/main/src/waste.h): open a model under a RAM ceiling, generate, save the session, close. The CLI is a client of that API and touches nothing private — if the CLI can do it, so can an embedding host.\n\n```\nwaste_cfg cfg;\nwaste_cfg_init(&cfg);\ncfg.ram_budget_bytes = 46ULL << 30;  /* a hard ceiling, not a hint;\n                                        0 sizes it to this machine */\n\nwaste_ctx *ctx;\nif (waste_open(\"/path/to/k3.waste\", &cfg, &ctx) != WASTE_OK) return 1;\nwaste_generate(ctx, ids, n, &params, on_token, user);\nwaste_close(ctx);\n```\n\nThe path is the container directory the converter wrote — no `~`\n\nexpansion here, that is the shell's job.\n\nA model is converted once into a `.waste`\n\ncontainer: a JSON manifest, a\nresident trunk, and one expert bank per layer. Each expert record is\n4 KiB-aligned with its gate, up and down matrices adjacent, so routing to\nan expert costs exactly **one pread** — not three, not a seek per\nmatrix. The arithmetic was never the bottleneck.\n\nReads bypass the page cache (`F_NOCACHE`\n\non macOS, `O_DIRECT`\n\non Linux,\n`FILE_FLAG_NO_BUFFERING`\n\non Windows). That is deliberate: with a container\nsmaller than RAM the kernel would cache everything, and the hit rates\nmeasured that way are a fiction that does not survive contact with a\n982 GB model.\n\nEvery record's header is checked on the way in — right magic, the expert\nthe index asked for, offsets that fit — so a bank that has been truncated\nor spliced stops the generation and names the record instead of answering\nfrom the wrong bytes. That costs nothing measurable. The record also\ncarries a `crc32`\n\nover its payload, and checking *that* is `--verify`\n\n,\noff by default: it is a pass over every record on every cache miss, about\n5% on Kimi-Linear and 1% on K3. Worth it for a container you copied or\ndownloaded and have not read since; not worth it on every token of one\nyou converted yourself. See [docs/FORMAT.md](/sqliteai/waste/blob/main/docs/FORMAT.md).\n\nExperts are stored as residual vector quantization — three stages of 256-entry codebooks over 8-dimensional vectors, 3.00 bits per weight — and the matrix is never materialized. For each token the engine builds a table of partial dot products, one per codebook entry per vector position, after which every expert row is three table reads and two adds.\n\nThe trunk stays at 4 and 8 bits. The model was trained with\nquantization-aware training on the *experts* only, so it has no trained\ntolerance for a squeezed trunk: a 3-bit trunk was built and measured, the\ncache prediction held, the throughput did not, and the output collapsed.\n\nThe most predictive number in this project. K3 touches 16 experts in each\nof 92 layers per token: **17.0 GB**. Below that, an expert cached for one\ntoken is evicted before the next token asks for it, and the hit rate is\nnot low — it is zero. Above it the curve bends sharply.\n\n| budget | expert cache | hit rate | decode |\n|---|---|---|---|\n| 32 GB | 3.32 GB | 0% | 0.31 tok/s |\n| 46 GB | 17.32 GB | 13% | 0.32 tok/s |\n| 52 GB | 23.32 GB | 27% | 0.11–0.14 tok/s |\n| 58 GB | 29.32 GB | 37% | 0.04 tok/s |\n\nMeasured in that order, on an otherwise idle machine. Order matters:\nre-run *after* the 52 and 58 GB rows have driven the machine into paging,\n46 GB gives 0.22–0.25 rather than 0.32 — while reporting hit and miss\ncounts identical to the digit. The engine is deterministic; the machine\nis not, and it does not fully recover between runs. Sweep upward.\n\nEverything in the memory design exists to get above that line, which is why the engine works to free RAM rather than to save it.\n\n**And there is a ceiling on the other side, closer than it looks.** Read\nthat table twice: the hit rate climbs all the way down. At 58 GB on a\n64 GB machine the cache serves 37% of experts from RAM and the engine is\n*eight times slower* than at 46 GB, where it serves 13%. The engine is\ninside its budget; the *machine* is not, so the OS pages out the expert\ncache, and a \"hit\" becomes a page fault instead of the disk read the\nengine was managing.\n\nSo the usable window is narrow. It opens at ~46 GB, where the cache finally clears one token's working set, and it has already closed by 52 — on an otherwise idle machine, with 49 GB free before the run. It is also sharp enough to move under a change that looks unrelated: taking 1.11 GB of embedding table off the resident set fed straight into the cache at a fixed budget, and that was enough to push 58 GB from 0.32 tok/s to 0.04.\n\n**So the default does not fill the machine.** Expert cache is only worth\nanything in whole multiples of that working set, and the remainder above a\nmultiple buys a few points of hit rate while pushing the machine towards\npaging. When it picks a budget for itself the engine steps down a whole\nworking set at a time and takes the largest that fits under seven eighths\nof RAM: K3 asks for floor + 3× — 80.63 GB — and gets floor + 1× on this\nlaptop, a 46 GB budget and a 17.56 GB cache. That is the top of the curve\nabove, reached with no flag. A 128 GB machine still gets the full 3×.\n\nAn earlier version took every byte up to the cap instead, which put a 27 GB cache on this machine — between two budgets measured at 0.11 and 0.04 tok/s. The real lesson is that a cache you do not control is not a cache, and the corollary is that an engine should stop asking for memory before the OS starts taking it back.\n\nK3's attention is a 3:1 hybrid: Kimi Delta Attention, which carries a\nfixed-size recurrent state instead of a growing KV cache, and gated\nmulti-head latent attention. The MLA layers cache the 512-wide latent\nrather than expanded per-head keys and values, with `kv_b_proj`\n\nabsorbed\ninto the query and the output:\n\n```\nq_nope · (W_kb c)    ==  (W_kbᵀ q_nope) · c\nΣ_s a_s (W_vb c_s)   ==  W_vb (Σ_s a_s c_s)\n```\n\nIdentical logits to 1.2e-05, and **53× less cache**: 11.25 GB becomes\n0.21 GB at 4K context. It is also what makes long context possible at all\n— the expanded layout wants 360 GB at 128K tokens, the latent one 7.2.\n\nMacBook Pro M5 Pro, 64 GB, container on the internal SSD. Every figure was measured on the commit it is published with.\n\n| minimum RAM | 29.05 GB at 4K context |\n| 30.54 GB at 32K, 35.63 GB at 128K, 83.21 GB at 1M | |\n| resident trunk | 27.28 GB |\n| read per token | 17.0 GB at ~9.9 GB/s, near the SSD's measured ceiling |\n| model load | 20 s |\n| prefill | 0.47 tok/s chunked, 0.29 sequential |\n| decode | 0.32–0.34 tok/s at the default budget, the best this machine gives |\n| vision tower | 15.7 s for a 1024-patch image, 27 layers |\n| image in a prompt | 256 positions for 896x896, 2.8 s each — as text |\n\nThe floor is almost entirely the resident trunk. Useful throughput starts above ~46 GB, where the expert cache finally clears one token's working set, and is gone again by 52, where the machine starts paging. Below the first line extra RAM buys nothing; above the second it costs, badly. The window is one budget wide on this machine.\n\nThe tower is not what an image costs. Encoding 1024 patches takes 15.7 s;\nthe 256 positions it produces then go through the 92 MoE layers like any\nother token, which is the other 731 s. An image is priced as text of the same\nlength, so the patch budget in `vision.json`\n\nis a real dial: halving the\ngrid halves the prompt.\n\n| minimum RAM | 1.86 GB |\n| decode | 8.92 tok/s at an 8 GB budget, 78% cache hit |\n\nThe same engine and the same format, on a model that fits comfortably. This is what WASTE looks like when it is not fighting.\n\nDecode on K3, 17.32 GB of cache and still cold — 6.7% hit over ten steps, which is the state a fresh prompt starts in:\n\n| share | |\n|---|---|\n| MoE, all of it | 82.5% |\n| of which expert I/O | 53.5% |\n| of which expert matmul | 20.0% |\n| KDA layers | 14.5% |\n| MLA layers | 2.8% |\n| lm_head | 0.2% |\n\nReproduce with `WASTE_PROFILE=1 WASTE_CACHE_MB=17735 ./test_forward MODEL 1008,10484,318,15383,387 out.bin 5`\n\n. The I/O share falls as the\ncache warms, so a long session sits lower than this; the ranking does\nnot change.\n\nThe I/O already runs near the hardware limit — 17.0 GB per token at ~9.9 GB/s against the SSD's measured 12.78 — so it only gets cheaper by happening less often, which means cache, which means RAM. That is the whole optimization story so far, and the reason the next steps are about memory rather than arithmetic.\n\n```\ngit clone https://github.com/sqliteai/waste && cd waste\nmake                          # libwaste.a, waste, libwastevq\nmake check                    # 23 pass, 11 skip on a fresh clone\n```\n\nNo configure step and no dependency resolution. `make check`\n\nneeds no\nmodel: it builds a small synthetic container and runs the engine against\nit. The eleven skips are the checks that need something a clone does not\ncarry — the PyTorch oracle, the round-trip against the source shards,\nanything driving the CLI with text, since the synthetic container carries\nno tokenizer, and the K3 checks, which want the container and the release\non disk. With both containers present the suite is 36 checks.\n\nConversion is the one step that needs Python, and it happens once. The\nsource is [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3)\nexactly as published — 96 safetensors shards, 1.42 TB, nothing patched:\n\n```\n# 1. preflight: reachable? how big? does it fit?\ntools/fetch_weights.sh --dest /Volumes/staging/k3 --dry-run\n\n# 2. download — resumable, safe to kill, safe to re-run\ntools/fetch_weights.sh --dest /Volumes/staging/k3\n\n# 3. convert into a container\nuv run --with torch --with safetensors python tools/convert.py \\\n    --src /Volumes/staging/k3 \\\n    --out ~/models/k3.waste --jobs 3\n```\n\nThat produces the 982 GB container every number above was measured on. It\ntakes about **4.7 hours** with three processes on the M5 Pro (23.7 with\nthe pure-torch encoder — see [docs/K3.md](/sqliteai/waste/blob/main/docs/K3.md)), and wants ~1.0 TB\nfree on the target volume. The converter is resumable too: a layer whose\nbank is already written is skipped, so an interrupted run costs only the\nlayer it was in the middle of.\n\nThe download is the part that goes wrong. A 1.42 TB pull over hours will\nhit dropped connections, CDN 5xx and at least one interrupted run, so\nevery shard resumes mid-file rather than restarting, retries with\nexponential backoff and jitter, and counts as done only when its size\nmatches Content-Length — recorded in a state file, so a re-run skips\nfinished shards without even a HEAD request. `--check`\n\nre-verifies\neverything on disk against the remote and downloads nothing (96 shards in\n34 s). `--repo`\n\npoints it at another model, `HF_TOKEN`\n\nat a gated one.\nmacOS and Linux.\n\nGive `--dest`\n\na staging disk rather than the volume that will hold the\ncontainer. The shards are read once, by the converter; the container is\nread continuously, at every token. On this machine the external enclosure\nmeasures **0.94 GB/s** against the internal NVMe's **12.78** — see\n[docs/GATES.md](/sqliteai/waste/blob/main/docs/GATES.md), Gate H — which is the difference between\na model that streams and one that stalls.\n\n`tools/pipeline.sh`\n\nchains the whole thing unattended — download, convert,\nround-trip the container against the source weights, generate, then diff\nthe logits against the PyTorch oracle — and leaves a report next to the\ncontainer. The same converter handles the other member of the family,\n`Kimi-Linear-48B-A3B-Instruct`\n\n, into the 19 GB container of the second\nbenchmark; `--src`\n\nis the only thing that changes.\n\n**Pre-converted containers are on their way to\nhuggingface.co/sqliteai**, at which\npoint this whole section becomes a download and the Python is only needed\nfor models we have not published.\n\nThe container is the directory the converter wrote, so give it that path —\n`~/models/k3.waste`\n\nthroughout this README:\n\n```\nwaste run   ~/models/k3.waste \"The capital of France is\" -n 32\nwaste chat  ~/models/k3.waste                     # multi-turn, state kept\nwaste eval  ~/models/k3.waste \"2 + 2 =\" --top-k 5 # next-token distribution\nwaste plan  ~/models/k3.waste --budget 46G        # what fits, what does not\necho \"prompt\" | waste run ~/models/k3.waste       # stdin works too\n```\n\n`-n`\n\nis a cap, not a requirement: without it generation stops at the\ncontainer's end-of-sequence token or at 128 tokens, whichever comes\nfirst. The examples pass it because 128 tokens of K3 is six minutes.\n\n`--budget`\n\nis optional, and leaving it out is the right default rather\nthan a fallback: the engine takes the container's recommendation, steps it\ndown a whole token working set at a time until it fits under seven eighths\nof physical RAM, and never goes below the floor — a budget you set\nexplicitly under the floor is refused rather than swapped into. It then\nsays on stderr what it landed on, so the same command on two machines is\nnot silently two different runs:\n\n```\nwaste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB)\n```\n\n`--verify`\n\nchecks each expert record's `crc32`\n\nas it comes off the disk.\nIt is off by default, and that is a throughput decision rather than a\nclaim that containers do not rot: it is a pass over every record on every\ncache miss, about 5% on Kimi-Linear and about 1% on K3, where the read\ndominates. Turn it on once for a container you copied, downloaded, or left\non a disk you do not trust, and for anything whose wrong answers would be\nbelieved; leave it off for one you converted yourself and have been\nreading since. `WASTE_VERIFY=1`\n\nin the environment does the same thing,\nand the server takes `--verify`\n\nas well. Any of them turns it on; none of\nthem turns it off.\n\nWhat is checked either way: a short read, and a record header that does\nnot describe the expert the bank index asked for. Those are O(1), they\ncost nothing measurable, and they are what keeps a damaged offset out of\nthe arithmetic — `--verify`\n\nonly adds the pass over the payload.\n\n`waste --help`\n\nlists all nine commands. `--json`\n\nmakes `eval`\n\n, `tokenize`\n\n,\n`plan`\n\n, `info`\n\nand `bench`\n\nmachine-readable.\n\n`serve/`\n\nis an OpenAI-compatible HTTP server — the second client of the\npublic API, alongside the CLI, reaching the same engine through ctypes\nrather than keeping a copy of the model code in Python:\n\n```\nmake libwaste.dylib                     # or libwaste.so on Linux\npython3 -m serve ~/models/k3.waste --port 8000\ncurl localhost:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"k3\",\"messages\":[{\"role\":\"user\",\"content\":\"Why is the sky blue?\"}]}'\n```\n\n`/v1/chat/completions`\n\n(streaming and not), `/v1/completions`\n\n,\n`/v1/models`\n\n, `/health`\n\n. It carries the whole of K3's prompt format, not\nthe four-string subset a container's `chat.json`\n\ncan hold: **tool\ndefinitions and tool results, typed call arguments, JSON response schemas,\ntool_choice, the think channel and thinking_effort, and images** — plus\nthe parser that reads the reply back into reasoning, answer and\n\n`tool_calls`\n\n. Stdlib only.The prompt renderer is a port of `encoding_k3.py`\n\nfrom the release, and the\ntest suite checks it against that file **segment for segment** on a corpus\nof 38 conversations whenever the weights directory is on disk.\n[docs/SERVE.md](/sqliteai/waste/blob/main/docs/SERVE.md) is the reference.\n\nK3 is multimodal — a 401M ViT, 27 layers, patch 14 — and so is the engine.\n`--image`\n\nattaches a picture; repeat it for several:\n\n``` bash\n$ waste run ~/models/k3.waste 'What is in this picture?' --image landscape.png\n[landscape.png: 192 image tokens]\nThe picture shows a simple, stylized landscape with:\n\n- A **blue sky** with a gradient from darker blue at the top to lighter blue near the horizon.\n- A **yellow sun** in the upper right.\n- A **gray hill or mountain** in the middle distance.\n- A **green field** covering the lower part of the image.\n[78 tokens, 234.25 s, 0.33 tok/s | experts 15314 hit / 99502 miss = 13%]\n```\n\nThat is a 448×336 image, and every element of the description is in it — including the sky gradient, which is the kind of detail that separates a tower that works from one that merely runs. The picture was generated by a twenty-line script rather than photographed, so the answer can be checked against what was drawn instead of against an impression.\n\nPNG, JPEG, GIF, BMP, TGA and PSD, decoded by the one vendored header in\n`third_party/`\n\n. It works on `run`\n\n, `chat`\n\nand `eval`\n\n; inside a chat,\n`/image FILE`\n\nattaches a picture to the next message, and it is spliced\nonce — the positions are in the attention state afterwards, so later turns\ndiscuss the same photograph without re-encoding it. The 27-layer ViT is\nloaded only when an image is present, because its 434 MB otherwise come\nstraight out of the expert cache.\n\nAn image is not one token. The tower turns a 14-pixel patch grid into one\nembedding per merged 2×2 patch, and each occupies a position in the\nsequence — the 448×336 above is 192 of them, an 896×896 photo at the\ndefault budget is 256. That is worth knowing before wondering where a\ncontext window went, and it is most of what an image costs: the 234 s in\nthe transcript is the 78 generated tokens alone, and the picture is paid\nfor before that, in prefill. **An image is priced as text of the same\nlength.** The tower is the cheap part — 15.7 s for a full 1024-patch\nimage — and its output then walks through 92 MoE layers like any other\ntoken. Halving `max_patches`\n\nin `vision.json`\n\nhalves the bill.\n\nThrough the library it is three calls, because a host needs to size the prompt before committing to it:\n\n```\nsize_t rows;\nwaste_image_add(ctx, \"photo.png\", &rows);          /* encode and queue   */\nwaste_image_expand(ctx, raw, n, ids, cap, &n_ids); /* placeholder -> N   */\nwaste_generate(ctx, ids, n_ids, &params, cb, u);   /* consumes the queue */\n```\n\nThe tower's shape, the patch budget and the pixel normalization live in\n`vision.json`\n\n, which the converter writes from the release's own nested\n`vision_config`\n\nand from `preprocessor_config.json`\n\n. K3 normalizes to\n[-1, 1] with mean = std = 0.5.\n\nThat last sentence was wrong here for a day, and the way it was wrong is\nworth keeping. This section used to say **K3 ships no preprocessor\nconfig**, so the normalization was \"the CLIP convention this lineage of\ntowers uses rather than a value read out of the release\" — an assumption,\nlabelled as one. The release does ship the file; the downloader fetched a\nhardcoded list of filenames and never asked the repo what it contained.\nThe tower still matched its oracle at 2.3e-06 throughout, because the\noracle is fed random pixels and never touches the normalization. An\nhonest caveat is not a substitute for reading the file.\n\n| build | model-free suite | backend | |\n|---|---|---|---|\n| macOS arm64 | yes | 23 pass / 0 fail / 11 skip | NEON |\n| Linux arm64 | yes | 23 pass / 0 fail / 11 skip | NEON |\n| Linux x86_64 | yes | 23 pass / 0 fail / 11 skip | AVX2 |\n| Windows x86_64 | yes | container, CLI and forward pass — see below | AVX2 |\n\nThe first three run the same suite and now agree check for check: same\n23 passes, same 11 skips, same list. CI has no container, so\n`tests/run.sh`\n\nbuilds a synthetic one and the checks that need real\nweights say SKIP rather than passing quietly. All three also pass the\nsanitizer suite and 400 fuzz cases.\n\nWindows is cross-compiled with MinGW-w64 on a Linux runner and then run\non a Windows one: the binary reads a synthetic container, opens it from\nthe CLI, and produces the same logits token-by-token as it does in\nchunks. It is not the same suite — `tests/run.sh`\n\nis a bash script that\nrebuilds first, and the Windows job runs binaries it did not build — so\nwhat is claimed is what that job checks and no more. Nobody has run it on\na real container there.\n\nThe platform is the variable, the suite is not, and that is the point:\nboth Linux targets produce the same continuation as macOS and pass\n*engine matches the PyTorch oracle* when given a container, so the\nnumerics carry across architectures and compilers.\n\nSIMD is selected at run time from CPUID, so a single x86 binary uses AVX-512 where it exists and AVX2 where it does not. Accelerator backends are build-time options. A Metal backend exists and is off by default because it is correct and 22% slower: this engine issues several hundred small dependent matvecs per token, the worst possible shape for an accelerator, and the CPU path already runs at the machine's memory bandwidth.\n\n```\nsrc/        the engine — 6,000 lines of C, no dependencies\n  model.c     forward pass, MoE routing, KDA and MLA layers\n  ecache.c    bounded LFRU expert cache over the banks\n  vision.c    the 27-layer ViT and the projector into text space\n  image.c     a file on disk to the patch tensor the tower wants\n  waste.c     the public API\n  simd_*.c    per-ISA kernels, selected at run time\ncli/        the CLI, a client of the public API\nserve/      the OpenAI-compatible server, the other client\n  xtml.py     K3's prompt format, ported from the release's encoding_k3.py\n  regions.py  its replies, back into reasoning / answer / tool calls\n  engine.py   libwaste through ctypes, and the request queue\n  server.py   /v1/chat/completions and friends\ntools/      conversion and validation (Python, never at run time)\ndocs/       format, engine, backends, and what was learned\ntests/      34 checks, and a diff against a PyTorch oracle given a model\n  serve/      149 more for the server, incl. a differential vs upstream\nexamples/   chat.json for K3 and ChatML, the format a container carries\nthird_party/ stb_image.h, the single vendored header — see its README\n```\n\n[docs/LEARNED.md](/sqliteai/waste/blob/main/docs/LEARNED.md) is the one to read before contributing.\nIt records what was measured, including the optimizations that were\nrefuted — index-layout blocking, a 3-bit trunk, GPU offload, per-expert\nbit allocation — with the numbers that killed them.\n\nThe API is not frozen, as above. The rest is stated plainly too, because finding these out for yourself is worse than reading them here:\n\n- a container carries its chat format in\n`chat.json`\n\n, and the converter can only fill it in for a model whose format has been transcribed from its reference encoder — K3 today. Neither Kimi release distributes a template, so for anything else the CLI says so and continues raw rather than guessing a format, which would produce plausible wrong answers instead of visibly wrong ones. Kimi-Linear is in that position now; - AVX-512 compiles and is dispatched from CPUID, and has still never\nexecuted an instruction. This laptop is ARM and its x86 emulation is\nRosetta, which reports AVX2 and leaves the ZMM state disabled in XCR0;\nthe hosted x86 runner is an AMD EPYC 7763, which answers\n`avx512f/bw/dq/vl: no`\n\n, so CI says**AVX2** as well — on Linux and on Windows both. The workflow prints the runner's flags before every build, so the day a runner has them the*SIMD backend matches the CPU baseline*check becomes the confirmation without anyone arranging it; **Windows builds and runs, on one toolchain and one CPU.** MinGW-w64 x86_64, cross-compiled, with`src/platform.h`\n\nholding the six calls that are not POSIX: the positional read, the aligned allocation, the CPU count, the file size and`FILE_FLAG_NO_BUFFERING`\n\nfor the cache-bypass open. MSVC is a different port and has not been attempted — the sources use GNU C. ARM64 Windows is not built. Neither is the page-cache bypass proven under load there: CI confirms Windows grants it on the runner's filesystem, which is not the same as measuring a hit rate against a container that does not fit in RAM;**the expert checksum is off unless you ask for it**(`--verify`\n\n), and the trunk has no checksum at all. The first is a decision — 5% of throughput on every token, against a container that is usually fine — and it means the default build of a rotted container still answers with whatever the damaged bytes decode to. Run`--verify`\n\nonce after copying a container, to establish that it arrived intact.`tools/verify_container.py`\n\ndoes not stand in for that: it re-derives records against the**source** weights, so it wants torch and the original checkpoint on disk, and it answers whether the conversion was right rather than whether the copy still is. The second is not a decision: the trunk and the codebooks have nothing to check against in the format, and nothing has been built in its place;- every expert in a container is at the same bit width. The non-uniform\nper-expert allocation the format was designed around is\n**not coming**: it was measured on both models rather than built, and the importance it would allocate against does not vary — the value of the third bit spreads at most 1.15x between experts in a layer and 1.01x between layers, so the optimal allocator and a coin flip write the same container. The one signal that is not flat, routing frequency, buys disk footprint and almost no I/O, which is the resource that is actually scarce.[docs/LEARNED.md](/sqliteai/waste/blob/main/docs/LEARNED.md)§20 has the table and the one measurement that would revive it.\n\nApache 2.0 — see [LICENSE](/sqliteai/waste/blob/main/LICENSE). Copyright 2026 SQLite Cloud, Inc.", "url": "https://wpnews.pro/news/show-hn-run-full-kimi-k3-with-29-gb-of-ram", "canonical_source": "https://github.com/sqliteai/waste/", "published_at": "2026-07-30 06:12:55+00:00", "updated_at": "2026-07-30 06:22:29.248593+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["Kimi K3", "WASTE", "MacBook Pro", "Kimi-Linear 48B"], "alternates": {"html": "https://wpnews.pro/news/show-hn-run-full-kimi-k3-with-29-gb-of-ram", "markdown": "https://wpnews.pro/news/show-hn-run-full-kimi-k3-with-29-gb-of-ram.md", "text": "https://wpnews.pro/news/show-hn-run-full-kimi-k3-with-29-gb-of-ram.txt", "jsonld": "https://wpnews.pro/news/show-hn-run-full-kimi-k3-with-29-gb-of-ram.jsonld"}}