Kimi k3 locally with 8gb ram 0 vram?!?!

A new C99 program under 1 MB enables running Kimi k3, a large language model requiring nearly 2 TB of storage, on systems with as little as 8 GB RAM and no GPU, though at slow speeds. Users report achieving 26 tokens per second on an AMD Strix Halo workstation with full precision, and the program also runs other models like DeepSeek v4 Flash on a phone at about one token per second. The approach uses sparse inference and drive-backed storage, with plans for Vulkan support and GPU/CPU mixing.

Wow this is a great write up and small program that can show you how LLMs run. I would suggest not just trying to run this but also try to understand how by reading the docs. This program is written in C99 and is less then a meg able to run kimi k3 with as small as 8gb ram and no gpu, you do still need almost 2 TB or memory for the model. Has anyone else tried this or something like it? I was able to run deepseek v4 flash on my phone with this yes this is not just for kimi and yes only one token a second but on my phone, still very cool . Very cool demo, but the 8 GB claim is a bit misleading. The model still needs nearly 2 TB of storage and runs very slowly. Still impressive that it can work without a GPU Yes a hard drive that can fit the models is needed, that is stated, and it does run slow on 8gb ram. But there is layers to how this works, in the workstation preset I am running full precision Kimi k3 on a strix halo and getting 26 tokens a second still no gpu used. That is very usable Currently downloading a copy to my 64-core 512gb ram epyc server to see how it runs. This changes alot for the price to run large models locally as you can run it off your nas if you wanted. Once again you can run alot of models this way not just kimi. 1 Like We need a nickname for these sub 5t/s methods. Many models can do this on various hardware 1 Like PM poor man frontier harness maybe? I think I’m going to start a fork of this with Vulkan support to run / get kv cache to work, and maybe an option to run the trunk on gpu / mix of gpu and cpu. So you can get more speed if you have it but still only limited by the harddrive. might check the various same concept, different model It’s almost once a day on HN/LB. Most of the good stuff is working with “ GitHub - NimbleEdge/sparse transformers: Sparse Inferencing for transformer based LLMs · GitHub https://github.com/NimbleEdge/sparse transformers ” and similar “only load what you need” methods, and drive backed storage a lot. Vulkna is achievable, but swapping models from disk to VRAM vs RAM isn’t really a win, the same bottleneck essentially in the CPU passing the data around. 1 Like How do these projects connect to opencode or something else to interact with? I’d love to try this out, I guess anything can happen if you are good with slow speeds.