FP4 Brain on X: "got deepseek V4.1 flash running locally on a 16GB m1 mac mini original FP4/FP8 weights, ssd streaming + custom mlx runner 108s ttft and about 23s/token (not to be confused with tok/s)"
got deepseek V4.1 flash running locally on a 16GB m1 mac mini original FP4/FP8 weights, ssd streaming + custom mlx runner 108s ttft and about 23s/token (not to be confused with tok/s)
got deepseek V4.1 flash running locally on a 16GB m1 mac mini original FP4/FP8 weights, ssd streaming + custom mlx runner 108s ttft and about 23s/token (not to be confused with tok/s)
code + recipe if you're not in a hurry github.com/atbender/deeps… weights stay on SSD. the runner loads selected experts + lookup rows on demand, with a 4 GiB cache for dense weights. built on @pipenetwork MLX port
got it from about 31 to 23 seconds/token by reusing allocation buffers, compiling weight decoding and caching 4 GiB of dense weights tried 6 GiB too but it was meh, barely faster and more swapping. that's why I stuck with 4 if anyone's wondering tests + logs in the repo