UT Austin and UC Berkeley have teamed up to find the bottlenecks of running LLM on consumer hardware, and have released FreeToken.
FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU.
Source: [[2608.16157] FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution](https://arxiv.org/abs/2608.16157)
From my initial reading, it seems to be a way to analyze a machine’s capabilities and adjust a model’s layers accordingly to simultaneously maximize all available bandwidth between CPU, Memory, PCIe lanes, and GPU.
Markettechposts has made a fantastic little widget to visualize how it breaks a model up according to your hardware specs:
Well worth the read for the details: Meet FreeToken: An Edge-Native MoE Serving Engine that Runs 753B GLM-5.2 on a Single Workstation GPU - MarkTechPost
Seems to be only for Nvidia at this time. I’m trying to find out what work is being done for AMD.
Lastly, give a Hat Tip to DigitalSpaceport as I found this through casual background viewing. This is worth a watch (WRX80, 3945WX, DDR4 3200 hindered at 89 GB/s + Nvidia 3090 - running DeepSeek Flash V4).
FreeToken Deepseek V4 Flash on a Single 3090 Local AI Testing I have a workstation laptop with an A5500 (aka 3080 Ti, but with 16GB ram) and an AM5 with an 4070 Super (12GB) to try. I’ll post details as I experiment.