MLC-LLM

Rockchip's RK3588 SoC, found in boards like the Orange Pi 5 and Radxa Rock 5B starting at $75, can run local LLMs but is limited by memory bandwidth rather than its advertised 6-TOPS NPU, according to…

XGrammar, a grammar-constrained decoding library used by vLLM, SGLang, TensorRT-LLM, and MLC-LLM, masks invalid tokens before the model samples, guaranteeing valid JSON output and eliminating retry lo…

EdgeSync-LLM, a new KV cache fragment engine for on-device LLM inference, stores and retrieves transformer KV tensors via HNSW approximate nearest-neighbor search, enabling exact hits at ~8ms TTFT and…