Charlie F. Ruan leads WebLLM, an open-source runtime that uses WebGPU, WebAssembly, and compiler-generated artifacts to run language models locally in a browser without an inference server. WebLLM, pu…
Rockchip's RK3588 SoC, found in boards like the Orange Pi 5 and Radxa Rock 5B starting at $75, can run local LLMs but is limited by memory bandwidth rather than its advertised 6-TOPS NPU, according to…
XGrammar, a grammar-constrained decoding library used by vLLM, SGLang, TensorRT-LLM, and MLC-LLM, masks invalid tokens before the model samples, guaranteeing valid JSON output and eliminating retry lo…
EdgeSync-LLM, a new KV cache fragment engine for on-device LLM inference, stores and retrieves transformer KV tensors via HNSW approximate nearest-neighbor search, enabling exact hits at ~8ms TTFT and…