A developer achieved 59,965 tokens per second running a 3.16M-parameter INT4 transformer entirely in the on-chip memory of a $250 AMD Xilinx Kria KV260 FPGA, with zero DRAM in the token loop. The bit-…
A 3.16M-parameter transformer running on a $250 FPGA achieves 44 tokens per second by moving the CPU out of the per-token loop, according to a technical blog post detailing the on-chip LLM design. The…
Mike Ayles' on-chip LLM project serves a large language model from a $250 FPGA (Xilinx Kria KV260) to the public internet via a Cloudflare Tunnel, achieving ~19,240 fabric tokens per second but only ~…
A $250 FPGA-based on-chip LLM project achieved a record 59,965.5 tokens per second with a split-brain N=16 configuration at 200 MHz, according to the developer's blog post. The project, built with han…
An FPGA-based on-chip LLM decoded replies in 6.6ms, but browser delivery took 100ms; profiling revealed that 58% of the delay came from a host-side loop reading 193 logits per token over /dev/mem. The…