{"slug": "show-hn-taalas-style-on-chip-llm-weights-on-a-250-amd-fpga-60k-tok-s", "title": "Show HN: Taalas-style on-chip LLM weights on a $250 AMD FPGA (60k tok/s)", "summary": "A developer achieved 59,965 tokens per second running a 3.16M-parameter INT4 transformer entirely in the on-chip memory of a $250 AMD Xilinx Kria KV260 FPGA, with zero DRAM in the token loop. The bit-exact result, measured on silicon, outperforms the board's Arm cores (11 tok/s) and an RTX 3050 Ti (719 tok/s). The project, inspired by Taalas's chatjimmy.ai demo, demonstrates that large language models can run efficiently on low-cost reconfigurable hardware.", "body_md": "# Taalas-Style On-Chip Weights on a $250 FPGA: a Language Model at 60k tok/s\n\nA 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.\n\nI was so impressed by the [chatjimmy.ai](https://chatjimmy.ai) demo by Taalas, I wanted to see what I could squeeze inside the The FPGA's reconfigurable logic. Same thing as 'PL'. Where the whole model runs here. of a $250 Field-Programmable Gate Array: a chip full of reconfigurable logic you wire into a custom digital circuit, instead of running software on a fixed CPU.. By not using the (4 GB available) The off-chip DRAM (the board's main memory), ~20 GB/s, shared by both the CPU and the fabric. This shared controller is 'the bandwidth wall'., and keeping everything in UltraRAM: the big, wide on-chip SRAM (~18 Mb, 64 blocks). Holds the resident INT4 weight image. Crucially, it is true dual-ported, which enables the two-cohort 'split-brain'./Block RAM: small, flexible on-chip SRAM blocks (~5 Mb total). Holds activations, scratch, and the KV cache here., I set myself a target of 100,000 tok/s.\n\nThe headline, bit-exact and measured on silicon: **59,965 tokens per second** on the fabric. The same model on this board’s own Arm cores manages 11 tok/s. My laptop’s RTX 3050 Ti manages 719.\n\nTaalas managed to bake Llama 3.1 8B into their chip, and have a usable chat. I had ~3 MB of memory to work with, which means the most powerful model we can fit is a A synthetic corpus of about 2.1 million very simple children's stories, designed so that even tiny models can learn coherent English from it. The training data here.-class one: 3.16M parameters, ~1.5 MB at INT4. Initially, I wanted to shrink it further by Reduce each word to its dictionary root ('saying' to 'say', 'words' to 'word'). The Keviniser lemmatises the corpus down to content words, about 30% fewer characters for the same story. the training corpus and making it sound like Kevin Malone, ‘why use many word when few do trick’, but since this didn’t actually make the model any smaller, it just speeds it up by ~30%, I went back on myself and just allowed the hardware to flip-flop between both models.\n\nFirst, talk to it.\n\n## The live demo\n\nThis widget is a real A persistent two-way connection between browser and server, unlike ordinary HTTP's request-then-hang-up. What lets the chat stream characters live. connection to the board. Your words go through a Cloudflare tunnel, to a serving box, to the Kria, into the fabric, and back. If the status dot is green, you are talking to a single FPGA in Wales. It’s a story generator, not an assistant. It doesn’t understand questions. Give it “once upon a time” and it finishes it.\n\nI hope you didn’t get your hopes up, there’s only so much you can do with ~1.5 MB of weights.\n\n## Why I even had this board\n\nThis is all running on my AMD KV260 dev board, featuring the K26 System-on-Module: the plug-in compute module that the KV260 board is built around.. This is one of the latest SoCs in the Zynq line (specifically the Zynq UltraScale+ ZU5CG/ZU5EV, catchy eh?), the successor to the OG 7000s. God, how I wanted a dev board for one of those 10 years ago.\n\nI bought the KV260 for a different side project, a deterministic vision pipeline. The board is sold as a “vision AI starter kit,” but the AMD/Xilinx's application development stack for their boards. The KV260's 'vision AI' demos run on it, mostly on the Arm cores rather than the fabric. object detection runs on the quad core The quad-core Arm Cortex-A53 CPU on the KV260 (~1.33 GHz). The baseline the fabric is compared against, and the orchestrator when the CPU is in the loop.s, and the A53 is a weak core with no hardware matmul. The Vitis libraries turned out to be basically OpenCV on Linux rather than anything that pushes the fabric. If I wanted probabilistic AI running on a CPU, I sure as heck wouldn’t choose a quad core A53.\n\nI did manage to get my deterministic depth engine running in Verilog on the fabric, but since it’s a robotics project, the hardware has a lot of downtime for safety reasons. So I put a fresh Linux image on a spare SD card and set up SSH so I can work on this from anywhere in what little free time I have (it very quickly spiralled out of control: another remote machine now has JTAG access to it, and the PSU is on a Zigbee relay so I can force reboots remotely).\n\n## The wall\n\nOne fact drives everything. Generating one token at a time is **memory bound, not compute bound**. To produce the next token you read every weight in the model once. The arithmetic is cheap, the reading is the cost.\n\nOn the KV260, the A53s and the fabric share one The off-chip DRAM (the board's main memory), ~20 GB/s, shared by both the CPU and the fabric. This shared controller is 'the bandwidth wall'. controller at roughly 20 GB/s. If the model lives in DDR, the fabric and the CPU drink through the same straw and the fabric buys you nothing. The round trip to DDR or CPU over The on-chip bus protocol family that connects the CPU to the fabric. kills you. The only escape is a model small enough that all of it fits in on-chip memory, where bandwidth is hundreds of GB/s. This is Taalas’ insight, and Groq’s, and Cerebras’: the memory wall is the enemy and on-chip weights are the escape. They spend hundreds of millions of dollars enlarging the on-chip budget. The KV260 gives you about 3 MB.\n\n3 MB has to hold the weights, the activations, and the KV cache. That is not enough room for a smart model. It is barely enough room for a model that can string a sentence together. So the second lever, the one the big players mostly can’t pull: shrink the model until the problem disappears.\n\n## Making the model small enough (this is where Kevin comes in)\n\nA 3.16M-parameter 4-bit integers (16 levels). The weights are stored as INT4, which is what makes the model small enough to fit on-chip. transformer fits in ~1.5 MB. That’s the budget met, but every byte still counts, and the last lever toward speed kept turning out to be “make the model dumber.” (or so I thought)\n\nSo the training corpus (TinyStories) is run through a tool that strips English to its content words. “Why waste time saying a lot of words when a few words do the trick” becomes “why waste time say lot word when few word do trick,” inspired by Kevin Malone’s communication philosophy from The Office, and yes, the model consequently talks like him. Across the full corpus the compression takes 371.7M words down to 260.5M, about 70%, measured.\n\nI expected the compressed-corpus model to come out smaller. It came out exactly the same size, and in hindsight that’s obvious: the parameter count is fixed by the architecture, not the corpus. What the compression does is shrink the output distribution. The same story takes ~30% fewer characters to tell, so effective speed goes up even though the per-token rate is identical.\n\nHonest version: the lemmatised corpus buys about 1.5x. The order-of-magnitude win is on-chip versus DDR. Kevin is the garnish on the chilli, not the meal.\n\n## Chasing 100k\n\nMy goal was 100,000 tok/s. I didn’t get there, but I gave it a bloody good go, eventually even compromising the usability by reducing the attention window to T=1 to remove as many round trips and race conditions as possible.\n\nThe record build is 16 parallel streams sharing one weight pass, entirely sequenced in the fabric, CPU touching nothing. It measures **59,965.5 tok/s at 200 MHz**, 16 of 16 streams bit-exact against the integer reference, three runs of three. The confession: those sixteen streams remember *nothing*. Each decodes with an attention window of one token. A chat built on it emits one faithful character and falls down the stairs, “he he he he he.” It is fast and it is meaningless, and those are the same property taken one step too far.\n\nSo the deployed chat is a different, honest build. One stream, full trained context window, KV caching bit-exact to a full recompute. That one runs **19,242 tok/s of fabric** by counted cycles, and **~21,300 tok/s measured live**, a number that held flat from 1 to 2,000 concurrent connections in a [load sweep](/blog/kevin-serving-strangers/) with zero errors (peak observed 21,479). It remembers your last couple of turns, and it’s the one in the widget above. Neither number gets to borrow the record’s headline.\n\nThe path from 11 tok/s to 60k was a ladder, every rung measured on silicon:\n\nAnd the ceiling is proven, not assumed. There’s a script that falsifies packing a third Multiply-ACcumulate: the one-multiply-one-add operation matrix multiplies are built from. The fundamental unit of compute here. into a DSP48E2: the FPGA's dedicated hardware multiplier blocks (1248 on this chip). Each can pack two INT4-by-INT8 multiply-accumulates. over 1.2M randomised products, and N=32 needs 2,048 DSPs the chip doesn’t have. The real limit of this architecture on this silicon is 62k to 78k. 100k was a finger-in-the-air guess before I even started, I just thought it would be a funny target number. Hitting 60k is close enough for me, and >20k *usable* beat the Taalas number, which was another goal I set myself.\n\n## Where it loses\n\nThe on-chip trick only wins while the model fits on-chip, and the crossover is around 6.3M parameters. Past that you spill to DDR and you’re back at the wall. Long context spills the KV cache the same way. This is a toy-model technique by construction, and the output is bad on purpose, which does not make it good. It’s a measurement instrument with a sense of humour.\n\nFor scale: Taalas etches weights into transistors on a taped-out ASIC per model. Cerebras keeps 44 GB of SRAM on a wafer. Groq keeps 230 MB per chip and gangs hundreds of them. Same one-sentence insight at budgets from nine figures down to the price of a nice dinner.\n\n## Go deeper\n\nEverything above is the story. The engineering is in the sub-articles, and every number in them is bit-exact on silicon before it’s quoted:\n\n, the wide-word GEMV trick, the dual-port split-brain, the non-linear bricks, and the proof that 16 streams is a hard ceiling.[Inside the chip](/blog/kevin-inside-the-chip/), how one unmeasured host loop ate 58% of every reply, and the Gumbel-max identity that collapsed 193 reads per token into one seed write.[Sampling without asking](/blog/kevin-gumbel-max/), the tunnel, the serving box, and why the dashboard deliberately lives off the board.[Serving an FPGA to strangers](/blog/kevin-serving-strangers/), where iverilog lies, why silicon beats the timing report by 1.3 to 1.76x, and the optimisation I won and stopped anyway.[War stories and the method](/blog/kevin-war-stories/)\n\nIt’s all hand and LLM written Verilog, no High-Level Synthesis: tools that compile C/C++ into hardware so you don't write Verilog by hand. This project has none; every block is RTL written directly, by me and by Claude Code, never compiled from C.. Claude Code did a solid amount, it’s a side project on a side project after all, though pushing an FPGA to its limit is definitely not as comfortable for it as writing a CRUD app in TypeScript. I would probably put the balance at:\n\n- Architecting: me 80%, Claude Code 20%\n- Implementing: me 20%, Claude Code 80%\n\nAnd despite saying it’s not that comfortable writing Hardware Description Language (Verilog, VHDL): code that describes circuits rather than instructions to execute. This project is Verilog throughout., it flipping did it. Outside of the stereo vision project, the last time I touched an FPGA was either a Lattice iCE40 about 10 years ago, or an even older Altera Cyclone IV on a DE0-Nano.\n\nI haven’t investigated OSS synthesis and bitstream generators, but my workflow didn’t involve opening Xilinx's FPGA design suite: synthesis, place, route, and bitstream generation. once. It was all called via the CLI. On my 20-core i7 with 32 GB of RAM, a full run was around 30 minutes. Place and route slowed right down as I got closer to 100% memory usage, but it’s still amazing what can be done.\n\n## The receipt\n\n**59,965.5 tok/s at 200 MHz, measured, N=16, bit-exact, three runs of three.** The deployed chat is the faithful build at 19,242 fabric tok/s by counted cycles, ~21,300 measured live under a 2,000-connection load test, live at [chat.mikeayles.com](https://chat.mikeayles.com) and right here. Same thesis as the startup AMD just bought, on AMD’s own $250 board, reprogrammable in 25 seconds.\n\nThe code is now public: [github.com/MichaelAyles/kev-gpt](https://github.com/MichaelAyles/kev-gpt), the Keviniser, the model, the RTL, and the serving stack, including the bit-exact gates you can run on your laptop without a board.", "url": "https://wpnews.pro/news/show-hn-taalas-style-on-chip-llm-weights-on-a-250-amd-fpga-60k-tok-s", "canonical_source": "https://www.mikeayles.com/blog/on-chip-llm-kv260/", "published_at": "2026-08-10 11:52:07+00:00", "updated_at": "2026-08-10 12:11:50.842596+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure"], "entities": ["AMD", "Xilinx Kria KV260", "Taalas", "chatjimmy.ai", "RTX 3050 Ti", "Zynq UltraScale+", "Arm Cortex-A53"], "alternates": {"html": "https://wpnews.pro/news/show-hn-taalas-style-on-chip-llm-weights-on-a-250-amd-fpga-60k-tok-s", "markdown": "https://wpnews.pro/news/show-hn-taalas-style-on-chip-llm-weights-on-a-250-amd-fpga-60k-tok-s.md", "text": "https://wpnews.pro/news/show-hn-taalas-style-on-chip-llm-weights-on-a-250-amd-fpga-60k-tok-s.txt", "jsonld": "https://wpnews.pro/news/show-hn-taalas-style-on-chip-llm-weights-on-a-250-amd-fpga-60k-tok-s.jsonld"}}