{"slug": "vibecoding-an-inference-chip", "title": "Vibecoding an inference chip", "summary": "A developer used coding agents from Anthropic and OpenAI to program an AWS F2 FPGA for AI inference, taking a ternary-weight model from a first bit-exact baseline to 213 tokens per second — a 33× speedup — over several days of autonomous experimentation. The cloud agent, run in Conductor, wrote roughly 3 million lines of C++ and Verilog across the experiment history and parsed 30 million lines of output files, at a cost of $10,000 in Claude and OpenAI tokens for hundreds of experiments. The author, who had never written Verilog, said the agents were \"incredible at programming them\" and noted the 16 GB of FPGA HBM made the ternary architecture suited to the memory-limited case.", "body_md": "One of the more entertaining things about programming recently is that coding agents let me work in entirely new domains by giving me access to new programming languages and paradigms. Lately, that has meant hacking on hardware, now that writing low-level software is much easier. So, in this blog post, I’m going to try to recount the experience of vibe coding an FPGA, going through a series of “oh shit” moments several times, and giving myself a new appreciation for what’s possible if you have a large token budget (thanks YC) and interesting problems to point the models at.\n\nI overheard some founders recently talk about how FPGAs work. I’ll admit I was relatively naive about them. I’d never really experimented with them in my startup or in college, so I knew little about them before. The basic idea I was familiar with: you can program a circuit into a reprogrammable chip. So rather than having a general purpose computing architecture (like a CPU) you can have a reprogrammable one, i.e. the configuration loaded onto the chip determines how its logic and routing are connected, and that (in theory) enables performance advantages. I also know that it took a mountain of C++ to write and involved manually analyzing waveforms for simulation and verification. I had never done this before though and it seemed cumbersome.\n\nThat’s about what I knew going into this.\n\nNaturally, my ML brain thought “could we use these for AI inference”. I’m not the first to think this — there are a few repos attempting to port ML models to FPGAs on Github. But I hadn’t actually seen a writeup on whether coding agents are good at writing the low level C++ and Verilog code to program them and verifying the outputs so that they actually run accurately. (Spoiler alert, they are incredible at programming them.)\n\nSo I decided I needed to get my hands on one. AWS has F2 instances with FPGA hardware, and that seemed easier than going and buying physical hardware, so I asked Codex to spin up a reservation and got cooking. It had a relatively limited memory of 16 GB of FPGA HBM. Around the same time, I was working with the DeepGrove founders who were discussing their new ternary weight architecture that is well suited for memory limited cases. It’s SOTA on Mac hardware with actual agentic tool calling, so I figured why not, let’s see if I can get this working on an FPGA and fast enough to be useful.\n\nWhat followed was a cloud agent running in Conductor for several days with me mostly just encouraging it periodically while it both (1) got a model fully running and (2) increased its speed by 33× from the first full-model bit-exact baseline to get it to a real-time relevant speed. It was one of the most mind blowing autoresearch loops I’ve run, ultimately writing roughly 3M lines across the experiment history and parsing 30M lines of output files, burning only $10,000 in Claude and OpenAI tokens to run hundreds of experiments. Most of the code is C++ and some Verilog. It’s been probably a decade since the last time I wrote any serious amount of C++ and I’ve never written Verilog. I honestly don’t know the first thing about Verilog. Most of my prompts were not particularly clever: they consisted mostly of me saying things like “you’re a superintelligent model, time to be more creative” and “let’s go for 2x faster, you got this”. Sometimes I brought some ML knowhow like “hey it’s time to consider speculative decoding” or “why is that running on CPU, you should port it”. But for the most part, it was just vibes and I’m pretty sure that with enough time the model would have figured out those things. Also, it seemed that both Astra and Opus (new Fable wasn’t out yet) were pretty capable at this and I could use them interchangeably. I’m sure someone has a good benchmark for which is smarter but both passed my smart-enough threshold.\n\nThe chart follows the measured optimization sequence: **6 → 213 tokens/s**, or **33×** the baseline.\n\nAt one point, around when it got to 10 TPS, I messaged the DeepGrove founders about this, and they replied surprised that it was working so easily, saying they had tried this a few months back and the models were not quite good enough to be particularly effective. They sent me a prompt that summarizes some of their lessons and I copy-pasted it into my session to see if it’d matter. It was essentially the most significant piece of actual guidance I added to this session. The prompt is reproduced below.\n\nhere is what someone else did for a dense bs=1 decode optimized fpga implementation for a dense qwen style model with ternary weights. you should take this with a grain of salt and pick things that you believe are acutally useful to add to your plan. Please also dont do things if you dont think is useful that take a lot of times like runnning really long test of a cycle accurate simulation etc or like physical design since those can take like hours. here are some ideas please pick and choose use this as context and not somehting that you shoudl be strictly guided by somethings that could help. prioritiize speed of like getting this up and running though..\n\nThe coding agent diligently considered these asks, but it didn’t seem to make all that much of a difference qualitatively. Indeed, when I asked it later to read all of the logs and determine which of these instructions it actually used, it said:\n\nIts biggest design ideas were already in my plan before you pasted it. You pasted it on Oct 29 at 21:50 UTC: 4,741 characters describing what someone else did for a dense, batch-1, ternary FPGA decoder. Within 70 seconds I had sorted it into ten ideas to adopt and five to reject, and written that into every design agent’s brief. The shorter remark from your friend Edward, about using lookup tables as logic gates, came 24 minutes earlier and left nothing on the speed line.\n\n**Design ideas I adopted: mostly confirmation.**\n\nAlready planned: streaming weights straight into the compute lanes, running rows in parallel, and reading the attention cache in bursts were all in the design brief I’d launched 35–40 minutes earlier.\n\n**Ideas I rejected: all five numerics shortcuts.**\n Lower-precision activations, a deferred norm, an 8-bit KV cache, online softmax and 16-bit probabilities each change the output bits. That is where their design got its efficiency, and your bit-exact rule ruled it out.\n\n**Process advice: this changed things right away.**\n\nWhat you asked: not to do things that take hours.\n\nWhat I stopped: a simulation that had run for about 24 hours, and two 250 MHz builds.\n\nWhat I switched to: a short loop of C tests, synthesis and a logic-depth check, with one place-and-route per integration.\n\nLikely gain: it got hardware results sooner.\n\nPossible cost, which I can’t prove: two hangs that only RTL simulation could show reached the board the next day, and the big integration lost 38 hours to them.\n\n**One real find it triggered.**\n While checking their memory ideas, I found AWS’s own reference memory design, which reaches 426 GB/s across all 32 memory channels. The memory subsystem was built from it.\n\nIn other words, for most of the actual design ideas it is unclear but there is good reason to believe the model would have figured them out, especially because it came up with them many cycles later after several compaction cycles. But the process advice was useful immediately for getting to hardware results sooner and increasing iteration speed. In other words, the humans are still useful at defining how to iterate fast and what correctness looks like, and the agents are great to manage the exploration.\n\nHere are the changes the autoresearch loop found. Interestingly, it’s mostly a series of modest improvements that compound 1.5x ish every time, rather than one or two giant ones.\n\nBaseline: the first full-model bit-exact FPGA image (R8), served with host-side prefix caching.\n\n1.0× from baseline\n\nHost software only: per-command overhead fell from ~10 ms to 0.9 ms (persistent register mapping, streaming reads, no per-command SDK calls or fsyncs).\n\n1.25× this step · 1.25× from baseline\n\nA 128-lane attention engine that reads the KV cache in bursts instead of a word at a time, keeping the reference's exact summation order.\n\n1.74× this step · 2.18× from baseline\n\nA parallel MoE router with an exact GCC-sort tie-breaker plus a streaming output head with built-in argmax; moderate-effort routing let it run at 250 MHz.\n\n1.46× this step · 3.19× from baseline\n\nA glue engine that parallelizes norms, RoPE, quantization and copies between engines in certified exact orders.\n\n1.47× this step · 4.70× from baseline\n\nThe full new pipeline (streaming ternary projection lanes + a new sequencer replacing R8's control path), after fixing two hangs that only appeared in hardware.\n\n1.63× this step · 7.65× from baseline\n\nExact speculative decoding: the host drafts tokens by prompt lookup and the FPGA verifies several in one pass.\n\n1.04× this step · 7.97× from baseline\n\nFixed a relay that moved router weights between dies at 1/6 speed: router stall per layer 43,862 → 12,292 cycles.\n\n1.14× this step · 9.08× from baseline\n\nHead shadow v2 (start the next token on a provisional guess; catch misses early) plus cross-die timing fixes, enabling 237.5 MHz kernel / 450 MHz memory.\n\n1.62× this step · 14.68× from baseline\n\nExact speculative decoding with a real draft model: a cut-down Maple on the F2's host CPUs drafts ahead, and the FPGA verifies every token.\n\n1.12× this step · 16.42× from baseline\n\nDoubled the memory ports from 4 to 8 (two projection lanes, output head striped across banks): 1.39× per clock, but congestion cut the clock to 181/375 MHz.\n\n1.03× this step · 16.91× from baseline\n\nHid the attention engine's KV fetches behind its compute, and replaced the router's 2,048-step serial sum with an exact order-independent one (12,292 → 2,338 cycles per layer).\n\n1.24× this step · 20.98× from baseline\n\nLossless 5-trit weight packing (−20% weight bytes), shorter serial steps, and moving the output head to the far die, which relieved congestion and allowed 225/400 MHz.\n\n1.27× this step · 26.63× from baseline\n\nRe-speculation after head-shadow misses, speculative MoE expert prefill during the router, and a die rebalance (without the unused DDR controller) that ran 231.25/450 MHz on both boots.\n\n1.08× this step · 28.63× from baseline\n\nA head loader that reads both of its memory banks at once (+17.3%, measured on and off on one image), a runtime expert-prefetch policy, and the router-job timing fix; run one clock step lower (225/425 MHz) after a soak test found random faults at 231.25.\n\n1.16× this step · 33.13× from baseline\n\nIt’s difficult to do a 1:1 comparison across these configurations. I didn’t run the same autoresearch loop on every alternative, and I assume there is more performance to extract if I did. The point is to show how slow the FPGA was in its first implementation, and how far the optimization loop took it.\n\nThe measurements below are from November 15 on AWS, replaying the same **8-chat × 3-turn benchmark**. Prices are the on-demand rates recorded with that experiment. “Exact turns” counts outputs that matched the pinned CPU reference. The GPU implementations are included for context but did not satisfy the same bit-exact criterion but are pretty close in practice.\n\n| What | Instance | Price per hour | Bench tok/s | Exact turns | FPGA is faster by | \n|---|---|---|---|---|---|\n| F2 FPGA, r9 (final image) | f2.6xlarge | $1.98 | 213.13 | 24/24 | — | \n| Exact CPU twin, 32 threads | m8a.8xlarge (32 Zen 5 cores) | $1.95 | 129.5 | 24/24 | 1.65× | \n| Exact CPU twin, 24 threads | c7i.12xlarge (24 Sapphire Rapids cores) | $2.14 | 99.6 | 24/24 | 2.14× | \n| Pinned scalar reference, all 32 cores | m8a.8xlarge | $1.95 | 40.6 | 24/24 | 5.25× | \n| Pinned scalar reference as pinned, 1 thread | r7i.2xlarge | $0.53 | 1.1 | 24/24 | 195× | \n| Exact CPU twin, 4 threads (small instance) | m8a.xlarge (4 Zen 5 cores) | $0.24 | 67.6 | 24/24 | 3.16× | \n| DeepGrove’s own fork with GPU offload (not bit-exact, shown for reference) | g4dn.xlarge (T4 GPU) | $0.53 | 35.7 | 1/24 | 5.97× | \n| Community CUDA port (not bit-exact, shown for reference) | g4dn.xlarge (T4 GPU) | $0.53 | 166.3 | 0/24 | 1.28× | \n\nThis is one generation starting with a 16-token prompt and continuing until the 8,192-token context is full: **8,176 generated tokens**. The five middle columns show decode speed in **tokens/s** at the indicated context length for several generations of iteration. The last column is wall-clock time for the whole run/\n\n| Image | 16–64 | 1–2k | 2–4k | 4–6k | 6–8k | Whole run (s) | \n|---|---|---|---|---|---|---|\n| r5 | 108 | 70 | 57 | 45 | 37 | 162.3 s | \n| r6b | 142 | 111 | 90 | 72 | 60 | 102.3 s | \n| r7 | 181 | 140 | 111 | 88 | 72 | 83.6 s | \n| r8 | 186* | 155* | 121 | 94 | 77 | 77.6 s | \n| r9 | 233* | 151* | 120 | 93 | 76 | 77.8 s | \n\n* For r8 and r9, the first two columns come from separate, shorter runs generating 2,032 tokens. The other columns come from the full-context run.\n\nr9 improves short-context speed, while the complete long-context run is roughly tied with r8.\n\nThese examples come from the final r9 image.\n\n```\nUSER: What is 37 × 48? Explain the steps briefly.\n\nMODEL: <think>\nWe need to compute 37 × 48. The user asks to explain steps briefly. So I'll do multiplication: 37 × 48 = 37 × (50 - 2) = 37×50 - 37×2 = 1850 - 74 = 1776. Or directly: 37×40=1480, 37×8=296, sum=1776. I'll present a brief explanation.\n</think>\n\nTo multiply 37 by 48, you can break it down:\n\n- 37 × 48 = 37 × (50 – 2) = (37 × 50) – (37 × 2) = 1850 – 74 = 1776.\n\nAlternatively, 37 × 40 = 1480 and 37 × 8 = 296, so 1480 + 296 = 1776.\n\nThe final answer is **1776**.\n```\n\nSystem prompt: “You are a helpful assistant. Think briefly before you answer, then reply clearly and concisely.”\n\n```\nUSER: Write a Python function is_palindrome(s) that ignores case, spaces and punctuation.\n\nMODEL: <think>\nWe need to write a Python function is_palindrome(s) that ignores case, spaces, and punctuation. So we need to filter out non-alphanumeric characters, convert to lowercase, then check if the resulting string is equal to its reverse. Provide the code.\n</think>\n\n``` python\ndef is_palindrome(s: str) -> bool:\n    \"\"\"\n    Returns True if the string s is a palindrome, ignoring case, spaces, and punctuation.\n    \"\"\"\n    # Filter to keep only alphanumeric characters and convert to lowercase\n    filtered = ''.join(ch.lower() for ch in s if ch.isalnum())\n    # Compare with its reverse\n    return filtered == filtered[::-1]\nThis was a fun project. I’m not sure I’m ever going to use this specific chip architecture in this specific way, and I’m certainly not claiming that an 8,192-token-context model running on an FPGA is the way I’d recommend anyone use something like this in production. But it’s pretty epic that this *just works*. Capability like this completely shifts the overton window for what a committed team of engineers can work on. It should mean that entirely new types of ideas (scientific discoveries, products, startups) are possible if you can point the models in the right direction. I’m excited to see what everyone builds.", "url": "https://wpnews.pro/news/vibecoding-an-inference-chip", "canonical_source": "https://ankitg.me/blog/vibecoding-inference-chip/", "published_at": "2026-11-20 00:00:00+00:00", "updated_at": "2026-11-20 05:43:44.135326+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "ai-chips", "developer-tools", "ai-research"], "entities": ["AWS", "Conductor", "Anthropic", "Claude", "OpenAI", "Codex", "DeepGrove", "Opus"], "also_reported_by": [], "takeaway": "A cloud coding agent took a ternary-weight model on a 16 GB-HBM AWS F2 FPGA from a bit-exact baseline to 213 tokens/s, a 33× speedup, for $10,000 in Claude and OpenAI tokens across hundreds of experiments.", "recommendation": {"advice": "Point a cloud coding agent at FPGA inference optimization in long autoresearch loops; it reached 33× speedup (6 to 213 tokens/s) on a 16 GB-HBM AWS F2 for $10,000 in tokens.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/vibecoding-an-inference-chip", "markdown": "https://wpnews.pro/news/vibecoding-an-inference-chip.md", "text": "https://wpnews.pro/news/vibecoding-an-inference-chip.txt", "jsonld": "https://wpnews.pro/news/vibecoding-an-inference-chip.jsonld"}}