{"slug": "show-hn-carmen-testing-ai-written-gpu-kernels", "title": "Show HN: carmen – testing AI-written GPU kernels", "summary": "A tool called carmen, built by an independent developer and posted to Hacker News, caught all 75 realistic bugs planted in AI-written Metal GPU kernels, while the standard single-input correctness check caught only 40. On an M4 MacBook Pro (16 GB), Claude-written fused kernels beat Apple's MLX by 1.82× on masked softmax and 1.41× on add + rmsnorm, but inside real Qwen 2.5 4-bit models the gains were +2–3% at 0.5B (214 to 220 tok/s) and negative at 1.5B (94 to 87 tok/s) and 3B (44 to 42 tok/s), because MLX already runs close to the Mac's memory-bandwidth ceiling. carmen measured a larger real-model gain from speculative decoding, an existing mlx-lm feature, at 1.48× on Qwen 3B.", "body_md": "**An LLM writes GPU kernels for Apple Silicon and carmen tests every one. Then it measures whether a real model on a Mac actually gets faster.**\n\n[TL;DR](#tldr) ·\n  [The CLI](#the-cli) ·\n  [**Try it**](#try-it) ·\n  [The problem](#the-problem) ·\n  [How it works](#how-carmen-works) ·\n  [Results](#results) ·\n  [What I learned](#what-i-learned)\n\n- **Most AI-kernel results are graded by a check that misses a lot.** I planted 75 realistic bugs in GPU kernels: the usual check caught 40. carmen's tests caught all 75.\n- **The kernels that passed were also faster.** Claude's masked softmax ran**1.82× faster** than Apple's MLX (MLX does it in 3 steps, Claude's in 1), and add + rmsnorm**1.41×** .\n- **But in a real model they barely mattered:** +2–3% on Qwen 0.5B, nothing on bigger models, because MLX already reads the model's weights about as fast as the Mac allows. What did speed up bigger models was speculative decoding, an existing mlx-lm feature carmen measured: 1.48× on Qwen 3B.\n\n**The home screen:** what's been proven, the models you can speed up, and the kernels carmen has written.\n\n**A model's page** (Qwen 2.5 3B): everything measured on it, and what you can run.\n\nRun `carmen`, press `m`, pick a model: it shows what's been measured on it and what you can run, no API key needed.\n\nLLMs can write GPU kernels, but a kernel can pass a quick test and still be wrong. Most AI-kernel benchmarks try a\nsingle input and accept any answer that's roughly close. So a kernel that works on 1,024 numbers but forgets the last\none when there are 1,025, or one whose answers are slightly off, still counts as correct\n([one study found this kind of check misses most precision bugs](https://arxiv.org/abs/2609.22220)).\n\nAnd even a kernel that really is correct and faster on its own may not make the model any faster. I wanted to know if an LLM could make a model on my Mac faster, and prove it.\n\nAt its core it's a loop: Claude writes a Metal kernel, carmen tests it, and the fastest correct one is kept and improved. Plenty of projects do that. The difference is that carmen doesn't trust the loop, because a plain loop fools itself in three ways:\n\n- **The tests miss bugs.** So carmen's tests are tested: I planted 75 realistic bugs in correct kernels.\ncarmen caught all 75; the usual one-input check caught 40. Failures tell Claude exactly where it went wrong.\n- **The model learns to pass the tests instead of being correct.** So part of the test set is hidden: inputs Claude\nnever sees, for correctness and for speed. Without that,[30% of LLM kernel \"wins\" on Metal broke](https://arxiv.org/abs/2608.08722) on inputs nobody had tested.\n- **\"Faster\" doesn't mean the model gets faster.** So every win is timed against MLX's own fusion with error bars,\nthen checked again inside a real model. That's how carmen found that kernels which win on their own barely move Qwen at all.\n\nEverything here ran on my M4 MacBook Pro (16 GB). Numbers in brackets are the range the real value is 95% likely to be in.\n\n**Single kernels.** Claude beat MLX where MLX splits the work into several small steps. On matmul and attention, which Apple has already tuned heavily, it lost:\n\n| kernel | MLX | carmen | speedup | \n|---|---|---|---|\n| masked softmax | 3 kernels | 1 fused kernel | **1.82×** | \n| add + rmsnorm | 2 kernels | 1 fused kernel | **1.41×** | \n| matmul | Apple's tuned kernel | 1 kernel | 0.93× | \n| attention | Apple's tuned kernel | 1 kernel | 0.54× | \n\nAgainst MLX's own automatic fusion (`mx.compile`), the first two are still 1.38× and 1.43×.\n\n**In real models,** carmen's fused MLP kernels (tuned on 0.5B) mostly disappear, because MLX is already close to the fastest the Mac can go:\n\n| Qwen 2.5, 4-bit | MLX | carmen | speedup | fastest possible* | \n|---|---|---|---|---|\n| 0.5B | 214 tok/s | 220 tok/s | **1.02×** [1.01–1.02] | ~337 tok/s | \n| 1.5B | 94 tok/s | 87 tok/s | 0.93× [0.93–0.95] | ~108 tok/s | \n| 3B | 44 tok/s† | 42 tok/s | 0.95× [0.93–0.97] | ~53 tok/s | \n\n<sub>*Each new word means reading all of the model's weights from memory once. My M4 reads about 94 GB/s, so a 1.74 GB\nmodel (3B) can't go much past ~53 words per second, however good the kernels are. On a quiet Mac MLX runs 3B at ~52 of 53; on 0.5B it's\nat 214 of 337, which is where carmen's kernels still had room.\n†This 3B run was on a busier Mac, so both sides are slower. The speedup compares them run by run, so it still holds.</sub>\n\n**Why fusion barely helped: Amdahl's law.** If you speed up only part of the work, the whole job gets faster only as\nmuch as that part allows: speeding up a fraction *p* of the time by *s* gives an overall speedup of 1 / ((1 − *p*) + *p*/*s*).\n\n- On **0.5B** , about 36% of each word's time is the part kernels can shrink (launches, small steps). Even making it free\nwould cap out around 1.56×. carmen's kernels got 1.02×.\n- On **3B** , that part is about 3%. Even infinitely fast kernels would give at most 1 / 0.97 ≈**1.03×** .\n\n**What did make bigger models faster: speculative decoding.**\n\n- A small model guesses the next few words, and the big model checks them all in one go.\n- So the big model reads its weights once for several words instead of once per word.\n- It's a feature mlx-lm already has. carmen just measured it (`carmen levers` ).\n- carmen also tried compressing the model's memory of the conversation (the KV cache). That made things slower, or broke the output:\n\nDecode speed in tokens per second, stock MLX → with the technique:\n\n| technique | model | 512-token prompt | 8k-token prompt | output | \n|---|---|---|---|---|\n| speculative decoding | 3B | 51.8 → 64.7 · **1.24×** [1.20–1.28] | 44.5 → 66.0 · **1.48×** [1.46–1.53] | same as stock | \n|  | 7B | 24.1 → 28.6 · **1.19×** [1.18–1.22] | 22.1 → 24.8 · **1.13×** [1.12–1.15] | same as stock | \n| 8-bit KV cache | 3B | 51.8 → 48.7 · 0.93× [0.93–0.94] | 44.5 → 37.8 · 0.86× [0.83–0.89] | 98–99% same | \n|  | 7B | 24.1 → 23.6 · 0.98× [0.98–0.98] | 22.1 → 20.3 · 0.92× [0.91–0.92] | 95–98% same | \n| 4-bit KV cache | 3B |  |  | broken | \n\n## Notes on these numbers\n\n- Speculative decoding beats the speed limit (66 vs ~46 tok/s on 3B at 8k) because it reads the weights once for several words. It helps less on 7B because the 0.5B draft is accepted less often (46–52% of words vs 57–71%).\n- The 8-bit KV cache is slower because MLX runs attention on it as four separate steps instead of one fused kernel. That's carmen's next target.\n- The 4-bit KV cache turns Qwen's output into gibberish (\"The skyuser\\nquiv\"): its attention keys have outliers 4 bits can't hold.\n- The \"fastest possible\" figures use my measured ~94 GB/s, which is a bit low (7B actually runs slightly faster than its figure), so treat them as approximate.\n- Spec output matches stock until a late fp16 rounding flip, as expected when the big model checks words in a batch.\n\n- AI kernels don't beat hand-tuned code. They beat code nobody fused.\n- A kernel that wins alone can lose inside a model.\n- Past ~1B parameters, memory is the limit, not kernels.\n- Getting numbers I could trust took more work than writing the kernels.\n\nNew to this? [How it all works](https://github.com/utsav1033/carmen/blob/main/docs/how-it-works.md) explains LLMs, GPUs and kernels from scratch.\n\nOn an Apple Silicon Mac:\n\n```\npip install carmen-kernels          # or: uv tool install carmen-kernels\ncarmen                              # the app\ncarmen speedup                      # Qwen 2.5 0.5B, stock vs carmen's kernels, on your Mac (no API key)\ncarmen speedup qwen0.5b qwen3b      # several models in turn, then one table side by side\ncarmen levers qwen3b                # speculative decoding, KV cache quantization, 3-bit weights vs stock (no API key)\ncarmen key                          # only to cook new kernels: paste an Anthropic key once, saved for you only\n```\n\n## All commands\n\n| command | what it does | \n|---|---|\n| `carmen` | the terminal app: what's proven, speed up a model, pick a kernel, watch it cook | \n| `carmen speedup [models…]` | stock MLX vs `mx.compile` vs carmen's bundled kernels on real models, taking turns; no API key (`--quick` for ~2 min a model) | \n| `carmen levers [model]` | speculative decoding, 8/4-bit KV cache and 3-bit weights vs stock, at 512 and 8k prompts, with drift scored for the lossy ones; no API key | \n| `carmen run <op> [--for qwen0.5b]` | let Claude write one kernel; `--for` tests it at a real model's sizes | \n| `carmen broken <op>` | plant bugs and check carmen's tests catch every one, next to the usual one-size check | \n| `carmen judge <op> <file>` | test a kernel you wrote | \n| `carmen e2e [model]` | run a real model stock vs with carmen's kernels: speed, noise range, same answers? | \n| `carmen profile [model]` | where a model spends its time, step by step | \n| `carmen bench` | the feedback loop vs plain best-of-N, same budget | \n\nFrom source: `git clone https://github.com/utsav1033/carmen`, then `pip install -e '.[dev]'`. Keys go in `carmen key`, a `.env` in the folder you run from, or your shell; the shell wins.\n\n[Measuring the Checker](https://arxiv.org/abs/2609.22220) · [Gaming Without an Attacker](https://arxiv.org/abs/2608.08722) · [Metal-Sci](https://arxiv.org/abs/2605.09708) · [Hacker-Fixer Loops](https://arxiv.org/abs/2606.08960) · [Test-Input Generation for Tensor Programs](https://arxiv.org/abs/2606.27396) · [Reward Hacking in Self-Improving Code Agents](https://openreview.net/forum?id=ikrQWGgxYg) · [Meta-Harness](https://arxiv.org/abs/2603.28052) · [Auditing Harness Tampering](https://arxiv.org/abs/2609.00069) · [SoL-Pi](https://arxiv.org/abs/2609.20519) · [SAGE](https://arxiv.org/abs/2609.35568) · [Prime Agent](https://arxiv.org/abs/2608.23552) · [KernelBench-Verified](https://github.com/facebookresearch/kernel_bench_verified) · [Contract-Grade Verifier](https://github.com/RishiShah99/lethe) · [Building a C compiler with parallel Claudes](https://www.anthropic.com/engineering/building-c-compiler)\n\n<sub>Named after Carmen \"Carmy\" Berzatto. Banner drawing used with the artist's permission: fan art, not affiliated with Jeremy Allen White or *The Bear*. Lettered in Caveat (SIL Open Font License). Diagrams drawn with rough.js in Virgil (Excalidraw's font, OFL).</sub>", "url": "https://wpnews.pro/news/show-hn-carmen-testing-ai-written-gpu-kernels", "canonical_source": "https://github.com/utsav1033/carmen", "published_at": "2026-10-27 14:30:45+00:00", "updated_at": "2026-10-27 14:41:53.551331+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-research", "developer-tools", "ai-infrastructure"], "entities": ["carmen", "Claude", "Apple", "MLX", "Qwen 2.5", "mlx-lm", "Metal", "M4 MacBook Pro"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-carmen-testing-ai-written-gpu-kernels", "markdown": "https://wpnews.pro/news/show-hn-carmen-testing-ai-written-gpu-kernels.md", "text": "https://wpnews.pro/news/show-hn-carmen-testing-ai-written-gpu-kernels.txt", "jsonld": "https://wpnews.pro/news/show-hn-carmen-testing-ai-written-gpu-kernels.jsonld"}}