Show HN: carmen – testing AI-written GPU kernels

A tool called carmen, built by an independent developer and posted to Hacker News, caught all 75 realistic bugs planted in AI-written Metal GPU kernels, while the standard single-input correctness check caught only 40. On an M4 MacBook Pro (16 GB), Claude-written fused kernels beat Apple's MLX by 1.82× on masked softmax and 1.41× on add + rmsnorm, but inside real Qwen 2.5 4-bit models the gains were +2–3% at 0.5B (214 to 220 tok/s) and negative at 1.5B (94 to 87 tok/s) and 3B (44 to 42 tok/s), because MLX already runs close to the Mac's memory-bandwidth ceiling. carmen measured a larger real-model gain from speculative decoding, an existing mlx-lm feature, at 1.48× on Qwen 3B.

An LLM writes GPU kernels for Apple Silicon and carmen tests every one. Then it measures whether a real model on a Mac actually gets faster. TL;DR tldr · The CLI the-cli · Try it try-it · The problem the-problem · How it works how-carmen-works · Results results · What I learned what-i-learned - Most AI-kernel results are graded by a check that misses a lot. I planted 75 realistic bugs in GPU kernels: the usual check caught 40. carmen's tests caught all 75. - The kernels that passed were also faster. Claude's masked softmax ran 1.82× faster than Apple's MLX MLX does it in 3 steps, Claude's in 1 , and add + rmsnorm 1.41× . - But in a real model they barely mattered: +2–3% on Qwen 0.5B, nothing on bigger models, because MLX already reads the model's weights about as fast as the Mac allows. What did speed up bigger models was speculative decoding, an existing mlx-lm feature carmen measured: 1.48× on Qwen 3B. The home screen: what's been proven, the models you can speed up, and the kernels carmen has written. A model's page Qwen 2.5 3B : everything measured on it, and what you can run. Run carmen , press m , pick a model: it shows what's been measured on it and what you can run, no API key needed. LLMs can write GPU kernels, but a kernel can pass a quick test and still be wrong. Most AI-kernel benchmarks try a single input and accept any answer that's roughly close. So a kernel that works on 1,024 numbers but forgets the last one when there are 1,025, or one whose answers are slightly off, still counts as correct one study found this kind of check misses most precision bugs https://arxiv.org/abs/2609.22220 . And even a kernel that really is correct and faster on its own may not make the model any faster. I wanted to know if an LLM could make a model on my Mac faster, and prove it. At its core it's a loop: Claude writes a Metal kernel, carmen tests it, and the fastest correct one is kept and improved. Plenty of projects do that. The difference is that carmen doesn't trust the loop, because a plain loop fools itself in three ways: - The tests miss bugs. So carmen's tests are tested: I planted 75 realistic bugs in correct kernels. carmen caught all 75; the usual one-input check caught 40. Failures tell Claude exactly where it went wrong. - The model learns to pass the tests instead of being correct. So part of the test set is hidden: inputs Claude never sees, for correctness and for speed. Without that, 30% of LLM kernel "wins" on Metal broke https://arxiv.org/abs/2608.08722 on inputs nobody had tested. - "Faster" doesn't mean the model gets faster. So every win is timed against MLX's own fusion with error bars, then checked again inside a real model. That's how carmen found that kernels which win on their own barely move Qwen at all. Everything here ran on my M4 MacBook Pro 16 GB . Numbers in brackets are the range the real value is 95% likely to be in. Single kernels. Claude beat MLX where MLX splits the work into several small steps. On matmul and attention, which Apple has already tuned heavily, it lost: | kernel | MLX | carmen | speedup | |---|---|---|---| | masked softmax | 3 kernels | 1 fused kernel | 1.82× | | add + rmsnorm | 2 kernels | 1 fused kernel | 1.41× | | matmul | Apple's tuned kernel | 1 kernel | 0.93× | | attention | Apple's tuned kernel | 1 kernel | 0.54× | Against MLX's own automatic fusion mx.compile , the first two are still 1.38× and 1.43×. In real models, carmen's fused MLP kernels tuned on 0.5B mostly disappear, because MLX is already close to the fastest the Mac can go: | Qwen 2.5, 4-bit | MLX | carmen | speedup | fastest possible | |---|---|---|---|---| | 0.5B | 214 tok/s | 220 tok/s | 1.02× 1.01–1.02 | ~337 tok/s | | 1.5B | 94 tok/s | 87 tok/s | 0.93× 0.93–0.95 | ~108 tok/s | | 3B | 44 tok/s† | 42 tok/s | 0.95× 0.93–0.97 | ~53 tok/s | <sub Each new word means reading all of the model's weights from memory once. My M4 reads about 94 GB/s, so a 1.74 GB model 3B can't go much past ~53 words per second, however good the kernels are. On a quiet Mac MLX runs 3B at ~52 of 53; on 0.5B it's at 214 of 337, which is where carmen's kernels still had room. †This 3B run was on a busier Mac, so both sides are slower. The speedup compares them run by run, so it still holds.</sub Why fusion barely helped: Amdahl's law. If you speed up only part of the work, the whole job gets faster only as much as that part allows: speeding up a fraction p of the time by s gives an overall speedup of 1 / 1 − p + p / s . - On 0.5B , about 36% of each word's time is the part kernels can shrink launches, small steps . Even making it free would cap out around 1.56×. carmen's kernels got 1.02×. - On 3B , that part is about 3%. Even infinitely fast kernels would give at most 1 / 0.97 ≈ 1.03× . What did make bigger models faster: speculative decoding. - A small model guesses the next few words, and the big model checks them all in one go. - So the big model reads its weights once for several words instead of once per word. - It's a feature mlx-lm already has. carmen just measured it carmen levers . - carmen also tried compressing the model's memory of the conversation the KV cache . That made things slower, or broke the output: Decode speed in tokens per second, stock MLX → with the technique: | technique | model | 512-token prompt | 8k-token prompt | output | |---|---|---|---|---| | speculative decoding | 3B | 51.8 → 64.7 · 1.24× 1.20–1.28 | 44.5 → 66.0 · 1.48× 1.46–1.53 | same as stock | | | 7B | 24.1 → 28.6 · 1.19× 1.18–1.22 | 22.1 → 24.8 · 1.13× 1.12–1.15 | same as stock | | 8-bit KV cache | 3B | 51.8 → 48.7 · 0.93× 0.93–0.94 | 44.5 → 37.8 · 0.86× 0.83–0.89 | 98–99% same | | | 7B | 24.1 → 23.6 · 0.98× 0.98–0.98 | 22.1 → 20.3 · 0.92× 0.91–0.92 | 95–98% same | | 4-bit KV cache | 3B | | | broken | Notes on these numbers - Speculative decoding beats the speed limit 66 vs ~46 tok/s on 3B at 8k because it reads the weights once for several words. It helps less on 7B because the 0.5B draft is accepted less often 46–52% of words vs 57–71% . - The 8-bit KV cache is slower because MLX runs attention on it as four separate steps instead of one fused kernel. That's carmen's next target. - The 4-bit KV cache turns Qwen's output into gibberish "The skyuser\nquiv" : its attention keys have outliers 4 bits can't hold. - The "fastest possible" figures use my measured ~94 GB/s, which is a bit low 7B actually runs slightly faster than its figure , so treat them as approximate. - Spec output matches stock until a late fp16 rounding flip, as expected when the big model checks words in a batch. - AI kernels don't beat hand-tuned code. They beat code nobody fused. - A kernel that wins alone can lose inside a model. - Past ~1B parameters, memory is the limit, not kernels. - Getting numbers I could trust took more work than writing the kernels. New to this? How it all works https://github.com/utsav1033/carmen/blob/main/docs/how-it-works.md explains LLMs, GPUs and kernels from scratch. On an Apple Silicon Mac: pip install carmen-kernels or: uv tool install carmen-kernels carmen the app carmen speedup Qwen 2.5 0.5B, stock vs carmen's kernels, on your Mac no API key carmen speedup qwen0.5b qwen3b several models in turn, then one table side by side carmen levers qwen3b speculative decoding, KV cache quantization, 3-bit weights vs stock no API key carmen key only to cook new kernels: paste an Anthropic key once, saved for you only All commands | command | what it does | |---|---| | carmen | the terminal app: what's proven, speed up a model, pick a kernel, watch it cook | | carmen speedup models… | stock MLX vs mx.compile vs carmen's bundled kernels on real models, taking turns; no API key --quick for ~2 min a model | | carmen levers model | speculative decoding, 8/4-bit KV cache and 3-bit weights vs stock, at 512 and 8k prompts, with drift scored for the lossy ones; no API key | | carmen run <op --for qwen0.5b | let Claude write one kernel; --for tests it at a real model's sizes | | carmen broken <op | plant bugs and check carmen's tests catch every one, next to the usual one-size check | | carmen judge <op <file | test a kernel you wrote | | carmen e2e model | run a real model stock vs with carmen's kernels: speed, noise range, same answers? | | carmen profile model | where a model spends its time, step by step | | carmen bench | the feedback loop vs plain best-of-N, same budget | From source: git clone https://github.com/utsav1033/carmen , then pip install -e '. dev ' . Keys go in carmen key , a .env in the folder you run from, or your shell; the shell wins. Measuring the Checker https://arxiv.org/abs/2609.22220 · Gaming Without an Attacker https://arxiv.org/abs/2608.08722 · Metal-Sci https://arxiv.org/abs/2605.09708 · Hacker-Fixer Loops https://arxiv.org/abs/2606.08960 · Test-Input Generation for Tensor Programs https://arxiv.org/abs/2606.27396 · Reward Hacking in Self-Improving Code Agents https://openreview.net/forum?id=ikrQWGgxYg · Meta-Harness https://arxiv.org/abs/2603.28052 · Auditing Harness Tampering https://arxiv.org/abs/2609.00069 · SoL-Pi https://arxiv.org/abs/2609.20519 · SAGE https://arxiv.org/abs/2609.35568 · Prime Agent https://arxiv.org/abs/2608.23552 · KernelBench-Verified https://github.com/facebookresearch/kernel bench verified · Contract-Grade Verifier https://github.com/RishiShah99/lethe · Building a C compiler with parallel Claudes https://www.anthropic.com/engineering/building-c-compiler <sub Named after Carmen "Carmy" Berzatto. Banner drawing used with the artist's permission: fan art, not affiliated with Jeremy Allen White or The Bear . Lettered in Caveat SIL Open Font License . Diagrams drawn with rough.js in Virgil Excalidraw's font, OFL .</sub