Open Kernel Challenge: reports from the top 50

Aleksey Smolenchuk placed in the top 5 of the Open Kernel Challenge, a two-week competition run by Paradigm with Meta and Andromeda that drew about 2,000 participants to rewrite the GPU kernels behind Meta's Muse Spark Open inference, timed against SGLang, for a share of $100,000 in prizes. The top finisher reached a 2.61× speedup on the decode layer versus SGLang's original 1.00×, while the author's 2.00× speedup landed around 500th place; the second round, serving the whole model, runs until Nov 19 and carries the $100,000. Smolenchuk told the author he launched a fresh Codex worker for every attempt, up to six in parallel, with one integrator combining the wins, because long-lived agents "would stall, repeat narrow experiments, or need nudging.

I participated in the Open Kernel Challenge https://openkernelchallenge.dev , a competition to make Meta’s Muse Spark Open run faster, with $100,000 in prizes. With GPU kernels 2x faster than the original, I landed around 500th place. So I contacted the top 50 to learn the tricks from the best. The challenge is run by Paradigm with Meta and Andromeda; Daniel Gross announced https://x.com/danielgross/status/2113351902402613558 it as “a $100,000 challenge to see who can make Muse Spark run fastest”. For two weeks, about 2,000 people rewrote the GPU kernels behind Muse Spark Open’s inference, timed against SGLang. That was the kernel round. The second round, serving the whole model, is where the $100,000 goes, and it runs until Nov 19. My naive weekend approach was yelling at Claude Code with Opus 5.5 while hiking in Pinnacles https://www.nps.gov/pinn/ . It cost me $150 for an H200 rented through Vast.ai, plus the weekly limits of my Claude Max 20x and ChatGPT Team Premium plans. I threw all my credentials onto a box in Japan and hoped for the best. After the first weekend, I peaked at 22, and I could have climbed higher by allocating more compute and tokens. Clearly, many people achieved better speedups than mine. | Speedup over original | Decode layer | Projections | Attention | MoE | Prefill layer | |---|---|---|---|---|---| | 1 | 2.61× | 1.73× | 3.34× | 2.84× | 2.15× | | 10 | 2.50× | 1.67× | 3.01× | 2.68× | 2.04× | | 50 | 2.40× | 1.60× | 2.81× | 2.53× | 1.95× | | Me ~500th | 2.00× | 1.48× | 2.41× | 2.22× | 1.78× | | Original SGLang | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | Leaderboard as of the kernel round’s close on Nov 5. The organizers are still reviewing submissions https://openkernelchallenge.dev/updates , so ranks may move. On the decode layer https://openkernelchallenge.dev/problems/muse decode layer , the whole top 50 fits within 9%. So on the last day of the kernel round, Nov 4, I asked the top 50 to share their setups. Twelve replied with their agents, models, GPUs and spend; a few of them slipped out of the top 50 later as the board moved. How much you paid to play Most of the out-of-pocket bill was GPU, because tokens came from flat-rate plans Claude Max 20x, ChatGPT Pro, the $500 Codex plan priced far below API rates https://quesma.com/blog/claude-code-pricing-for-enterprise/ . Shishir Pai https://x.com/shishirpai ’s model usage came to about $68k at API cost. I estimated roughly the GPU cost and how far from the winner. The effect of bigger spend on results is smaller than I thought. On inspection, I was the sucker paying $4 an hour for one H200 on Vast.ai, or $8 for two, while preemptible Nebius spot instances are often available for less than $1 an hour https://nebius.com/prices . Modal https://modal.com/pricing and Daytona https://www.daytona.io/pricing offer more granular per-second billing, which can squeeze out more, as agents mostly think, plan and code, and need GPUs only to test. Some reported that NVIDIA Nsight profiling helps a lot, but its kernel profiler, Nsight Compute, needs GPU performance counters. Only some platforms, such as Nebius, AWS and Google Cloud, fully support them; others, like some Lium machines https://docs.lium.io/pod-users/gpu-profiling , support them partially. Container platforms such as RunPod and Vast.ai block them. The intelligence behind agents Almost everyone used frontier models from Anthropic and OpenAI. Nobody ran on Fable 5.1: the one entrant who tried it dropped it for cost and speed. Even with a stock harness, most reports described several agents working together: sub-agents, a small parallel team or a council of reviewer models. Four ran custom harnesses: - Aleksey Smolenchuk https://x.com/lxe 5% launched a fresh Codex worker for every attempt, up to six in parallel, with one integrator combining the wins. Long-lived agents “would stall, repeat narrow experiments, or need nudging”. - Shiyi Cao https://x.com/shiyi c98 4% used a supervisor from Shiyi’s NeurIPS 2026 work https://github.com/caoshiyi/EAT that decides how many of up to six Opus 5.5 agents run in each iteration. - Shishir Pai 6% ran oh-my-pi https://github.com/can1357/oh-my-pi sessions that spawn other sessions, about 1,100 threads in total, while an advisor agent watched threads, costs and the leaderboard and patched the shared rules live. - Colin Snyder https://colinsnyder.com 9% “built my own custom evolutionary search harness”. Three of the four replies closest to 1 4 to 6% behind ran custom harnesses; the fourth didn’t say. There is an edge in custom, but customizing Claude Code/Codex can also get you far. What we can learn from the top 50 On the harness and the loop - “The biggest lesson for me was that the harness and evaluation loop mattered as much as the individual agents.” Aleksey Smolenchuk 5% behind 1 - “The most impactful thing throughout the challenge was the ability to monitor and improve the harness live, based on any problem that occurred.” Shishir Pai 6% behind 1 - “More agents didn’t automatically mean more progress. The hard part was turning their ideas into measured improvements.” UnderAndOver 12% behind 1 - “I honestly don’t know every detail of what happens inside.” meganpark https://x.com/ meganpark 9% behind 1 , on Claude orchestrating its own sub-agents and Codex On cost and compute - “That phase cost about $4.8k in model usage, produced 86.5% of the total gain and brought me to 6 overall.” Then: “Model usage reached about $68k at API cost.” Shishir Pai - “At the higher ranks I think it was just a matter of pure compute, both with agents and with GPU access.” Aamir Ahmed https://github.com/elbourne12345 8% behind 1 On agents and expertise - “You can get very far with no kernel knowledge … Beyond a point though you still need kernel expertise.” George Pantazis 7% behind 1 - “I found a lot of success by having the agent write some basic ‘proof’ kernels that established top-lines for mem bandwidth, compute, latency, etc.” Andrew Pullin https://github.com/apullin 7% behind 1 On models - “Opus did most of the heavy lifting. Fable did not justify its output quality for the cost and speed. GPT Sol and Astra were weak at hill climbing for this challenge, but they did a good job reviewing code.” Shishir Pai - “I tried Codex/Astra several times but it wasn’t effective on this challenge.” George Pantazis And one I can’t verify - “gpt-5.6 does noticeably better work if you randomly put bible quotes into your prompts. Serious.” Andrew Pullin What I would do differently Next time I would rent a preemptible Nebius H200, do all the work there and push to git, so it can resume elsewhere. I would install Nsight and every other perf tool I can find, and fetch all the knowledge the agents can use. Instead of one agent per problem, I would point a Claude Code swarm with distinct roles, Codex as one subagent, at one problem, then switch problems. I would stack several seat-based plans so tokens never run out, and keep it running for the whole competition, checking in from my phone. Where is the alpha for humans Agents will make a big difference in hardware choice. Many companies standardized on NVIDIA because porting kernels to another chip took scarce, expensive specialists. The training moat might stay, but the inference moat will shrink. Challengers compete on speed d-Matrix, Cerebras or price AMD, Google TPUs, AWS Trainium . The largest labs are already on several chips: in its S-1 https://www.sec.gov/Archives/edgar/data/0002130357/000162828026067168/anthropicpbc-sx1.htm , Anthropic says it runs Claude natively on Trainium, TPUs, NVIDIA GPUs and AMD GPUs, even for training. The competition resulted in a ranking, not just a few clusters. Humans made the difference in how they allocated resources, organized the work and sometimes overrode the agents. Though agents produced superior results, a lot of their code will be hard to merge back into SGLang: the organizers say many entries run past 10,000 lines, with separate variants for each tested shape. The agents clearly succeeded at a well-defined, easily measurable goal, but humans still have an edge in balancing trade-offs. What is hard to learn during RL, such as long-term maintainability, is still a blind spot for LLMs. It usually comes with an unclear reward function: how do you score how maintainable an architecture is? Agents speeding up the software that runs AI sounds like recursive self-improvement RSI . To me it feels like self-driving in 2005. A year earlier no car finished the DARPA Grand Challenge; that year five did. Ever since, full self-driving has been a few years away. Twenty-one years later, the vast majority of cars still have drivers. Progress in AI can be extremely fast and extremely slow at once. Predicting S-curves is hard.