Show HN: Don't trust me bro: fixing GPT-OSS (3.49B tokens, 1k GPU hours, 1x3090)

A developer known as iamskeole released burrito, an open-source inference harness and evaluation suite, after running 320,192 questions across 8 seeds and processing 3.49B tokens over 1,062 GPU hours on a single RTX 3090 to test OpenAI's gpt-oss-20b model. The results show the model was not broken, contrary to community criticism, with fixes for tool calling and the Harmony template provided on GitHub and Hugging Face.

Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full 128k context, factory-precision weights, across parallel requests on a single RTX 3090 at close to 200 tps well... eventually, but it was still flying at around 100 tps initially which was mind blowing in the before-times . Could and would being two different things, turned out both llama.cpp and vLLM were shitting their pants running the model at the time love you guys, I know this model was a pita , particularly around tool calling vLLM was / is broken seven ways to Sunday , mostly due to the Harmony template introduced by OpenAI which, coincidentally ? is almost identically implemented in Gemma 4 and somehwat similar in Muse Glimmer, 9-12 months after the gpt-oss release, so OpenAI was on to something there and likely not just for the OSS release but their bigger and closed siblings too . Anyway, validating my hypothesis with the vanilla backends proved impossible at the time. So I did the only rational thing: built an inference harness that fixes the model, then ran probably the most autistic evals in history -- 320,192 questions across 8 seeds, prefilling and decoding over 3.49B tokens, for 1,062 hours of batch size 1 GPU time on a single 3090. In the words of Carl Sagan, to make an apple pie from scratch, you first have to invent the universe. I spent my nights inventing this one in parking lots between food delivery gigs, so I named it burrito. All that just to test whether OpenAI shipped a broken model spoiler: it didn't . Did it work? Here's the hero shots for the final boss of tool calling evals: multi-turn, pass@8 at least 1 seed of 8 and pass^8 every seed . https://raw.githubusercontent.com/iamskeole/burrito-evals/re... https://raw.githubusercontent.com/iamskeole/burrito-evals/refs/heads/main/plots/phase 5-f02-pass@8.png task solve rate on at least one seed https://raw.githubusercontent.com/iamskeole/burrito-evals/re... https://raw.githubusercontent.com/iamskeole/burrito-evals/refs/heads/main/plots/phase 5-f03-pass%5E8.png task solve rate on every seed Sharing everything, MIT: - harness: https://github.com/iamskeole/burrito-core https://github.com/iamskeole/burrito-core - evals incl. full inference traces : https://github.com/iamskeole/burrito-evals https://github.com/iamskeole/burrito-evals - fixed jinja template: https://huggingface.co/openai/gpt-oss-20b/discussions/274/fi... https://huggingface.co/openai/gpt-oss-20b/discussions/274/files Detailed analysis on reasoning zones and "optimal effort" levels can be found in the evals repo Comments URL: https://news.ycombinator.com/item?id=49523381 https://news.ycombinator.com/item?id=49523381 Points: 1 Comments: 0