{"slug": "show-hn-don-t-trust-me-bro-fixing-gpt-oss-3-49b-tokens-1k-gpu-hours-1x3090", "title": "Show HN: Don't trust me bro: fixing GPT-OSS (3.49B tokens, 1k GPU hours, 1x3090)", "summary": "A developer known as iamskeole released burrito, an open-source inference harness and evaluation suite, after running 320,192 questions across 8 seeds and processing 3.49B tokens over 1,062 GPU hours on a single RTX 3090 to test OpenAI's gpt-oss-20b model. The results show the model was not broken, contrary to community criticism, with fixes for tool calling and the Harmony template provided on GitHub and Hugging Face.", "body_md": "Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full 128k context, factory-precision weights, across parallel requests on a single RTX 3090 at close to 200 tps (well... eventually, but it was still flying at around 100 tps initially which was mind blowing in the before-times).\n\nCould and would being two different things, turned out both llama.cpp and vLLM were shitting their pants running the model at the time (love you guys, I know this model was a pita!), particularly around tool calling (vLLM was / is broken seven ways to Sunday), mostly due to the Harmony template introduced by OpenAI (which, coincidentally (?) is almost identically implemented in Gemma 4 and somehwat similar in Muse Glimmer, 9-12 months after the gpt-oss release, so OpenAI was on to something there and likely not just for the OSS release but their bigger and closed siblings too).\n\nAnyway, validating my hypothesis with the vanilla backends proved impossible at the time.\n\nSo I did the only rational thing: built an inference harness that fixes the model, then ran probably the most autistic evals in history -- 320,192 questions across 8 seeds, prefilling and decoding over 3.49B tokens, for 1,062 hours of batch size 1 GPU time on a single 3090.\n\nIn the words of Carl Sagan, to make an apple pie from scratch, you first have to invent the universe. I spent my nights inventing this one in parking lots between food delivery gigs, so I named it burrito.\n\nAll that just to test whether OpenAI shipped a broken model (spoiler: it didn't). Did it work? Here's the hero shots for the final boss of tool calling evals: multi-turn, pass@8 (at least 1 seed of 8) and pass^8 (every seed).\n\n[https://raw.githubusercontent.com/iamskeole/burrito-evals/re...](https://raw.githubusercontent.com/iamskeole/burrito-evals/refs/heads/main/plots/phase_5-f02-pass@8.png)\n> task solve rate on at least one seed\n\n[https://raw.githubusercontent.com/iamskeole/burrito-evals/re...](https://raw.githubusercontent.com/iamskeole/burrito-evals/refs/heads/main/plots/phase_5-f03-pass%5E8.png)\n> task solve rate on every seed\n\nSharing everything, MIT:\n\n- harness: [https://github.com/iamskeole/burrito-core](https://github.com/iamskeole/burrito-core)\n- evals (incl. full inference traces): [https://github.com/iamskeole/burrito-evals](https://github.com/iamskeole/burrito-evals)\n- fixed jinja template: [https://huggingface.co/openai/gpt-oss-20b/discussions/274/fi...](https://huggingface.co/openai/gpt-oss-20b/discussions/274/files)\n\n(Detailed analysis on reasoning zones and \"optimal effort\" levels can be found in the evals repo)\n\nComments URL: [https://news.ycombinator.com/item?id=49523381](https://news.ycombinator.com/item?id=49523381)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-don-t-trust-me-bro-fixing-gpt-oss-3-49b-tokens-1k-gpu-hours-1x3090", "canonical_source": "https://github.com/iamskeole/burrito-core", "published_at": "2026-09-01 15:34:48+00:00", "updated_at": "2026-09-01 15:53:52.007404+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools", "ai-infrastructure"], "entities": ["OpenAI", "gpt-oss-20b", "burrito", "iamskeole", "llama.cpp", "vLLM", "Hugging Face", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/show-hn-don-t-trust-me-bro-fixing-gpt-oss-3-49b-tokens-1k-gpu-hours-1x3090", "markdown": "https://wpnews.pro/news/show-hn-don-t-trust-me-bro-fixing-gpt-oss-3-49b-tokens-1k-gpu-hours-1x3090.md", "text": "https://wpnews.pro/news/show-hn-don-t-trust-me-bro-fixing-gpt-oss-3-49b-tokens-1k-gpu-hours-1x3090.txt", "jsonld": "https://wpnews.pro/news/show-hn-don-t-trust-me-bro-fixing-gpt-oss-3-49b-tokens-1k-gpu-hours-1x3090.jsonld"}}