{"slug": "the-frontier-of-gpqa-dumb-models", "title": "The frontier of GPQA-Dumb models", "summary": "Ninjahawk released bongochat, a 125.8-million-parameter language model trained end to end with nanochat on a single RTX 5070 (12GB) GPU in about 2.5 hours with no cloud spend, and it ranks #1 on the GPQA-Dumb Models frontier, barely trailing Gemini. The model scores 30.10% raw on MMLU (6.8% chance-normalized), 14.65% on GPQA Diamond (below the 25% chance floor), 0.15% on GSM8K, and 0.00% on AIME 2025, with the README openly stating it writes fluent English and invents facts with total confidence.", "body_md": "*Current Rank #1 Model on the GPQA-Dumb Models Frontier, Barely Trailing Gemini.*\n\n** ⬇ Download the weights** ·\n\n**·**\n\n[📊 Benchmarks](#benchmarks)**·**\n\n[📄 Model card](/ninjahawk/bongochat/blob/main/MODEL.md)\n\n[🧠 Built on nanochat](https://github.com/karpathy/nanochat)\n\n```\ngit clone https://github.com/ninjahawk/bongochat.git\ncd bongochat\npip install -r requirements.txt\npython download_weights.py     # ~320MB from the release\npython bongochat_tui.py\n```\n\nThat is the whole setup. It runs on CPU if you have no GPU — slower, but it works.\nInference needs only `torch`\n\n, `tiktoken`\n\nand `numpy`\n\n: no Triton, no Rust toolchain,\nno CUDA-specific build.\n\nPut the repo directory on your `PATH`\n\nto call it by name from any terminal — `bongochat`\n\non Windows, the `bongochat`\n\nbash script on macOS and Linux.\n\n```\nbongochat                              interactive TUI\nbongochat -p \"why is the sky blue?\"    one-shot, prints and exits\nbongochat -t 0.9 -k 100                temperature / top-k\nbongochat -m 512                       longer replies\nbongochat -i base -p \"Once upon a\"     the raw pre-SFT base model\n```\n\nIn the TUI: `/help`\n\n`/clear`\n\n`/temp`\n\n`/topk`\n\n`/max`\n\n`/stats`\n\n`/retry`\n\n`/exit`\n\n.\nCtrl-C stops a reply mid-stream without killing the session.\n\nbongochat is a complete language model — tokenizer, pretraining, supervised finetuning —\ntrained end to end with [nanochat](https://github.com/karpathy/nanochat) on a single\n**RTX 5070 (12GB)** running Windows. Total cost: about two and a half hours of a desktop\nGPU and no cloud spend at all.\n\n| Params | 125,829,354 |\n| Architecture | 8 layers, 512 dim, 4 heads, rotary, QK-norm, relu² MLP, per-layer value embeddings |\n| Context | 2048 tokens |\n| Vocab | 32,768 BPE, trained on ~2B characters |\n| Pretraining | 503M tokens, 70 min |\n| SFT | 789,759 conversations, 68 min |\n| Throughput | ~119k tok/s training, ~150–195 tok/s inference |\n\nIt writes fluent English and invents facts with total confidence. That is what a model this size does, and this README will not pretend otherwise.\n\nEvery bongochat score is measured locally on the **complete test set** — no sampling, no\ncaps, no cherry-picked subset. Frontier scores are published figures, cited and dated.\n\n| benchmark | n | chance | raw | chance-normalized |\n|---|---|---|---|---|\n| MMLU | 14,042 | 25.00% | 30.10% | 6.8% |\n| ARC-Easy | 2,376 | 25.02% | 31.94% | 9.2% |\n| ARC-Challenge | 1,172 | 25.02% | 26.88% | 2.5% |\n| GPQA Diamond | 198 | 25.00% | 14.65% | 0.0% |\n| HumanEval | 164 | 0% | 6.71% | 6.71% |\n| MATH-500 | 500 | 0% | 2.60% | 2.60% |\n| GSM8K | 1,319 | 0% | 0.15% | 0.15% |\n| SWE-bench Verified | 500 | 0% | ≤0.60% | ≤0.60% |\n| AIME 2025 | 30 | 0% | 0.00% | 0.00% |\n| ChatCORE | — | — | 0.0508 | — |\n\nFour of these benchmarks are 4-way multiple choice, so guessing scores ~25% for free.\nAlways answering \"D\" on MMLU scores **26.89%** — bongochat's 30.10% beats a rock by 3.2\npoints. Normalized for guessing it reaches **6.8%**, against ~92.8% for a frontier model.\nThe honest gap is **13.6×**, not the 3.1× the raw numbers suggest.\n\nThe generative benchmarks do not move between those two charts, because there was never anything to guess between. Those bars were always the truth:\n\n**GSM8K 0.15%**(2 of 1,319). It emits calculator tool calls but cannot carry a multi-step arithmetic chain. Asked`17 + 25`\n\n, it answered`28`\n\n.**AIME 2025 0.00%**(0 of 30).** SWE-bench ≤0.60%.**Only 3 of 500 outputs were even syntactically valid diffs. A patch that cannot apply cannot pass, so that bounds the true score at essentially zero.**GPQA 14.65%, below the chance floor**— because it emitted no answer at all 37.4% of the time and picked \"A\" for 88% of the answers it did give.\n\nBase model, before finetuning: **val bpb 0.9397**, **CORE 0.1034**. GPT-2 (1.6B) scores\n0.2565 CORE — so this reaches roughly 40% of GPT-2 with 8% of the parameters.\n\nAsked an AIME problem — find all integer bases `b > 9`\n\nwhere `17`\n\nin base `b`\n\ndivides `97`\n\nin base `b`\n\n— bongochat opened with:\n\n\"To find the sum of all integer bases\n\nb>9 for which 17ᵦ is a divisor of 97ᵦ, we can use themethod of Lagrange multipliers.\"\n\nIt is a divisibility problem solvable in four lines of integer arithmetic. Lagrange\nmultipliers are a continuous-optimization technique requiring derivatives — nothing is\nbeing optimized, and divisibility cannot be differentiated. The phrase *\"find the sum of\nall X for which Y\"* merely has the grammatical shape of a constrained optimization problem,\nso the model reached for the most authoritative-sounding method attached to that shape.\n\nThat is the model in one sentence. It has learned the register of expertise — the cadence,\nthe vocabulary, the confident *\"we can use the method of\"* — with none of the machinery\nunderneath.\n\nGitHub rejects any file over 100MB and the model is 320MB. Git LFS's free tier serves 1GB\nof bandwidth per month, so it would stop working after three clones. The weights are\n**release assets** instead — 2GB limit, unlimited bandwidth — and `download_weights.py`\n\nfetches them.\n\n```\nbongochat_tui.py      the terminal UI (stdlib only)\nbongochat / .cmd      launchers for unix / windows\ndownload_weights.py   fetch weights from the release\nnanochat/             vendored inference runtime (MIT, karpathy/nanochat)\ntools/                scripts that regenerate every number and chart here\nMODEL.md              full model card: training config, provenance, all metrics\nmedia/                demo recording and benchmark charts\nmodels/               weights land here after download (gitignored)\n```\n\nEvery figure in this README is reproducible from the published weights — `tools/README.md`\n\nhas the commands, the runtimes, and the method caveats.\n\nThis repo ships **inference only**. To train, use\n[karpathy/nanochat](https://github.com/karpathy/nanochat) directly. Reproducing this model:\n\n```\npython -m scripts.tok_train\npython -m scripts.base_train --depth=8 --window-pattern=L --device-batch-size=16\npython -m scripts.chat_sft   --model-tag=d8 --device-batch-size=16\n```\n\n`--window-pattern=L`\n\nmatters on any GPU without FlashAttention 3 kernels (including all\nRTX 50-series): SDPA has no sliding-window support and utilization collapses without it.\n\nUpstream is Linux-first and breaks in four places on Windows. The vendored `nanochat/`\n\nhere\nalready includes these fixes; if you use upstream for training you will need them:\n\n— no Triton on Windows. Install`torch.compile`\n\nfails`triton-windows`\n\n(needs MSVC build tools). Without it the Muon optimizer cannot run at all.imports`execution.py`\n\n`resource`\n\n(Unix-only) and scrubs the subprocess env down to a Unix`PATH`\n\n; on Windows Python won't start without`SYSTEMROOT`\n\n.uses`engine.py`\n\n`signal.SIGALRM`\n\nfor the calculator timeout, which does not exist on Windows. Needs a thread-based fallback.**Peak-FLOPS table** has no entry for most consumer cards, so MFU reports as 0%.\n\n**Not loadable in Ollama or llama.cpp.** nanochat's architecture uses per-layer value embeddings, learned`resid_lambdas`\n\n/`x0_lambdas`\n\n, and a mid-layer residual \"backout\". llama.cpp has no graph for any of it, so there is no GGUF conversion path without implementing a new architecture in C++.**2048-token context.** Long documents do not fit.**English only**, and it will confabulate freely about anything factual.The base checkpoint never saw conversation tokens; it continues text rather than answering.`-i base`\n\ndoes not chat.**SWE-bench was not run through the official harness**— no Docker, no test execution. The reported figure bounds the real score rather than measuring it.\n\nBuilt on [nanochat](https://github.com/karpathy/nanochat) by Andrej Karpathy (MIT). All the\nhard parts — the architecture, the Muon optimizer, the training loop, the eval harness —\nare his. This repo is a trained checkpoint, a terminal UI, and the Windows fixes needed to\nget there. Frontier comparison figures via TokenCalculator, LM Council and PricePerToken,\nJuly 2026.\n\nMIT licensed. Weights included. Do whatever you want with it.", "url": "https://wpnews.pro/news/the-frontier-of-gpqa-dumb-models", "canonical_source": "https://github.com/ninjahawk/bongochat", "published_at": "2026-08-04 07:17:37+00:00", "updated_at": "2026-08-04 07:53:03.499523+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-products"], "entities": ["Ninjahawk", "bongochat", "nanochat", "RTX 5070", "Gemini", "GPT-2"], "alternates": {"html": "https://wpnews.pro/news/the-frontier-of-gpqa-dumb-models", "markdown": "https://wpnews.pro/news/the-frontier-of-gpqa-dumb-models.md", "text": "https://wpnews.pro/news/the-frontier-of-gpqa-dumb-models.txt", "jsonld": "https://wpnews.pro/news/the-frontier-of-gpqa-dumb-models.jsonld"}}