The frontier of GPQA-Dumb models

Ninjahawk released bongochat, a 125.8-million-parameter language model trained end to end with nanochat on a single RTX 5070 (12GB) GPU in about 2.5 hours with no cloud spend, and it ranks #1 on the GPQA-Dumb Models frontier, barely trailing Gemini. The model scores 30.10% raw on MMLU (6.8% chance-normalized), 14.65% on GPQA Diamond (below the 25% chance floor), 0.15% on GSM8K, and 0.00% on AIME 2025, with the README openly stating it writes fluent English and invents facts with total confidence.

Current Rank 1 Model on the GPQA-Dumb Models Frontier, Barely Trailing Gemini. ⬇ Download the weights · · 📊 Benchmarks benchmarks · 📄 Model card /ninjahawk/bongochat/blob/main/MODEL.md 🧠 Built on nanochat https://github.com/karpathy/nanochat git clone https://github.com/ninjahawk/bongochat.git cd bongochat pip install -r requirements.txt python download weights.py ~320MB from the release python bongochat tui.py That is the whole setup. It runs on CPU if you have no GPU — slower, but it works. Inference needs only torch , tiktoken and numpy : no Triton, no Rust toolchain, no CUDA-specific build. Put the repo directory on your PATH to call it by name from any terminal — bongochat on Windows, the bongochat bash script on macOS and Linux. bongochat interactive TUI bongochat -p "why is the sky blue?" one-shot, prints and exits bongochat -t 0.9 -k 100 temperature / top-k bongochat -m 512 longer replies bongochat -i base -p "Once upon a" the raw pre-SFT base model In the TUI: /help /clear /temp /topk /max /stats /retry /exit . Ctrl-C stops a reply mid-stream without killing the session. bongochat is a complete language model — tokenizer, pretraining, supervised finetuning — trained end to end with nanochat https://github.com/karpathy/nanochat on a single RTX 5070 12GB running Windows. Total cost: about two and a half hours of a desktop GPU and no cloud spend at all. | Params | 125,829,354 | | Architecture | 8 layers, 512 dim, 4 heads, rotary, QK-norm, relu² MLP, per-layer value embeddings | | Context | 2048 tokens | | Vocab | 32,768 BPE, trained on ~2B characters | | Pretraining | 503M tokens, 70 min | | SFT | 789,759 conversations, 68 min | | Throughput | ~119k tok/s training, ~150–195 tok/s inference | It writes fluent English and invents facts with total confidence. That is what a model this size does, and this README will not pretend otherwise. Every bongochat score is measured locally on the complete test set — no sampling, no caps, no cherry-picked subset. Frontier scores are published figures, cited and dated. | benchmark | n | chance | raw | chance-normalized | |---|---|---|---|---| | MMLU | 14,042 | 25.00% | 30.10% | 6.8% | | ARC-Easy | 2,376 | 25.02% | 31.94% | 9.2% | | ARC-Challenge | 1,172 | 25.02% | 26.88% | 2.5% | | GPQA Diamond | 198 | 25.00% | 14.65% | 0.0% | | HumanEval | 164 | 0% | 6.71% | 6.71% | | MATH-500 | 500 | 0% | 2.60% | 2.60% | | GSM8K | 1,319 | 0% | 0.15% | 0.15% | | SWE-bench Verified | 500 | 0% | ≤0.60% | ≤0.60% | | AIME 2025 | 30 | 0% | 0.00% | 0.00% | | ChatCORE | — | — | 0.0508 | — | Four of these benchmarks are 4-way multiple choice, so guessing scores ~25% for free. Always answering "D" on MMLU scores 26.89% — bongochat's 30.10% beats a rock by 3.2 points. Normalized for guessing it reaches 6.8% , against ~92.8% for a frontier model. The honest gap is 13.6× , not the 3.1× the raw numbers suggest. The generative benchmarks do not move between those two charts, because there was never anything to guess between. Those bars were always the truth: GSM8K 0.15% 2 of 1,319 . It emits calculator tool calls but cannot carry a multi-step arithmetic chain. Asked 17 + 25 , it answered 28 . AIME 2025 0.00% 0 of 30 . SWE-bench ≤0.60%. Only 3 of 500 outputs were even syntactically valid diffs. A patch that cannot apply cannot pass, so that bounds the true score at essentially zero. GPQA 14.65%, below the chance floor — because it emitted no answer at all 37.4% of the time and picked "A" for 88% of the answers it did give. Base model, before finetuning: val bpb 0.9397 , CORE 0.1034 . GPT-2 1.6B scores 0.2565 CORE — so this reaches roughly 40% of GPT-2 with 8% of the parameters. Asked an AIME problem — find all integer bases b 9 where 17 in base b divides 97 in base b — bongochat opened with: "To find the sum of all integer bases b 9 for which 17ᵦ is a divisor of 97ᵦ, we can use themethod of Lagrange multipliers." It is a divisibility problem solvable in four lines of integer arithmetic. Lagrange multipliers are a continuous-optimization technique requiring derivatives — nothing is being optimized, and divisibility cannot be differentiated. The phrase "find the sum of all X for which Y" merely has the grammatical shape of a constrained optimization problem, so the model reached for the most authoritative-sounding method attached to that shape. That is the model in one sentence. It has learned the register of expertise — the cadence, the vocabulary, the confident "we can use the method of" — with none of the machinery underneath. GitHub rejects any file over 100MB and the model is 320MB. Git LFS's free tier serves 1GB of bandwidth per month, so it would stop working after three clones. The weights are release assets instead — 2GB limit, unlimited bandwidth — and download weights.py fetches them. bongochat tui.py the terminal UI stdlib only bongochat / .cmd launchers for unix / windows download weights.py fetch weights from the release nanochat/ vendored inference runtime MIT, karpathy/nanochat tools/ scripts that regenerate every number and chart here MODEL.md full model card: training config, provenance, all metrics media/ demo recording and benchmark charts models/ weights land here after download gitignored Every figure in this README is reproducible from the published weights — tools/README.md has the commands, the runtimes, and the method caveats. This repo ships inference only . To train, use karpathy/nanochat https://github.com/karpathy/nanochat directly. Reproducing this model: python -m scripts.tok train python -m scripts.base train --depth=8 --window-pattern=L --device-batch-size=16 python -m scripts.chat sft --model-tag=d8 --device-batch-size=16 --window-pattern=L matters on any GPU without FlashAttention 3 kernels including all RTX 50-series : SDPA has no sliding-window support and utilization collapses without it. Upstream is Linux-first and breaks in four places on Windows. The vendored nanochat/ here already includes these fixes; if you use upstream for training you will need them: — no Triton on Windows. Install torch.compile fails triton-windows needs MSVC build tools . Without it the Muon optimizer cannot run at all.imports execution.py resource Unix-only and scrubs the subprocess env down to a Unix PATH ; on Windows Python won't start without SYSTEMROOT .uses engine.py signal.SIGALRM for the calculator timeout, which does not exist on Windows. Needs a thread-based fallback. Peak-FLOPS table has no entry for most consumer cards, so MFU reports as 0%. Not loadable in Ollama or llama.cpp. nanochat's architecture uses per-layer value embeddings, learned resid lambdas / x0 lambdas , and a mid-layer residual "backout". llama.cpp has no graph for any of it, so there is no GGUF conversion path without implementing a new architecture in C++. 2048-token context. Long documents do not fit. English only , and it will confabulate freely about anything factual.The base checkpoint never saw conversation tokens; it continues text rather than answering. -i base does not chat. SWE-bench was not run through the official harness — no Docker, no test execution. The reported figure bounds the real score rather than measuring it. Built on nanochat https://github.com/karpathy/nanochat by Andrej Karpathy MIT . All the hard parts — the architecture, the Muon optimizer, the training loop, the eval harness — are his. This repo is a trained checkpoint, a terminal UI, and the Windows fixes needed to get there. Frontier comparison figures via TokenCalculator, LM Council and PricePerToken, July 2026. MIT licensed. Weights included. Do whatever you want with it.