Current Rank #1 Model on the GPQA-Dumb Models Frontier, Barely Trailing Gemini.
** β¬ Download the weights** Β·
Β·
git clone https://github.com/ninjahawk/bongochat.git
cd bongochat
pip install -r requirements.txt
python download_weights.py # ~320MB from the release
python bongochat_tui.py
That is the whole setup. It runs on CPU if you have no GPU β slower, but it works.
Inference needs only torch
, tiktoken
and numpy
: no Triton, no Rust toolchain, no CUDA-specific build.
Put the repo directory on your PATH
to call it by name from any terminal β bongochat
on Windows, the bongochat
bash script on macOS and Linux.
bongochat interactive TUI
bongochat -p "why is the sky blue?" one-shot, prints and exits
bongochat -t 0.9 -k 100 temperature / top-k
bongochat -m 512 longer replies
bongochat -i base -p "Once upon a" the raw pre-SFT base model
In the TUI: /help
/clear
/temp
/topk
/max
/stats
/retry
/exit
. Ctrl-C stops a reply mid-stream without killing the session.
bongochat is a complete language model β tokenizer, pretraining, supervised finetuning β trained end to end with nanochat on a single RTX 5070 (12GB) running Windows. Total cost: about two and a half hours of a desktop GPU and no cloud spend at all.
| Params | 125,829,354 | | Architecture | 8 layers, 512 dim, 4 heads, rotary, QK-norm, reluΒ² MLP, per-layer value embeddings | | Context | 2048 tokens | | Vocab | 32,768 BPE, trained on ~2B characters | | Pretraining | 503M tokens, 70 min | | SFT | 789,759 conversations, 68 min | | Throughput | ~119k tok/s training, ~150β195 tok/s inference |
It writes fluent English and invents facts with total confidence. That is what a model this size does, and this README will not pretend otherwise.
Every bongochat score is measured locally on the complete test set β no sampling, no caps, no cherry-picked subset. Frontier scores are published figures, cited and dated.
| benchmark | n | chance | raw | chance-normalized |
|---|---|---|---|---|
| MMLU | 14,042 | 25.00% | 30.10% | 6.8% |
| ARC-Easy | 2,376 | 25.02% | 31.94% | 9.2% |
| ARC-Challenge | 1,172 | 25.02% | 26.88% | 2.5% |
| GPQA Diamond | 198 | 25.00% | 14.65% | 0.0% |
| HumanEval | 164 | 0% | 6.71% | 6.71% |
| MATH-500 | 500 | 0% | 2.60% | 2.60% |
| GSM8K | 1,319 | 0% | 0.15% | 0.15% |
| SWE-bench Verified | 500 | 0% | β€0.60% | β€0.60% |
| AIME 2025 | 30 | 0% | 0.00% | 0.00% |
| ChatCORE | β | β | 0.0508 | β |
Four of these benchmarks are 4-way multiple choice, so guessing scores ~25% for free. Always answering "D" on MMLU scores 26.89% β bongochat's 30.10% beats a rock by 3.2 points. Normalized for guessing it reaches 6.8%, against ~92.8% for a frontier model. The honest gap is 13.6Γ, not the 3.1Γ the raw numbers suggest.
The generative benchmarks do not move between those two charts, because there was never anything to guess between. Those bars were always the truth:
GSM8K 0.15%(2 of 1,319). It emits calculator tool calls but cannot carry a multi-step arithmetic chain. Asked17 + 25
, it answered28
.AIME 2025 0.00%(0 of 30).** SWE-bench β€0.60%.**Only 3 of 500 outputs were even syntactically valid diffs. A patch that cannot apply cannot pass, so that bounds the true score at essentially zero.GPQA 14.65%, below the chance floorβ because it emitted no answer at all 37.4% of the time and picked "A" for 88% of the answers it did give.
Base model, before finetuning: val bpb 0.9397, CORE 0.1034. GPT-2 (1.6B) scores 0.2565 CORE β so this reaches roughly 40% of GPT-2 with 8% of the parameters.
Asked an AIME problem β find all integer bases b > 9
where 17
in base b
divides 97
in base b
β bongochat opened with:
"To find the sum of all integer bases
b>9 for which 17ᡦ is a divisor of 97ᡦ, we can use themethod of Lagrange multipliers."
It is a divisibility problem solvable in four lines of integer arithmetic. Lagrange multipliers are a continuous-optimization technique requiring derivatives β nothing is being optimized, and divisibility cannot be differentiated. The phrase "find the sum of all X for which Y" merely has the grammatical shape of a constrained optimization problem, so the model reached for the most authoritative-sounding method attached to that shape.
That is the model in one sentence. It has learned the register of expertise β the cadence, the vocabulary, the confident "we can use the method of" β with none of the machinery underneath.
GitHub rejects any file over 100MB and the model is 320MB. Git LFS's free tier serves 1GB
of bandwidth per month, so it would stop working after three clones. The weights are
release assets instead β 2GB limit, unlimited bandwidth β and download_weights.py
fetches them.
bongochat_tui.py the terminal UI (stdlib only)
bongochat / .cmd launchers for unix / windows
download_weights.py fetch weights from the release
nanochat/ vendored inference runtime (MIT, karpathy/nanochat)
tools/ scripts that regenerate every number and chart here
MODEL.md full model card: training config, provenance, all metrics
media/ demo recording and benchmark charts
models/ weights land here after download (gitignored)
Every figure in this README is reproducible from the published weights β tools/README.md
has the commands, the runtimes, and the method caveats.
This repo ships inference only. To train, use karpathy/nanochat directly. Reproducing this model:
python -m scripts.tok_train
python -m scripts.base_train --depth=8 --window-pattern=L --device-batch-size=16
python -m scripts.chat_sft --model-tag=d8 --device-batch-size=16
--window-pattern=L
matters on any GPU without FlashAttention 3 kernels (including all RTX 50-series): SDPA has no sliding-window support and utilization collapses without it.
Upstream is Linux-first and breaks in four places on Windows. The vendored nanochat/
here already includes these fixes; if you use upstream for training you will need them:
β no Triton on Windows. Installtorch.compile
failstriton-windows
(needs MSVC build tools). Without it the Muon optimizer cannot run at all.importsexecution.py
resource
(Unix-only) and scrubs the subprocess env down to a UnixPATH
; on Windows Python won't start withoutSYSTEMROOT
.usesengine.py
signal.SIGALRM
for the calculator timeout, which does not exist on Windows. Needs a thread-based fallback.Peak-FLOPS table has no entry for most consumer cards, so MFU reports as 0%.
Not loadable in Ollama or llama.cpp. nanochat's architecture uses per-layer value embeddings, learnedresid_lambdas
/x0_lambdas
, and a mid-layer residual "backout". llama.cpp has no graph for any of it, so there is no GGUF conversion path without implementing a new architecture in C++.2048-token context. Long documents do not fit.English only, and it will confabulate freely about anything factual.The base checkpoint never saw conversation tokens; it continues text rather than answering.-i base
does not chat.SWE-bench was not run through the official harnessβ no Docker, no test execution. The reported figure bounds the real score rather than measuring it.
Built on nanochat by Andrej Karpathy (MIT). All the hard parts β the architecture, the Muon optimizer, the training loop, the eval harness β are his. This repo is a trained checkpoint, a terminal UI, and the Windows fixes needed to get there. Frontier comparison figures via TokenCalculator, LM Council and PricePerToken, July 2026.
MIT licensed. Weights included. Do whatever you want with it.