{"slug": "show-hn-turbogpt-train-22kib-transformer-in-13s", "title": "Show HN: TurboGPT: train 22KiB transformer in 13s", "summary": "A developer released TurboGPT, an MIT-licensed byte-level GPT trainer written in CUDA C++ that trains a 22KiB transformer in 13 seconds. The tool builds on Windows with Visual Studio 2022 C++ tools and CUDA 13.4 via `build.ps1 -CudaArch 86`, stores checkpoints with model, optimizer, scheduler and trainer state for resuming with `--load`, and emits TensorBoard-compatible logs plus a `report.json`; its author reports 2.52435 BPB on hn1g after 1.5G training tokens.", "body_md": "Tiny byte-level GPT training in CUDA C++. MIT.\n\nWindows, Visual Studio 2022 C++ tools, and CUDA 13.4:\n\n```\n.\\build.ps1 -CudaArch 86\n```\n\n`CudaArch` is the GPU compute capability from [NVIDIA's CUDA GPU list](https://developer.nvidia.com/cuda-gpus).\n\n```\n.\\build\\turbogpt.exe --data hn1g.txt --log-to runs/ctx4\n```\n\nThe run stores its checkpoint at\n`runs/ctx4/ctx4.pt`, containing model, optimizer, scheduler, and trainer state.\nUse `--load CHECKPOINT.pt` to resume it.\n\n`runs/ctx4/report.json` is derived from the log directory. Logs are TensorBoard-compatible:\none report per batch, capped at 8Mi reports, and flushed with periodic or final checkpoints.\n\n- hn1g after 1.5G training tokens: **2.52435 BPB** .\n\n```\npython tests\\verify.py\n```\n\n", "url": "https://wpnews.pro/news/show-hn-turbogpt-train-22kib-transformer-in-13s", "canonical_source": "https://github.com/lostmsu/TurboGPT", "published_at": "2026-09-29 19:20:02+00:00", "updated_at": "2026-09-29 20:48:39.085163+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "developer-tools"], "entities": ["TurboGPT", "CUDA", "NVIDIA", "Visual Studio 2022", "hn1g", "TensorBoard"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-turbogpt-train-22kib-transformer-in-13s", "markdown": "https://wpnews.pro/news/show-hn-turbogpt-train-22kib-transformer-in-13s.md", "text": "https://wpnews.pro/news/show-hn-turbogpt-train-22kib-transformer-in-13s.txt", "jsonld": "https://wpnews.pro/news/show-hn-turbogpt-train-22kib-transformer-in-13s.jsonld"}}