{"slug": "quaternion-small-language-model-comparison", "title": "Quaternion small language-model comparison", "summary": "A developer compared real, complex, and quaternion Transformer projections in a small language model experiment, finding that quaternion projections achieved lower validation loss through 2 million training tokens but lost to real projections from 10 million to 50 million tokens. The results suggest a data-dependent crossover where quaternion models perform better early but are overtaken by real-valued models with more data.", "body_md": "This is a compact Karpathy-style decoder experiment where only the Transformer projections change. The real, complex, and quaternion models otherwise use the same data, batches, optimizer, schedule, attention, and head.\n\nPhase 2 measured a data-dependent crossover. Quaternion projections had lower validation loss through 2M training tokens, 5M was inconclusive, and real projections won from 10M through 50M.\n\n[Why I ran this test](/ibackstrom/QuartAI/blob/main/articles/quaternion-transformers-preface.md)[Quaternion Transformers win early, then lose](/ibackstrom/QuartAI/blob/main/articles/quaternion-transformers-crossover.md)[Short LinkedIn post](/ibackstrom/QuartAI/blob/main/articles/linkedin-post.md)[Short Twitter/X post](/ibackstrom/QuartAI/blob/main/articles/twitter-post.md)\n\n```\nsource .venv/bin/activate\npython -m unittest qllm.tests.test_layers\npython -m qllm.compare                    # quick pipeline test\npython -m qllm.compare --token-budget 50000000  # minimum evidence run (slow)\n```\n\nThe quick command compares a normal model with quaternion models at (1) equal width and (2) equal total\nparameter count, then writes an honest table and plots to `qllm/report/`\n\n. It is deliberately labeled a smoke\ntest because a few thousand tokens and one seed cannot establish the hypothesis.\n\nFull-size YAML configurations are in `qllm/experiments/configs/`\n\n; run one with:\n\n```\npython -m qllm.data.prepare --vocab-size 8192 --train-mb 64 --val-mb 8\npython -m qllm.train qllm/experiments/configs/real-base.yaml\n```\n\nRun the fixed-budget, three-seed equal-parameter follow-up overnight with:\n\n```\n./run_overnight.sh\n```\n\nIt trains only the two arms needed to answer the main hypothesis (300M total tokens), skips completed runs when\nrelaunched with the same output directory, and writes `manifest.json`\n\n, per-run results, `summary.json`\n\n, and\n`summary.md`\n\nunder `qllm/runs/overnight-50m/`\n\n.\n\nPhase 1 commands and caches above remain unchanged. Phase 2 byte-copies the pinned Phase-1 tokenizer, training binary, and training text, then encodes a separate 8 MiB validation prefix with that tokenizer:\n\n```\npython -m qbench.data\npython -m unittest tests.test_models\npython -m qbench.verify --device auto                 # strict 2x200-step Stage A\npython -m qbench.benchmark                         # 50 warmups, 200 iterations\npython -m qbench.run --model real --tokens 500000 --seed 1\npython -m qbench.analysis.analyze\npython -m qbench.generate                             # after final 50M checkpoints exist\npython -m qbench.run_crossover                     # full 635M-token sweep; long\npython -m qbench.run_ablations\n```\n\n`qbench/all.sh`\n\nprepares data, runs Phase-1/layer/model tests and strict real-data verification, benchmarks, runs\nthe crossover and ablations, generates samples, then builds the report. Do not invoke it merely as a quick test.\nRuns use source/data/config digests and atomic resumable 5%-boundary checkpoints. Every final metric uses exactly\n2M fixed sequential validation tokens at batch 64; every curve point uses the same preregistered 131,072-token\nprefix. Generated caches, results, checkpoints, and plots are ignored, while\n`qbench/benchmarks/throughput.csv`\n\nand `qbench/REPORT.md`\n\nare intended deliverables. The Stage-D width-100 real\ncontrol is deliberately omitted because introducing low-rank projections would not be a clean algebra-only control.", "url": "https://wpnews.pro/news/quaternion-small-language-model-comparison", "canonical_source": "https://github.com/ibackstrom/QuartAI", "published_at": "2026-07-15 06:08:49+00:00", "updated_at": "2026-07-15 06:17:42.199510+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "ai-research"], "entities": ["Karpathy"], "alternates": {"html": "https://wpnews.pro/news/quaternion-small-language-model-comparison", "markdown": "https://wpnews.pro/news/quaternion-small-language-model-comparison.md", "text": "https://wpnews.pro/news/quaternion-small-language-model-comparison.txt", "jsonld": "https://wpnews.pro/news/quaternion-small-language-model-comparison.jsonld"}}