{"slug": "i-trained-a-30m-param-llm-from-scratch-and-the-scaling-floor-was-a-mirage", "title": "I trained a 30M-param LLM from scratch and the scaling \"floor\" was a mirage", "summary": "A developer trained a 30M-parameter decoder-only transformer from scratch on the TinyStories dataset using Kaggle's free T4 GPUs, achieving a validation loss of 1.401 at a learning rate of 1e-3. The project's scaling study across five model sizes (1.8M–56.6M non-embedding parameters) fit a power law L ≈ 1.16 + 0.78·N^−0.276 with R² 0.9975, revealing that the perceived scaling floor was actually compute-limited rather than a data ceiling. Ablations found that RMSNorm vs LayerNorm tied in quality, warmup length had negligible effect at this scale, and peak learning rate was the most impactful hyperparameter.", "body_md": "A small language model built **from scratch** as a learning project: a ~30M\nparameter decoder-only transformer trained on the\n[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset,\nrunning on Kaggle's free T4 GPUs.\n\nThe goal is to understand transformers deeply by implementing the core pieces by hand — model, training loop, data pipeline, and tokenizer — and then running a set of controlled experiments, ending in a technical write-up.\n\n📝\n\nRead the write-up:[What a 30M-Parameter Language Model Taught Me About Training LLMs]— the narrative tour of the findings (training-trick ablations, the scaling \"floor\" that wasn't, tiny-model circuits, and specialization beating scale on-distribution).\n\n**From scratch, no shortcuts.** The transformer is built directly on PyTorch tensor ops — embeddings, multi-head causal self-attention, and the training loop are hand-written, with no`nn.Transformer`\n\nor HuggingFace`model`\n\nclasses doing the heavy lifting.**Multi-head attention done the clean way.** Heads are vectorized into a single batched matmul (reshape Q/K/V to`(batch, num_heads, seq, head_dim)`\n\n) rather than looped — the same approach production implementations use.**Built bottom-up and tested as it grows.** Each component is its own`nn.Module`\n\nsnapping together against a known`(batch, seq_len, hidden_size)`\n\ncontract, covered by a runnable end-to-end shape test plus a gradient-flow test that proves the whole model is trainable (see[Tests](#tests)).**Config-driven and reproducible.** Hyperparameters live in versioned YAML configs; a fixed seed keeps weights and inputs deterministic during development.**Designed for a real compute budget.**~30M params targeting a single free Kaggle T4 — small enough to iterate fast, large enough to produce coherent text.\n\n- Implement a decoder-only transformer end to end, no high-level model libraries.\n- Train it to generate coherent TinyStories-style text on a single T4.\n- Run\n**ablations** to build intuition for which design choices matter:- RMSNorm vs LayerNorm\n- Warmup length\n- (add others as you go)\n\n- Run a small\n**scaling experiment**(model size / data / compute vs loss). - Write it all up as a technical blog post.\n\nImplemented and smoke-tested so far:\n\n- BPE tokenizer training + encode/decode round-trip checks\n- Dataset inspection (story counts, char stats, samples)\n- YAML config loader (attribute + item access)\n- Token + learned-position embeddings\n- Multi-head causal self-attention (vectorized across heads)\n- Position-wise feed-forward network (GeLU,\n`d_ff`\n\n= 4 × hidden_size) - Full transformer block (pre-norm attention + FFN, residual connections)\n- GPT wrapper (stacked blocks + final norm + LM head → logits)\n- Training loop (AdamW, warmup + cosine schedule, AMP, grad clipping, checkpointing, wandb)\n- Kaggle T4 runner (notebook + tokenized Kaggle Dataset)\n- Sampling / generation (\n`scripts/generate.py`\n\n, stops at end-of-text) - Baseline trained: 30M model, 80k steps,\n**val_loss 1.401** at`max_lr: 1e-3`\n\n(1.43 at the original 3e-4) (see[Results](#results)) - Config-selectable norm type + hand-written RMSNorm;\n**RMSNorm-vs-LayerNorm ablation trained**— quality tie, LayerNorm faster (see[Ablations](#ablations)) - Warmup-length ablation (50/500/2000) + high-LR/no-clip stress variant trained — negligible effect at this scale; AdamW makes warmup redundant (see\n[Ablations](#ablations)) - LR sweep (3e-4 / 1e-3 / 3e-3) —\n**peak LR is the real lever**: 1e-3 cuts val_loss 0.042 vs baseline, then plateaus (see[Ablations](#ablations)) - Scaling study — 5-point width+depth ladder (1.8M–56.6M non-embed) fit to\n`L ≈ 1.16 + 0.78·N^−0.276`\n\n(R² 0.9975); 80k diagnostic shows the floor is**compute-limited, not a data ceiling**(see[Scaling study](#scaling-study)) - Attention interpretability deep-dive — previous-token & attention-sink heads at both scales; induction only weakly emerging (see\n[Interpretability](#interpretability--what-the-attention-heads-learned)) - Evaluation: custom TinyStories rubric (LLM-as-judge, Llama-3.3-70B) + GPT-2 comparison — specialization beats scale on the target distribution (see\n[Evaluation](#evaluation--llm-as-judge--gpt-2-comparison)) - Technical blog post write-up\n\nThe model has no external test runner yet — tests live in each module's\n`__main__`\n\nblock and run on execution. For the model:\n\n```\npython src/model.py\n```\n\nThis runs three checks against the `configs/tiny.yaml`\n\nmodel:\n\n| Check | What it verifies |\n|---|---|\nEnd-to-end shape |\nA `(batch, seq_len)` batch of token IDs produces logits of shape `(batch, seq_len, vocab_size)` . One forward pass exercises embeddings, every block's attention + FFN + norms, the final norm, and the LM head. |\nParameter count |\nReports total params (≈ budget). Doubles as a wiring check: a plain `list` instead of `nn.ModuleList` , or a submodule not assigned to `self.` , would silently drop blocks from the count. |\nGradient flow |\nBuilds a tiny throwaway GPT, backprops `logits.sum()` , and asserts every parameter receives a non-zero gradient — i.e. the whole model is connected to the autograd graph and trainable. Catches disconnected/untrained submodules that a shape check can't. |\n\nLatest run (`tiny.yaml`\n\n: 4 layers, hidden 128, 4 heads, vocab 16k):\n\n```\ngpt out shape: (2, 128, 16000)\nOK: GPT returns (batch, seq_len, vocab_size)\nparams: 4.9M\nOK: gradients flow to all parameters\n```\n\nThe 4.9M here is the fast\n\ndebugconfig; the ~30M target model uses a larger`hidden_size`\n\n. At this size the token embedding + LM head (vocab × hidden) account for ~84% of params.\n\n```\nmy-LLM/\n├── configs/              # YAML experiment configs (see configs/tiny.yaml)\n├── notebooks/\n│   ├── exploration.ipynb  # scratch\n│   └── kaggle_train.ipynb # Kaggle T4 run: clone → install → link data → train\n├── scripts/\n│   ├── dataset_stats.py  # inspect raw TinyStories (counts, char stats, samples)\n│   ├── prepare_data.py   # download + tokenize TinyStories\n│   ├── generate.py       # sample text from a trained checkpoint\n│   └── kaggle_dataset/   # dataset-metadata.json for the tokenized Kaggle Dataset\n├── src/\n│   ├── model.py          # transformer: embeddings, attention, FFN, blocks, GPT wrapper\n│   ├── train.py          # training loop\n│   ├── data.py           # data loading / batching\n│   ├── tokenizer.py      # tokenizer training\n│   └── paths.py          # env-overridable paths (DATA_DIR, TOKENIZER_PATH, CKPT_DIR, CONFIG_PATH)\n├── requirements.txt\n└── README.md\n```\n\nData, checkpoints, tokenizers, and logs are git-ignored (regenerable / large).\n\nLocal (CPU or GPU):\n\n```\npython -m venv .venv && source .venv/bin/activate\npip install -r requirements.txt\n# Install a torch build matching your CUDA — see https://pytorch.org/get-started/\n```\n\nOn **Kaggle**, `torch`\n\nand `numpy`\n\nare preinstalled with CUDA for the T4. Don't\nreinstall torch there; just add the lighter deps:\n\n```\npip install tokenizers datasets pyyaml tqdm matplotlib\n# 0. (optional) Inspect the raw dataset: story count, char stats, samples\npython scripts/dataset_stats.py\n\n# 1. Prepare data (download + train tokenizer + tokenize corpus)\npython scripts/prepare_data.py --config configs/tiny.yaml\n\n# 1b. (optional) Sanity-check the trained tokenizer: vocab size + encode/decode round-trip\npython -m src.tokenizer\n\n# 2. Train (defaults to configs/tiny.yaml; pick another with CONFIG_PATH=...)\npython src/train.py\n\n# 3. Generate\npython scripts/generate.py --checkpoint checkpoints/... --prompt \"Once upon a time\"\n```\n\nTraining targets a free Kaggle T4. The tokenized corpus and tokenizer ship as a\nprivate **Kaggle Dataset** (the `.bin`\n\nfiles are too large for git), and\n`notebooks/kaggle_train.ipynb`\n\nwires it all together: clone the repo, install\ndeps, point the code at the dataset, train.\n\n**One-time: publish the tokenized data as a private Dataset** (from the repo root,\nneeds the [Kaggle CLI](https://github.com/Kaggle/kaggle-api) authenticated via\n`kaggle auth login`\n\n):\n\n``` php\n# stage the payload next to its metadata (hardlinks -> no ~900 MB copy)\nln -f data/train.bin data/val.bin tokenizer/tokenizer.json scripts/kaggle_dataset/\nkaggle datasets create  -p scripts/kaggle_dataset                  # first upload\nkaggle datasets version -p scripts/kaggle_dataset -m \"refresh\"     # later updates\n```\n\n**Then on Kaggle:** open `notebooks/kaggle_train.ipynb`\n\n, set Accelerator → GPU T4,\nadd the dataset as input, optionally add a `WANDB_API_KEY`\n\nsecret, and Run All.\n\n`src/paths.py`\n\nresolves every path the run touches from environment variables,\nfalling back to the local layout — so the same code runs unchanged on Kaggle\n(read-only data under `/kaggle/input`\n\n) with no edits or symlinks:\n\n| Env var | Default | What |\n|---|---|---|\n`DATA_DIR` |\n`data` |\ndir holding `train.bin` / `val.bin` |\n`TOKENIZER_PATH` |\n`tokenizer/tokenizer.json` |\ntokenizer JSON |\n`CKPT_DIR` |\n`checkpoints` |\ncheckpoint output dir |\n`CONFIG_PATH` |\n`configs/tiny.yaml` |\nYAML config to load |\n\nSo a one-off ablation needs no code change:\n\n```\nCONFIG_PATH=configs/no_warmup.yaml python src/train.py\n```\n\n| Experiment | Variable | Configs | Status |\n|---|---|---|---|\n| Norm ablation | RMSNorm vs LayerNorm | `ablation_norm_{layernorm,rmsnorm}.yaml` |\nDone — loss tie; LayerNorm ~33% faster (unfused RMSNorm) |\n| Warmup length | warmup_steps (50/500/2000) | `ablation_warmup_{50,500,2000}.yaml` |\nDone — null; warmup redundant with AdamW |\n| Warmup under stress | warmup_steps @ 10× LR, no clip | `ablation_warmup_stress_{50,2000}.yaml` |\nDone — still null; neither diverges |\n| LR sweep | max_lr (3e-4/1e-3/3e-3) | `ablation_lr_{3e-4,1e-3,3e-3}.yaml` |\nDone — 1e-3 best (−0.042); LR is the real lever |\n| Scaling | model size (5-point ladder) + 80k diagnostic | `scale_{xs,s,m,base,l}.yaml` , `scale_l_80k.yaml` |\nDone — power-law fit; floor is compute-limited |\n\nThe ~30M model (`configs/small.yaml`\n\n: hidden 512, 8 layers, 8 heads, vocab 8k,\nseq 256 → **33.6M params**) trained on a single Kaggle T4 for the full 80k-step\nwarmup + cosine schedule (~4.5 h at ~42k tokens/sec):\n\n| Metric | Value |\n|---|---|\nval_loss |\n1.401 (train 1.433) at `max_lr: 1e-3` ; 1.43 at the original 3e-4 |\n| Throughput | ~42k tokens/sec (fp16 AMP, batch 32) |\n| Params | 33.6M |\n| Schedule | linear warmup (2k) → cosine decay to 10% of peak |\n\nSample (prompt `\"Once upon a time\"`\n\n, temperature 0.8):\n\nOnce upon a time, there was a little girl named Lily. She loved to play in the garden with her toys. One day, she found a shiny rock and she wanted to keep it safe. She put it in her pocket and went to play with her toys. … She remembered the shiny rock in her pocket and thought it might make a magic wand.\n\nThe model produces fluent, coherent TinyStories-style narratives with consistent characters and a clear story arc. Residual small-model artifacts (occasional repetition or mild logic drift) are expected at this scale and are exactly what the planned scaling study is meant to probe.\n\n**Status:** complete — both 22k-step arms trained on a Kaggle T4.\n\nRMSNorm is hand-written (`src/model.py`\n\n, ~6 lines: divide each token vector by\nits root-mean-square, then apply one learned scale — no mean-subtraction and no\nshift, unlike LayerNorm). The norm type is selectable per run via `norm_type`\n\nin\nthe config, so the two arms share identical code and differ in exactly one field:\n\n`configs/ablation_norm_layernorm.yaml`\n\n—`norm_type: layernorm`\n\n`configs/ablation_norm_rmsnorm.yaml`\n\n—`norm_type: rmsnorm`\n\nBoth run at a reduced **22k-step** budget (vs the 80k baseline) to keep ablations\ncheap, and share `wandb_group: norm_ablation`\n\nso their curves auto-overlay in\nWeights & Biases.\n\n**Implementation sanity checks** (done before spending any GPU time — a random\n`(2, 4, 16)`\n\ninput through each norm):\n\n| Property | LayerNorm | RMSNorm |\n|---|---|---|\n| Per-token mean of output | ≈ 0 (it re-centers) | non-zero (never centers) |\n| Per-token RMS of output | ≈ 1 | ≈ 1 |\n\nBoth control magnitude identically (RMS ≈ 1); only LayerNorm also forces the mean to 0. That single difference — LayerNorm's extra centering + shift parameter — is the whole substance of the ablation.\n\n**Parameter cost.** On the `tiny.yaml`\n\ndebug build (hidden 128, 4 layers → 9\nnorms), LayerNorm gives 2.866M params vs RMSNorm's 2.865M. The ~1.15k gap is\nexactly RMSNorm dropping LayerNorm's per-feature shift vector (9 norms × 128).\nRMSNorm is strictly lighter.\n\n**Pipeline smoke test.** A 200-step CPU run of the LayerNorm arm confirmed the\nfull path end to end: config → model builds with `norm_type`\n\n→ training loop\n(`train_loss`\n\n9.13 → 3.94) → eval (`val_loss`\n\n8.98 → 4.13) → sample + checkpoint\n→ correct W&B grouping. So the real 22k runs should launch clean on the T4.\n\n**Results (22k steps, Kaggle T4).** Both arms started from the identical seed, so\ntheir `lr`\n\nand early curves overlay exactly — the comparison is clean.\n\n| Metric | LayerNorm | RMSNorm |\n|---|---|---|\nval_loss (final) |\n1.601 |\n1.603 |\n| train_loss (final) | 1.796 | 1.800 |\n| Throughput | ~43k tok/s | ~32k tok/s |\n| grad_norm (final) | 0.71 | 0.70 |\n\n**Takeaway — a quality tie, a speed win for LayerNorm.** The two arms are\nstatistically indistinguishable on loss (Δval_loss ≈ 0.002, within run-to-run\nnoise): at 30M scale, dropping LayerNorm's mean-centering costs nothing in\nquality — RMSNorm gives up centering essentially for free. The one real\ndifference is throughput: LayerNorm ran ~33% faster, because PyTorch's\n`nn.LayerNorm`\n\nis a single fused CUDA kernel while the hand-written RMSNorm here\nis several unfused tensor ops. That gap is an *implementation* artifact, not a\nproperty of RMSNorm — a fused RMSNorm (as used in Llama) would match or beat\nLayerNorm. Net: with this unfused RMSNorm, LayerNorm is the better default at\nthis scale, so subsequent ablations fork from the LayerNorm arm.\n\n**Status:** complete — three 22k-step arms trained on a Kaggle T4.\n\nThe LR schedule is a linear warmup ramp (0 → peak over `warmup_steps`\n\n) followed\nby cosine decay to 10% of peak (`src/train.py`\n\n). Warmup is meant to prevent an\nearly gradient blow-up while the model is still random and the LR is high. This\nablation sweeps how long that ramp lasts, holding everything else fixed at the\nLayerNorm winner from Ablation 1:\n\n`configs/ablation_warmup_50.yaml`\n\n—`warmup_steps: 50`\n\n`configs/ablation_warmup_500.yaml`\n\n—`warmup_steps: 500`\n\n`configs/ablation_warmup_2000.yaml`\n\n—`warmup_steps: 2000`\n\nAll three share `wandb_group: warmup_ablation`\n\nand the same seed, so their `lr`\n\nramps are the only thing that visibly differs early on.\n\n**Results (22k steps, Kaggle T4).**\n\n| Metric | warmup 50 | warmup 500 | warmup 2000 |\n|---|---|---|---|\nval_loss (final) |\n1.590 |\n1.599 | 1.601 |\n| train_loss (final) | 1.666 | 1.675 | 1.682 |\n| Throughput | ~40k tok/s | ~38k tok/s | ~40k tok/s |\n| grad_norm (final) | 0.72 | 0.69 | 0.68 |\n\n**Takeaway — warmup length doesn't matter here.** All three arms land within\nΔval_loss ≈ 0.011, i.e. run-to-run noise; if anything the *shortest* warmup edges\nahead, the opposite of what warmup is \"supposed\" to buy. The reason is visible in\nthe `grad_norm`\n\npanel: even the 50-step arm shows **no early spike** — gradient\nclipping (max-norm 1.0) already caps the one bad-batch shock that warmup exists to\nsmooth, and at 30M params with a modest 3e-4 peak LR the model never approaches an\ninstability boundary. Warmup only touches the first ≤2k of 22k steps anyway; the\nremaining ~90% is the identical cosine decay. So at this scale warmup is a no-op —\na legitimate (if anticlimactic) result: the baseline is already in a robust regime.\n\nThat raises the obvious question — *when does warmup earn its keep?* Follow-up\n[stress ablation](#3-warmup-under-stress) removes the safety nets (10× LR,\nclipping off) to find out.\n\n**Status:** complete — two 22k-step arms trained on a Kaggle T4.\n\nAblation 2 showed warmup does nothing in the baseline's comfortable regime. This\nfollow-up deliberately pushes the model toward instability to see whether warmup\nthen starts to matter. Both arms fix the same **stress condition** — peak LR\n`1e-3`\n\n(10× the baseline) and gradient clipping **disabled** (`gradient_clip_norm: .inf`\n\n, wired through from config after fixing a hardcoded `max_norm=1.0`\n\n) — and\nvary only the warmup length:\n\n`configs/ablation_warmup_stress_50.yaml`\n\n—`warmup_steps: 50`\n\n`configs/ablation_warmup_stress_2000.yaml`\n\n—`warmup_steps: 2000`\n\nGrouped under `wandb_group: warmup_stress`\n\n. The hypothesis: with clipping off, the\n50-step arm should blow up early while the 2000-step arm ramps in gently.\n\n**Results (22k steps, Kaggle T4).**\n\n| Metric | warmup 50 | warmup 2000 |\n|---|---|---|\nval_loss (final) |\n1.553 | 1.549 |\n| train_loss (final) | 1.609 | 1.620 |\n| Throughput | ~39k tok/s | ~41k tok/s |\n| grad_norm (final) | 0.39 | 0.39 |\n\n**Takeaway — the hypothesis was wrong; warmup still doesn't matter.** Even with\nclipping removed and the LR cranked 10×, the two arms tie again (Δval_loss ≈ 0.004)\nand *neither diverges* — both train smoothly to convergence. The reason is the\noptimizer: **AdamW already normalizes every parameter's update by a running\nestimate of its own gradient magnitude, so the effective step size self-scales and\nis largely decoupled from raw gradient spikes.** That adaptive normalization does\nthe very job warmup and clipping were meant to do, so removing clipping and raising\nthe LR wasn't enough to break training. Warmup is essentially redundant with\nAdam-family optimizers at this scale — to actually make it earn its keep you'd\nlikely need plain SGD, or a far more extreme LR.\n\nTwo footnotes worth the reader's attention:\n\n**The 10× LR** Both stress arms (~1.55) beat every baseline-LR arm (~1.59). The real lever at this scale isn't warmup, it's*helped*.**peak learning rate**— the conservative 3e-4 baseline was leaving quality on the table. A dedicated[LR sweep](#4-learning-rate-sweep)is the natural next experiment — done below.- LR and clipping were changed together, so this cleanly shows\n*warmup*is robust under aggressive optimization; isolating clipping's individual effect would need extra arms.\n\n**Status:** complete — three 22k-step arms trained on a Kaggle T4.\n\nThe stress ablation hinted that **peak learning rate**, not warmup, is the lever\nthat actually moves quality. This sweep confirms it directly: it varies `max_lr`\n\n(with `min_lr`\n\ntracking at 10% and `learning_rate`\n\nmirroring) across a 10× range,\nholding gradient clipping **ON at 1.0** so — unlike the stress runs — LR is the\nonly thing that changes:\n\n`configs/ablation_lr_3e-4.yaml`\n\n—`max_lr: 3e-4`\n\n(the original baseline)`configs/ablation_lr_1e-3.yaml`\n\n—`max_lr: 1e-3`\n\n`configs/ablation_lr_3e-3.yaml`\n\n—`max_lr: 3e-3`\n\nGrouped under `wandb_group: lr_sweep`\n\n.\n\n**Results (22k steps, Kaggle T4).**\n\n| Peak LR | val_loss (final) | train_loss (final) | grad_norm (final) |\n|---|---|---|---|\n| 3e-4 (baseline) | 1.601 | 1.682 | 0.68 |\n1e-3 |\n1.559 |\n1.626 | 0.37 |\n| 3e-3 | 1.557 | 1.617 | 0.27 |\n\n**Takeaway — the first ablation that actually moves the loss, and LR is the lever.**\nAfter three ties (norm, warmup, warmup-under-stress), this one shows a real effect:\ntripling-ish the LR from 3e-4 to 1e-3 cuts val_loss by **0.042** — ~20× the\nrun-to-run noise that swallowed every earlier comparison. But the gain **saturates**:\n1e-3 → 3e-3 moves val_loss only 0.002 (noise), so it's a plateau, not a U-shape.\nThe 3e-3 arm didn't diverge (clipping + AdamW's adaptive steps kept it stable — its\nfinal grad_norm is actually the lowest), it just bought nothing extra on validation.\n\n** 1e-3 is the new sweet spot:** it captures essentially all of the improvement\nwhile sitting further from the stability edge than 3e-3, which matters because\nhigher LRs get riskier as the model scales. The original\n\n`3e-4`\n\nbaseline was simply\ntoo conservative — a reminder that LR is worth tuning *before*chasing architectural knobs.\n\n**Re-confirmed on the full 80k run.** Promoting `max_lr: 1e-3`\n\nto `small.yaml`\n\nand\nre-running the complete 80k-step schedule carried the sweep's gain all the way\nthrough: **val_loss 1.401** (train 1.433), down from the original 3e-4 baseline's\n1.43. LR peaked cleanly at 0.001 and annealed to ~3e-5 under the cosine schedule,\ngrad_norm settled around 0.46 with no instability, and throughput held at ~39k\ntokens/sec. The 22k-budget signal was real and scaled to the full run.\n\nThis `1e-3`\n\nrun is now the baseline the scaling study builds on.\n\n**Status:** complete — a 5-point ladder plus one diagnostic run, all on a Kaggle T4.\n\nWith the ablations settling the *training recipe* (LayerNorm, warmup irrelevant,\n`max_lr: 1e-3`\n\n), the question turns to **model size**: how does loss fall as the\nnetwork grows, and where does it stop paying off? The study trains a ladder of\nfive models that grow **width and depth together** (aspect ratio roughly fixed),\nholding everything else at the tuned recipe — a shared **fixed-token budget**\n(40k steps ≈ 328M tokens), `max_lr: 1e-3`\n\n, LayerNorm, warmup 2000, seed 1337,\nbatch 32. Loss is fit against **non-embedding parameters** `N ≈ 12·layers·d²`\n\n(the attention + MLP compute, excluding the vocab-dominated embedding tables —\nthe Kaplan et al. convention), so small-model totals aren't swamped by the fixed\n8k-token embedding.\n\n| Config | hidden / layers / heads | Non-embed N | Total params | val_loss (40k) |\n|---|---|---|---|---|\n`scale_xs.yaml` |\n192 / 4 / 3 | 1.77M | 4.9M | 1.826 |\n`scale_s.yaml` |\n256 / 6 / 4 | 4.72M | 8.9M | 1.676 |\n`scale_m.yaml` |\n384 / 6 / 6 | 10.62M | 16.9M | 1.555 |\n`scale_base.yaml` |\n512 / 8 / 8 | 25.17M | 33.6M | 1.487 |\n`scale_l.yaml` |\n768 / 8 / 12 | 56.62M | 69.2M | 1.415 |\n\nAll five share `wandb_group: scaling_study`\n\n, and no core code changed — `train.py`\n\nbuilds the model entirely from config fields, and the depth-aware residual init\n(`1/√(2·num_layers)`\n\n) adapts to each depth automatically.\n\n**A power law with a floor.** On log-log axes the five points are monotone but\nvisibly *bent* — the slope shallows as `N`\n\ngrows, which is the signature of an\nirreducible floor. Fitting `L = E + A·N^−α`\n\nby an **E-sweep** (grid-search the\nfloor `E`\n\nand keep the value that makes `(L − E)`\n\nstraightest on log-log — more\nrobust than `curve_fit`\n\nwith only five points) gives:\n\nL ≈ 1.16 + 0.78·N^−0.276(R² = 0.9975)\n\nThe fit is excellent, and it's tempting to read `E = 1.16`\n\nas \"TinyStories'\nirreducible loss.\" Each 10× in params roughly *halves* the gap to that floor\n(`10^−0.276 ≈ 0.53`\n\n), and extrapolating, reaching even `L = 1.30`\n\nwould demand\n~500M non-embedding params — steeply diminishing returns.\n\n**But that floor is an artifact of the budget, not the data.** Every ladder point\nshares the same 40k-step budget, so the fitted `E`\n\nis really a *fixed-compute*\nfloor. To test it, one diagnostic run (`scale_l_80k.yaml`\n\n) retrains the largest\nmodel for **double the budget** (80k steps, ≈656M tokens) — identical in every\nother respect:\n\n| Model | Non-embed N | Budget | val_loss |\n|---|---|---|---|\n`scale_l` (on-curve) |\n56.62M | 40k | 1.415 |\n| 30M baseline (reference) | 25.17M | 80k | 1.401 |\n`scale_l_80k` |\n56.62M | 80k |\n1.325 |\n\nDoubling the budget drops the largest model by **−0.090** (1.415 → 1.325),\nstraight *through* the fitted 1.16-floor's neighbourhood and well past the\nfully-trained 30M baseline (by 0.076). Validation sat slightly *below* train\nloss (1.325 vs 1.352 — the dropout signature, not overfitting), so the data still\nhad signal to give. The red arrow in the figure marks this drop off the curve.\n\n**Takeaway — the ladder's floor was compute-limited, not fundamental.** The\nclean bending power law looked like it was approaching an irreducible loss, but\nthat apparent floor was conditioned on the training budget: give the biggest\nmodel 2× the compute and it punches right through. The real lesson is a\ncautionary one — **don't read an irreducible floor off an undertrained sweep.**\nTinyStories' true data ceiling is lower than 1.325 and was *not* reached here;\nfinding it would take longer runs than a free T4 budget justifies, and it's left\nas an honest open thread.\n\n**Caveats.** The ladder points are near-converged, not compute-optimal\n(Chinchilla-style), so the fit is illustrative rather than a rigorous scaling\nlaw; and holding LR fixed across a 32× param range is a mild confound at the\nextremes. Both are acceptable for a study whose point is the *shape* of the\ncurve and the budget-vs-floor distinction, not a precise α.\n\n**Status:** complete — attention patterns visualized from the trained `scale_xs`\n\n(1.8M) and `scale_l`\n\n(56.6M) checkpoints.\n\nLoss curves say *how well* a model does; they say nothing about *what it\nlearned*. This section opens the hood. A one-line hook in `SelfAttention.forward`\n\nstashes the post-softmax attention weights (`self.last_attn`\n\n, shape\n`(batch, heads, T, T)`\n\n), so after a single forward pass on a prompt — on CPU, no\nGPU needed — every head's attention map can be read back and plotted\n(`scripts/viz_attention.py`\n\nfor one head, `scripts/viz_attention_grid.py`\n\nfor the\nwhole model). Row `i`\n\nof a map is \"where token `i`\n\nlooks\"; the dark upper triangle\nin every plot is the causal mask (no token attends to the future).\n\n**Finding 1 — clean positional heads.** Layer 0, head 0 of even the *smallest*\nmodel is a crisp **previous-token head**: every token attends to the one\nimmediately before it (the bright sub-diagonal).\n\n**Finding 2 — a recognizable head taxonomy, at both scales.** Plotting all heads\nat once (below, `scale_l`\n\n: 8 layers × 12 heads) reveals a division of labour:\n**previous-token** heads (sub-diagonal), **attend-to-self / identity** heads\n(main diagonal, e.g. L0 H11), **attention-sink** heads (a bright first column —\nnearly every token dumps attention onto the initial token), and diffuse\ncontext-mixing heads in the deeper layers. The larger model has a visibly richer\ntaxonomy (a dedicated identity head, several content-selective heads), but the\n**core archetypes — previous-token and attention-sink — appear in the 1.8M model\ntoo.**\n\nThis is the interpretability payoff and it ties straight back to the project's\nthesis: **attention sinks and previous-token heads are the same circuits\ndocumented in GPT-scale models** — a 1.8M-parameter network trained only on\nchildren's stories independently rediscovers them. Same tricks, tiny scale.\n\n**Finding 3 (honest footnote) — induction is only emerging.** Induction heads\n(the copy/pattern-completion circuit: see a repeated token, attend to whatever\nfollowed its previous occurrence) are the natural next circuit up from\nprevious-token heads. Probing with a repeated prompt (\n\n`\"Once upon a time. Once upon a time\"`\n\n) and scoring every head on the induction cells\n(`scripts/induction_score.py`\n\n):| Model | Layers | Top induction head | Score (chance ≈ 0.12) |\n|---|---|---|---|\n`scale_xs` |\n4 | L3 H1 | 0.254 |\n`scale_l` |\n8 | L4 H0 | 0.314 |\n\nBoth models sit at ~2× chance — a mild, mid-layer induction *bias* — but neither\napproaches the 0.6+ of a *dedicated* induction head. `scale_l`\n\nscores a little\nhigher, hinting at a scale trend, but a 0.06 gap on a single-prompt probe is\nwithin noise, so no firm claim is made. The likely reason there's no crisp\ncircuit: TinyStories rarely rewards long-range verbatim copying, so there's\nlittle pressure to build one. A robust version would average the score over many\nrandom repeated-token sequences.\n\n**Caveat.** These maps are from a *single prompt / single forward pass*. The\narchetypes (previous-token, sink) are robust and visible immediately; any\nquantitative claim (the induction scores especially) would want averaging over\nmany prompts to be publication-grade.\n\n**Status:** complete — 30 stories (3 models × 10 prompts) scored by an LLM judge.\n\nLoss numbers don't tell you whether the stories are any *good*. Following the\nTinyStories paper's approach, each model continues 10 shared story prompts\n(`scripts/eval_generate.py`\n\n), and every continuation is scored 1–5 on six\ndimensions by an LLM judge — **Llama-3.3-70B via the free Groq API**\n(`scripts/eval_judge.py`\n\n, JSON mode, temperature 0). The judge is instructed to\ngrade **fitness for a young child's story** (not raw sophistication) and to give\nhonest strengths *and* weaknesses for every model. Two of our models\n(`scale_xs`\n\n, `scale_l`\n\n) are compared against **GPT-2 (124M)** — a general model\n~2× the size of our largest.\n\n| Dimension | scale_xs (1.8M) | scale_l (56.6M) | gpt2 (124M) |\n|---|---|---|---|\n| grammar | 3.60 | 4.30 |\n2.70 |\n| fluency | 3.30 | 4.10 |\n2.00 |\n| coherence | 2.60 | 4.00 |\n1.10 |\n| contextual correctness | 3.20 | 4.20 |\n1.20 |\n| creativity | 2.70 | 3.20 |\n1.60 |\n| plot / completion | 3.50 | 4.30 |\n1.00 |\noverall |\n3.15 | 4.02 |\n1.60 |\n\n**Takeaway — specialization beats scale on the target distribution.** The\n56.6M model tuned on TinyStories decisively wins the children's-story task, and\nscores rise monotonically with our model size (3.15 → 4.02). **GPT-2, despite\nbeing larger, collapses on the story dimensions** — coherence 1.1, contextual\ncorrectness 1.2, plot 1.0 — because it drifts off-genre: police stations,\ninconsistent character names, dark themes unsuitable for a 3-year-old. The\njudge's own words:\n\n*\"incoherent, inconsistent character names, unsuitable for a young child's story.\"*A model specialized for a narrow distribution beats a general model twice its size\n\n*at that distribution*.\n\n**Honest caveat.** GPT-2's grammar (2.70) and fluency (2.00) scores almost\ncertainly *understate* its real linguistic quality — GPT-2's raw English is\nfluent, but the task-framed judge let the genre-drift bleed into the language\nscores (a halo effect). The defensible claim is narrower and truer: **GPT-2's\nweakness here is task-fit — coherence, genre, appropriateness — not language\nitself.** Its one genuine strength shows through too: the judge repeatedly noted\nGPT-2 *\"attempts to introduce a new character and conflict\"* — it is the more\n*ambitious* writer, it just can't stay on the rails of a simple story.\n\n**Per-model strengths / weaknesses:**\n\n**scale_l**—*+*coherent, complete, age-appropriate, on-prompt story arcs;*−*predictable, thin conflict, occasional logic slips.**scale_xs**—*+*stays in genre, decent grammar;*−*weaker coherence, characters appearing from nowhere.**GPT-2**—*+*sophisticated vocabulary, ambitious plots;*−*off-genre drift, incoherent as a children's story, sometimes inappropriate.\n\n**Limitations.** One generation per prompt, 10 prompts, a single judge model —\nenough for a clear directional result, not a rigorous benchmark. More prompts,\nmultiple judges, and multiple samples per prompt would tighten it.\n\nDataset: TinyStories (Eldan & Li, 2023).", "url": "https://wpnews.pro/news/i-trained-a-30m-param-llm-from-scratch-and-the-scaling-floor-was-a-mirage", "canonical_source": "https://github.com/rishipadhye/my-LLM", "published_at": "2026-07-21 19:52:00+00:00", "updated_at": "2026-07-21 20:12:39.288278+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research", "ai-tools"], "entities": ["TinyStories", "Kaggle", "PyTorch", "AdamW", "RMSNorm", "LayerNorm", "GeLU"], "alternates": {"html": "https://wpnews.pro/news/i-trained-a-30m-param-llm-from-scratch-and-the-scaling-floor-was-a-mirage", "markdown": "https://wpnews.pro/news/i-trained-a-30m-param-llm-from-scratch-and-the-scaling-floor-was-a-mirage.md", "text": "https://wpnews.pro/news/i-trained-a-30m-param-llm-from-scratch-and-the-scaling-floor-was-a-mirage.txt", "jsonld": "https://wpnews.pro/news/i-trained-a-30m-param-llm-from-scratch-and-the-scaling-floor-was-a-mirage.jsonld"}}