The symptom – loss curves looked healthy for the first few thousand steps, then plateaued 0.2 nats above the OpenAI baseline while the validation perplexity refused to cross the 35‑point mark. I blamed the dataset mix, batch size, even the tokeniser caching. Each check came up clean.
The culprit – in model.py
the normal_
call for embedding layers had a missing std
argument, falling back to a default of 1.0 instead of the 0.02
used in the original papers. That blew out the variance in every embedding table, forcing the transformer to spend precious capacity correcting the initial noise rather than learning meaningful representations.
nn.init.normal_(self.wte.weight)
nn.init.normal_(self.wte.weight, mean=0.0, std=0.02)
I also found the same regression in the linear projection layers after the attention block. Without a consistent std=0.02
the residual stream accumulated variance at every layer, making the final hidden states twice as scattered as they should have been.
Diagnosis path – I dumped the norms of each weight matrix at step 0 and compared them against a reference run from the original gpt-2
repo. The embedding layer was already 8× larger in Frobenius norm. From there a git bisect
on my config changes pointed straight to an overlooked line in the init function.
Re‑run results – after the fix the model hit the reported perplexity at 40k steps, and the validation loss matched the curve published in the GPT‑2 blog post. The training time didn’t change noticeably, so the fix is purely about fidelity, not efficiency.
For anyone trying to reproduce old transformers from scratch: always verify weight initialisation against the original codebase, not just the paper. The papers often omit the exact std
values, and the reference implementations can get damaged by refactoring. A two‑minute sanity check on initial norms can save you two weeks of head‑scratching.
If you want the full diff of all the other small corrections I found along the way, drop a reply. I’ll put together a clean patch set if there’s interest.
Next GPT-5.6 Revenue Surge: How July Beat Q2 →