Use the built-in GELU, don't roll your own! PyTorch's built-in GELU function is 20% faster than a hand-rolled version when training GPT-2 small models, according to a developer's benchmark. The same code training the same model on the same data ran at about 21,000 tokens per second with the hand-coded GELU and about 25,200 tokens per second with PyTorch's built-in version, a 20% throughput increase. The discovery suggests that the hand-rolled GELU was consuming about 17% of training time, and the developer plans to switch to the built-in version. Unsurprisingly, PyTorch's own built-in GELU function https://docs.pytorch.org/docs/2.13/generated/torch.nn.GELU.html is faster than the hand-rolled one I've been using to date. But I was surprised at how much faster using it made things when training my models. I discovered this accidentally just now while working on something unrelated, but am logging the details here for anyone else that might find it useful. The headline numbers: the same code, training the same model on the same data, ran at about: approximate="tanh" , which uses the same maths as Raschka's version under the hood.That's a 20% increase in throughput for both of the built-in versions -- definitely nothing to be sneezed at. And what is particularly interesting is that there aren't that many GELUs going on -- it's a GPT-2 small-style model, with 12 layers. So that's 12 GELUs handling tensors shaped batch size, seq len, 4 d emb , which is 6, 1024, 3072 for my training setup. Given that the rest of the model is doing all of the normal full attention stuff for GPT-2, it's really surprising that the GELUs alone must have been taking up so much of the time. The throughput numbers mean that we must have been spending about 17% of our time on the extra overhead from the hand-rolled version, so that sets a lower bound for how much time the GELUs were taking up. More info below the fold. Back when I was doing the "interventions" part of my LLM from scratch series /llm-from-scratch , training dozens of GPT-2 small-sized models in the cloud and on my local machines, to keep things simple I used the original model code from Raschka's book. That happens to have its own implementation of the GELU function -- you can see my copy here https://github.com/gpjt/ddp-base-model-from-scratch/blob/16dd249447754c3a7be4f1212ccaf312c7416d92/gpt.py L78 . I'm not that sure why the hand-rolled version is in there -- he covers the maths, but the specific implementation isn't explained in that much depth, and it seems rather like boilerplate, just a "type this in and use it" kind of thing. By contrast, for example, while he does explain the maths behind cross-entropy loss in similar detail, we use the built-in function for it rather than coding it up ourselves. When I switched to using JAX for my own from-scratch implementation /2026/07/llm-from-scratch-34b-building-and-training-gpt-2-small-in-jax , I decided to not bother porting the boilerplate, and just used JAX's own built-in version https://docs.jax.dev/en/latest/ autosummary/jax.nn.gelu.html jax.nn.gelu . I was revisiting the PyTorch code -- I'm in the process of extending it with mixture-of-experts support, about which more in a later post -- and decided to switch from the hand-written GELU to the PyTorch one just to tidy things up a bit. I noticed something interesting -- my new MoE code suddenly seemed to speed up. Was that a mirage? Or had I discovered part -- or even all -- of the reason why the JAX code was so much faster than the PyTorch code? With PyTorch, I was typically getting training speeds of about 21,000 tokens per second, while in JAX I was getting 24,000 tps or so. I'd been chalking that up to JAX's JIT compilation, but could it have been just a result of a random implementation choice I'd made? I did three partial test training runs, letting each one run for 20 minutes to allow the training speed to settle down from any startup overhead. Firstly, with the old hand-coded GELU: bash giles@poppy:~/Dev/ddp-base-model-from-scratch HEAD detached at 16dd249 $ uv run torchrun --nproc per node=1 ddp train.py 1xrtx3090-baseline datasets/ Fetching 4 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 00:00<00:00, 3173.90it/s Starting rank 0 training at global step 0 0%| | 0/33165 00:04