My Master’s research turned into a PyTorch layer that calms down unstable Transformers

A developer released AION-Torch, an open-source PyTorch library on PyPI that implements an adaptive residual layer scaling x + α·y by input/output energy instead of a fixed residual weight, based on the author's Master's degree research on stabilizing very deep Transformers. On a personal RTX 4060, a 600-layer Pre-LN Transformer test showed AION produced more stable gradients and a lower final loss than the standard baseline, though the author notes the compute was very limited and describes the project as an alpha research release seeking feedback.

Hi everyone, I turned my Master’s degree research on stabilizing very deep Transformers into an open-source PyTorch library called AION-Torch aion-torch on PyPI . It implements an adaptive residual layer that scales x + α·y based on input/output energy instead of using a fixed residual weight. On my personal RTX 4060 I ran a 600-layer Pre-LN Transformer test where AION seemed to give more stable gradients and a lower final loss than the standard baseline. But my compute is very limited and I’d love to see how it behaves in more realistic settings. Repo: https://github.com/Croxus-Labs/aion-torch/ https://github.com/Croxus-Labs/aion-torch/ PyPI: https://pypi.org/project/aion-torch/ https://pypi.org/project/aion-torch/ This is an alpha research project, so honest feedback and criticism are very welcome. Are you talking about an action potential threshold like in humans? No, I’ve just checked, but it serves the same function. The action potential threshold provides several crucial advantages for the function and efficiency of the nervous system by ensuring reliable, high-fidelity, and regulated signal transmission . That’s a nice analogy, it isn’t a hard threshold, it’s a continuous scaling based on input/output energy, but the goal is similar: keep signals in a stable, high-fidelity regime instead of letting them blow up or die out. i’m going to give this a try. these are what you want us to track? Yes, I will be happy if you can give it a try It’s fascinating to see academic research turned into a practical PyTorch layer for improving Transformer stability. Masters Royale https://masterroyaleapk.com/ runs on Android devices through a lightweight APK that installs outside the Google Play Store. Players test custom decks, champion cards, and evolved troops in fast matches with zero chest timers.