{"slug": "i-scaled-a-pure-spiking-neural-network-snn-to-1-088b-parameters-from-scratch-ran", "title": "I scaled a pure Spiking Neural Network (SNN) to 1.088B parameters from scratch. Ran out of budget, but here is what I found", "summary": "An 18-year-old independent developer scaled a pure Spiking Neural Network (SNN) to 1.088 billion parameters from scratch, achieving convergence with a loss of 4.4 after 27,000 steps despite budget constraints. The model exhibited 93% sparsity, cross-lingual emergence of Russian text, and a spontaneous memory routing shift at larger scales. The developer has open-sourced the code and checkpoint on GitHub.", "body_md": "Hey everyone. I’m an 18yo indie dev, and I’ve been experimenting with Spiking Neural Networks (SNNs) for language modeling. A lot of papers (like SpikeBERT) mention that training 1B+ SNNs directly from random initialization fails due to vanishing gradients, so people usually do ANN-to-SNN conversion or distillation. I wanted to see if I could force it to converge purely in the spike domain. I had to stop at 27k steps because my wallet is literally empty lol, but the loss converged to 4.4.\n\nHere are the most interesting things that happened:\n\nMassive Sparsity: It maintains ~93% sparsity. Only about 7% of neurons fire per token. It's incredibly cheap on memory during inference compared to dense models.\n\nCross-lingual emergence: Around step 25K, it randomly started generating structurally correct Russian text, even though it wasn't explicitly targeted/weighted for it in the dataset mix.\n\nMemory routing shift: As I scaled the architecture past 600M to 1B, the model spontaneously shifted 39% of its activation routing into the persistent memory module. It basically learned on its own that memory is more valuable at a larger scale.\n\nLimitations (Being honest):\n\nThe text generation is still janky and nowhere near GPT-2 fluency yet. The loss (4.4) is high, mostly because I couldn't train it longer. But proving that a 1B pure SNN can converge from random init feels like a solid milestone.\n\nI'm sharing this because I'd love some harsh technical feedback.\n\nDoes anyone here have experience with neuromorphic hardware? Would an architecture like this map well to Loihi?\n\nIf anyone has tips on pushing SNN loss lower or stabilizing surrogate gradients further, I'm all ears.\n\nThe code, architecture details, and the 12GB full training checkpoint (weights + optimizer states) are on my GitHub:[https://github.com/gtausa197-svg/-Project-Nord-Spiking-Neural-Network-Language-Model.git](https://github.com/gtausa197-svg/-Project-Nord-Spiking-Neural-Network-Language-Model.git)", "url": "https://wpnews.pro/news/i-scaled-a-pure-spiking-neural-network-snn-to-1-088b-parameters-from-scratch-ran", "canonical_source": "https://dev.to/gtausa197svg/i-scaled-a-pure-spiking-neural-network-snn-to-1088b-parameters-from-scratch-ran-out-of-budget-3pg7", "published_at": "2026-06-17 10:26:23+00:00", "updated_at": "2026-06-17 10:51:35.668040+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "neural-networks", "ai-research", "ai-infrastructure"], "entities": ["Spiking Neural Network", "SNN", "SpikeBERT", "Loihi", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/i-scaled-a-pure-spiking-neural-network-snn-to-1-088b-parameters-from-scratch-ran", "markdown": "https://wpnews.pro/news/i-scaled-a-pure-spiking-neural-network-snn-to-1-088b-parameters-from-scratch-ran.md", "text": "https://wpnews.pro/news/i-scaled-a-pure-spiking-neural-network-snn-to-1-088b-parameters-from-scratch-ran.txt", "jsonld": "https://wpnews.pro/news/i-scaled-a-pure-spiking-neural-network-snn-to-1-088b-parameters-from-scratch-ran.jsonld"}}