Building a Tiny LM from Scratch in Node.js

A developer built a tiny language model from scratch in Node.js using a custom autograd engine and explicit neuron implementations, achieving correct outputs on basic logic tests after pre-training and supervised fine-tuning. The model, which uses two causal Transformer blocks with multi-head self-attention and an Adam optimizer implemented from scratch, starts with random weights and fails initially but reaches >95% target token probability after training. The project is available on GitHub at https://github.com/sekretov/tiny-language-model-neuro-js and requires Node.js 18.19+.

Building a Tiny LM from Scratch in Node.js The architecture follows the standard pipeline: tokenization → embeddings → causal Transformer blocks → LM head → softmax → loss. Because there's no tensor API hiding the logic, you can actually see the weight matrices changing in the logs. The "No-Magic" Implementation Most frameworks treat the computation graph as a black box. In this setup, every number is a Value object that tracks its own gradient and children: js class Value { constructor data, children = , backward = = {} { this.data = data; this.grad = 0; this.children = children; this. backward = backward; } } When a multiplication happens y = a × b , the operation stores the local derivatives dy/da = b, dy/db = a . The backward function then performs a topological sort of the graph to apply the chain rule from the loss back to the weights. Even the neuron implementation is explicit rather than being a matrix operation: js forward input { let output = sum input.map value, i = value.mul this.weights i ; if this.useBias output = output.add this.bias ; if this.activation === 'relu' return output.relu ; return output; } Performance and Results Since this avoids optimized BLAS libraries, it's significantly slower than a production LLM agent, but the educational value is huge. The model starts with random weights and fails basic logic tests: can human read ? model: ? ... expected: human can read. WRONG After going through pre-training and SFT Supervised Fine-Tuning , it hits a stable criterion where target tokens reach 95% probability, resulting in: can human read ? model: human can read. CORRECT can cat read ? model: cat cannot read. CORRECT Technical Breakdown Architecture: Two causal Transformer blocks with multi-head self-attention. Optimization: Adam optimizer implemented from scratch. Embeddings: Combined token and position embeddings to handle sequence order. Requirement: Node.js 18.19+ For anyone wanting a deep dive into the actual mechanics of a Transformer without the overhead of a massive framework, this is a perfect practical tutorial. https://github.com/sekretov/tiny-language-model-neuro-js Next AI from Scratch: What It Actually Is → /en/threads/3270/