{"slug": "building-a-tiny-lm-from-scratch-in-node-js", "title": "Building a Tiny LM from Scratch in Node.js", "summary": "A developer built a tiny language model from scratch in Node.js using a custom autograd engine and explicit neuron implementations, achieving correct outputs on basic logic tests after pre-training and supervised fine-tuning. The model, which uses two causal Transformer blocks with multi-head self-attention and an Adam optimizer implemented from scratch, starts with random weights and fails initially but reaches >95% target token probability after training. The project is available on GitHub at https://github.com/sekretov/tiny-language-model-neuro-js and requires Node.js 18.19+.", "body_md": "# Building a Tiny LM from Scratch in Node.js\n\nThe architecture follows the standard pipeline: tokenization → embeddings → causal Transformer blocks → LM head → softmax → loss. Because there's no tensor API hiding the logic, you can actually see the weight matrices changing in the logs.\n\n## The \"No-Magic\" Implementation\n\nMost frameworks treat the computation graph as a black box. In this setup, every number is a `Value`\n\nobject that tracks its own gradient and children:\n\n``` js\nclass Value {\n constructor(data, children = [], backward = () => {}) {\n this.data = data;\n this.grad = 0;\n this.children = children;\n this._backward = backward;\n }\n}\n```\n\nWhen a multiplication happens (y = a × b), the operation stores the local derivatives (dy/da = b, dy/db = a). The `backward()`\n\nfunction then performs a topological sort of the graph to apply the chain rule from the loss back to the weights.\n\nEven the neuron implementation is explicit rather than being a matrix operation:\n\n``` js\nforward(input) {\n let output = sum(\n input.map((value, i) => value.mul(this.weights[i]))\n );\n\n if (this.useBias) output = output.add(this.bias);\n if (this.activation === 'relu') return output.relu();\n return output;\n}\n```\n\n## Performance and Results\n\nSince this avoids optimized BLAS libraries, it's significantly slower than a production LLM agent, but the educational value is huge. The model starts with random weights and fails basic logic tests:\n\n```\n> can human read ?\n model: ? ...\n expected: human can read. [WRONG]\n```\n\nAfter going through pre-training and SFT (Supervised Fine-Tuning), it hits a stable criterion where target tokens reach >95% probability, resulting in:\n\n```\n> can human read ?\n model: human can read. [CORRECT]\n> can cat read ?\n model: cat cannot read. [CORRECT]\n```\n\n## Technical Breakdown\n\n**Architecture:** Two causal Transformer blocks with multi-head self-attention.**Optimization:** Adam optimizer implemented from scratch.**Embeddings:** Combined token and position embeddings to handle sequence order.**Requirement:** Node.js 18.19+\n\nFor anyone wanting a deep dive into the actual mechanics of a Transformer without the overhead of a massive framework, this is a perfect practical tutorial.\n\n```\nhttps://github.com/sekretov/tiny-language-model-neuro-js\n```\n\n[Next AI from Scratch: What It Actually Is →](/en/threads/3270/)", "url": "https://wpnews.pro/news/building-a-tiny-lm-from-scratch-in-node-js", "canonical_source": "https://promptcube3.com/en/threads/3274/", "published_at": "2026-07-25 16:46:37+00:00", "updated_at": "2026-07-25 17:06:29.186978+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools"], "entities": ["Node.js", "GitHub", "Adam optimizer", "Transformer"], "alternates": {"html": "https://wpnews.pro/news/building-a-tiny-lm-from-scratch-in-node-js", "markdown": "https://wpnews.pro/news/building-a-tiny-lm-from-scratch-in-node-js.md", "text": "https://wpnews.pro/news/building-a-tiny-lm-from-scratch-in-node-js.txt", "jsonld": "https://wpnews.pro/news/building-a-tiny-lm-from-scratch-in-node-js.jsonld"}}