{"slug": "ml-without-magic-building-a-tiny-language-model-in-pure-node-js-and-watching", "title": "ML Without Magic: Building a Tiny Language Model in Pure Node.js and Watching Every Weight Change", "summary": "A developer built a tiny language model in pure Node.js with no dependencies, implementing tokenization, embeddings, causal Transformer blocks, and backpropagation from scratch. The model trains on simple sentence patterns and demonstrates correct generalization after adaptive supervised fine-tuning, with every weight change visible through a custom Value class for autograd.", "body_md": "Tokenization → embeddings → causal Transformer → LM head → softmax → loss → backpropagation. No TensorFlow, no PyTorch, and no hidden autograd.\n\nRepository: ** tiny-language-model-neuro-js**.\n\nMost explanations of language models present correct formulas but hide the path between them inside a framework. I wanted the opposite: one small scenario where every scalar is visible and where the terminal clearly shows incorrect answers before learning and correct answers after it.\n\nThe project now has one command:\n\n```\nnode src/train.js --generalize --adaptive-teach\n```\n\nIt requires Node.js 18.19+ and has no dependencies.\n\nThe model is queried immediately after random initialization:\n\n```\nBEFORE TRAINING — random, usually wrong answers\n> can human read ?\n  model:    ? <unk> ...\n  expected: human can read.  [WRONG]\n\n> can fish swim ?\n  model:    ? <unk> ...\n  expected: fish can swim.   [WRONG]\n\n> can cat read ?\n  model:    ? <unk> ...\n  expected: cat cannot read. [WRONG]\n```\n\nAfter pre-training, SFT, and adaptive SFT, the same model produces:\n\n```\nFINAL ANSWERS AFTER ADAPTIVE SFT\n> can human read ?\n  model:    human can read.  [CORRECT]\n> can fish swim ?\n  model:    fish can swim.   [CORRECT]\n> can bird fly ?\n  model:    bird can fly.    [CORRECT]\n> can cat read ?\n  model:    cat cannot read. [CORRECT]\n\nRehearsal controls preserved: 14/14.\nStable criterion reached 11 times in a row.\n```\n\nThe initial text varies because initialization is random. The final acceptance criterion does not: all answers must be correct, every target token must have at least 95% probability, and the complete check must pass more than ten times consecutively.\n\nThe code previously contained several debug and training modes. They were useful while experimenting but obscured the main idea. The final version keeps one educational pipeline:\n\n```\ntext → word tokenization → token IDs\n     → token + position embeddings\n     → two causal Transformer blocks\n        → multi-head self-attention\n        → two-hidden-layer FFN\n     → LM head → softmax → next-token probabilities\n     → cross-entropy → backpropagation → Adam\n```\n\n`train.js`\n\nnow reads as one story rather than a command-line framework.\n\nEvery number participating in learning is a `Value`\n\n:\n\n``` js\nclass Value {\n  constructor(data, children = [], backward = () => {}) {\n    this.data = data;\n    this.grad = 0;\n    this.children = children;\n    this._backward = backward;\n  }\n}\n```\n\nFor multiplication:\n\n```\ny = a × b\ndy/da = b\ndy/db = a\n```\n\nThe operation stores these local derivatives. `backward()`\n\nsorts the graph topologically and applies the chain rule from the final loss back to embeddings and weights.\n\nThe neuron formula is not hidden behind a tensor API:\n\n```\noutput = activation(sum(input[i] × weight[i]) + bias)\n```\n\nIts implementation follows the formula:\n\n``` js\nforward(input) {\n  let output = sum(\n    input.map((value, i) => value.mul(this.weights[i]))\n  );\n\n  if (this.useBias) output = output.add(this.bias);\n  if (this.activation === 'relu') return output.relu();\n  return output;\n}\n```\n\nA `Linear`\n\nlayer is just an array of neurons receiving the same input. This is slower than matrix multiplication but far easier to inspect.\n\nEach token ID selects one trainable vector:\n\n```\ntoken representation = tokenEmbedding[id] + positionEmbedding[position]\n```\n\nEmbeddings contain random values initially. They acquire useful relations only because gradients repeatedly change them in training contexts. No `meaning`\n\nproperty is assigned to `cat`\n\n, `read`\n\n, or `cannot`\n\n.\n\nFor every token:\n\n```\nQ = X × Wq\nK = X × Wk\nV = X × Wv\n\nscore = dot(Q, K) / sqrt(headSize)\nattention = softmax(score)\noutput = attention × V\n```\n\nThe implementation loops only while `past <= position`\n\n. That is the causal mask: the model can attend to the current token and its history but never to a future target.\n\nAfter attention, every token passes through a two-hidden-layer feed-forward network:\n\n```\ndModel → hidden ReLU → hidden ReLU → dModel\n```\n\nLayerNorm and residual paths preserve stable information flow around attention and FFN.\n\nThe most important code in the project is only a few lines:\n\n``` js\nfunction learnOneToken({ model, optimizer, input, targetId }) {\n  const loss = model.loss(input, targetId);\n\n  optimizer.zeroGrad();\n  loss.backward();\n  optimizer.step();\n\n  return loss.data;\n}\n```\n\nThe loss is ordinary next-token cross-entropy:\n\n```\nloss = -log(P(target | previous tokens))\n```\n\nIf the correct token has low probability, loss is large. Backpropagation computes `dLoss/dWeight`\n\n; Adam changes each parameter; the next forward pass gives a different distribution.\n\nThe tiny world contains 14 ability relations:\n\n```\nhuman can read .\nfish can swim .\nbird can fly .\ndog cannot read .\n```\n\nThe `cat + read`\n\nrelation is missing deliberately. Pre-training samples positions from this text and learns ordinary next-token prediction.\n\nThe same relations are converted into 42 prompt-answer examples:\n\n```\ncan fish swim ?\nis fish able to swim ?\ndoes fish know how to swim ?\n```\n\nOnly answer tokens contribute to SFT loss. The implementation visits every pair and every answer position on each epoch, making the training loop deterministic and readable.\n\nThe missing answer is represented only by target tokens:\n\n```\n['cat', 'cannot', 'read', '.']\n```\n\nSix question variants receive those targets. This is direct supervision: the model did not discover a zoological fact on its own. The teacher introduced the fact through loss, and backpropagation distributed that information across embeddings, attention, FFN, LayerNorm, and the LM head.\n\nWhy not stop after one correct answer? Because one generation can be fragile. The loop continues until every target token exceeds 95% probability and the whole evaluation succeeds 11 times in a row.\n\nAn early implementation trained only the six new cat prompts. It successfully learned the new answer and destroyed old behavior:\n\n```\ncan human read ? → cat cannot read.\ncan fish swim ?  → cat cannot read.\n```\n\nThat is catastrophic forgetting in miniature. The fix is rehearsal: adaptive epochs also repeat the 14 older `can ... ?`\n\nexamples. The final criterion evaluates both new and old examples, so training cannot finish by overwriting everything with one response.\n\nThe command automatically creates:\n\n```\nlogs/training-log.txt\n```\n\nIt is a sequential ASCII diagram rather than a raw JSON dump. It includes every\n\nforward/loss/backward/update event, followed by the complete matrices at three\n\ncheckpoints:\n\n```\ninitial random matrices\n        |\n        v\nmatrices after pre-training + SFT\n        |\n        v\nfinal matrices after adaptive SFT\n```\n\nFor every transition, the log prints the AFTER matrix and its exact DELTA matrix.\n\nLinear rows are named `neuron[n]`\n\n, columns are named `weight[n]`\n\n, and biases\n\nare shown beside their neuron. It also points out the largest concrete change as\n\n`layer / neuron / weight: before -> after -> delta`\n\n.\n\nThe architecture and learning rule are real; the scale is intentionally tiny.\n\n| This model | Production model |\n|---|---|\n| 24 word tokens | Large subword/byte vocabulary |\n| 2,160 parameters | Millions or billions |\n| Two Transformer blocks | Tens or hundreds |\n| Scalar JavaScript graph | Batched tensor graph on accelerators |\n| Small structured corpus | Massive curated datasets |\n| Narrow trained behavior | Broad language and reasoning |\n\nThe project is not a GPT competitor. It is a causal language model reduced until the complete path fits in one repository and one mental model:\n\n```\ntoken → embedding → attention → FFN → probability\n      → loss → gradient → updated weight → changed answer\n```\n\nThat path is the point. Once it is visible, frameworks stop looking magical: they execute the same classes of operations at a scale and speed this scalar implementation deliberately avoids.\n\nRepository: ** tiny-language-model-neuro-js**.\n\nAuthor: ** Maksim Sekretov**.", "url": "https://wpnews.pro/news/ml-without-magic-building-a-tiny-language-model-in-pure-node-js-and-watching", "canonical_source": "https://dev.to/maktordev/ml-without-magic-building-a-tiny-language-model-in-pure-nodejs-and-watching-every-weight-change-5dfh", "published_at": "2026-07-25 09:34:59+00:00", "updated_at": "2026-07-25 10:02:09.491832+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools"], "entities": ["Node.js", "tiny-language-model-neuro-js", "Adam"], "alternates": {"html": "https://wpnews.pro/news/ml-without-magic-building-a-tiny-language-model-in-pure-node-js-and-watching", "markdown": "https://wpnews.pro/news/ml-without-magic-building-a-tiny-language-model-in-pure-node-js-and-watching.md", "text": "https://wpnews.pro/news/ml-without-magic-building-a-tiny-language-model-in-pure-node-js-and-watching.txt", "jsonld": "https://wpnews.pro/news/ml-without-magic-building-a-tiny-language-model-in-pure-node-js-and-watching.jsonld"}}