{"slug": "from-zero-to-shakespeare-gpt-i-built-a-language-model-in-javascript", "title": "From Zero to Shakespeare GPT: I Built a Language Model in JavaScript", "summary": "A developer built a character-level GPT language model entirely from scratch in JavaScript, including a custom Tensor implementation with autograd, a full Transformer stack (embeddings, causal multi-head attention, layer norm, GELU, feed-forward blocks), a training loop with optimizer and checkpoints, and a text-generation pipeline. The model was trained on Andrej Karpathy's roughly 1.1-million-character Tiny Shakespeare corpus, yielding a 65-token character vocabulary, with the codebase deliberately split into modules for data, tokenizer, tensor, layers, models, training and generation to make debugging tractable.", "body_md": "By the time I reached this stage, I had spent a considerable amount of time learning how neural networks work.\n\nI had implemented neurons, trained MLPs, worked through backpropagation, experimented with activation functions, and eventually built the components of a Transformer.\n\nBut there was still one thing missing.\n\nI hadn't brought everything together into a language model that could learn from actual text.\n\nThat was the goal from the beginning.\n\nI wanted to take everything I had learned and build a tiny GPT from scratch using JavaScript.\n\nNot a chatbot powered by an API. Not a wrapper around a pretrained model.\n\nAn actual neural network with its own trainable weights, attention mechanism, training loop and text-prediction pipeline.\n\nAnd for the dataset, I chose Shakespeare.\n\nI needed a real text corpus to train the model.\n\nI didn't want to keep testing everything with small artificial examples. I wanted the model to learn from actual sequences of language.\n\nShakespeare was a good starting point for a few reasons:\n\nI used the Tiny Shakespeare corpus from Andrej Karpathy's \"char-rnn\" project.\n\nThe dataset contains roughly 1.1 million characters and, with our character-level tokeniser, produces a vocabulary of 65 unique characters.\n\nThat meant our model had 65 possible character IDs to work with.\n\nThis was the first time our learning experiments were connected to a meaningful language dataset.\n\nOne thing I had learned from the earlier experiments was that putting everything into one enormous file would make the project difficult to understand.\n\nI wanted the architecture to reflect the actual concepts we were learning.\n\nSo I separated the implementation into modules.\n\n```\ntiny-shakespeare-gpt/\n│\n├── data/\n│   ├── shakespeare.txt\n│   ├── nextTokenDataset.js\n│   └── prepareData.js\n│\n├── tokenizer/\n│   └── tokenizer.js\n│\n├── tensor/\n│   ├── Tensor.js\n│   ├── operations.js\n│   ├── autograd.js\n│   └── random.js\n│\n├── layers/\n│   ├── embedding.js\n│   ├── positionalEmbedding.js\n│   ├── linear.js\n│   ├── layerNorm.js\n│   ├── gelu.js\n│   ├── causalAttention.js\n│   ├── selfAttentionHead.js\n│   ├── multiHeadAttention.js\n│   ├── feedForward.js\n│   └── transformerBlock.js\n│\n├── models/\n│   └── gptModel.js\n│\n├── training/\n│   ├── loss.js\n│   ├── optimizer.js\n│   ├── trainStep.js\n│   ├── trainLoop.js\n│   └── checkpoint.js\n│\n└── generation/\n    └── generate.js\n```\n\nEach folder had a responsibility.\n\nThe tokeniser handled text. The tensor module handled numerical operations and gradients. The layers contained the neural-network components. The model assembled the architecture. The training module handled learning.\n\nIt might seem like ordinary software organisation, but it mattered a lot here.\n\nWhen something went wrong, I could investigate a particular operation or layer instead of debugging one massive file.\n\n*The project is split into one folder per concept: data, tokenizer, tensor, layers, models, training and generation.*\n\nThis was one of the most important implementation decisions.\n\nMachine-learning frameworks normally provide tensors and automatic differentiation.\n\nI wasn't using one.\n\nSo I needed my own basic Tensor structure.\n\nAt its simplest, a Tensor stores:\n\nConceptually:\n\n```\nclass Tensor {\n  constructor(data, shape) {\n    this.data = data;\n    this.shape = shape;\n    this.grad = new Array(data.length).fill(0);\n\n    this.parent = [];\n    this.backward = () => {};\n  }\n}\n```\n\nThe `parent` references allow us to track the computation graph.\n\nWhen we perform an operation, we create a new Tensor and record which earlier tensors contributed to it.\n\nFor example:\n\n``` php\nA ----\\\n       Add ----> C\nB ----/\n```\n\nIf we want to calculate gradients for C, we need to propagate them back to A and B.\n\nI implemented operations such as addition, multiplication, matrix multiplication, transpose, exponential, logarithm and square root, with their corresponding backward functions.\n\nThen I implemented a topological traversal of the computation graph so that backward operations could execute in the correct order.\n\nThis is the foundation of automatic differentiation.\n\n*Each Tensor records its parents. The forward pass builds the graph (A + B → C), and backward functions run in topological order to send gradients from C back to A and B.*\n\nIt was also a very different experience from calling a framework's `backward()` function without knowing what happened underneath.\n\nWith tensors and gradients available, I could assemble the actual model components.\n\nThe implementation included:\n\n**Embedding:** Converts token IDs into learned vectors.\n\n**Positional embedding:** Adds learned position information to the token representations.\n\n**Linear layers:** Apply trainable weight matrices and biases.\n\n**LayerNorm:** Normalises feature values and learns scale and shift parameters.\n\n**GELU:** Adds the nonlinearity used in the feed-forward network.\n\n**Causal attention:** Calculates attention while preventing access to future positions.\n\n**Multi-head attention:** Combines multiple attention heads.\n\n**Feed-forward network:** Applies learned transformations to each token representation.\n\n**Transformer block:** Combines attention, residual connections, normalisation and feed-forward layers.\n\nThe most interesting part was not writing each class independently.\n\nIt was making sure their dimensions and gradients connected correctly.\n\nFor example, our multi-head attention had to combine the outputs of individual heads and project the concatenated representation back into the model dimension.\n\nThe Transformer block then had to preserve the input shape so that residual connections worked.\n\nA mistake in any of these interfaces could break the entire model.\n\nNow came the actual language-modelling task.\n\nWe don't need to teach the model to generate an entire paragraph in one operation.\n\nInstead, we train it to predict the next character.\n\nConsider:\n\n```\nInput:  HELLO\nTarget: ELLO!\n```\n\nFor a sequence of input characters:\n\n```\nH E L L O\n```\n\nThe targets are the next characters:\n\n```\nE L L O !\n```\n\nEvery position in the sequence contributes a prediction task.\n\nThis is called next-token prediction.\n\nOur dataset helper creates input and target sequences by shifting the token IDs by one position.\n\nIn simplified JavaScript:\n\n``` js\nfunction createNextTokenPairs(tokenIds, start, length) {\n  const inputs = [];\n  const targets = [];\n\n  for (let i = 0; i < length; i++) {\n    inputs.push(tokenIds[start + i]);\n    targets.push(tokenIds[start + i + 1]);\n  }\n\n  return { inputs, targets };\n}\n```\n\nOur actual batching implementation also samples multiple random starting positions and combines them into batches.\n\nFor example, with:\n\n```\nBatch size: 32\nSequence length: 256\n```\n\nThe model receives 32 sequences, each containing 256 characters.\n\nThe positional embedding matrix is sized to the configured maximum sequence length, so here it has 256 rows.\n\nIt predicts the next character at every position.\n\nThis gives us 32 × 256 prediction targets in one training step.\n\nAnd because we use causal attention, the model cannot look at future characters when making those predictions.\n\n*Next-token prediction: the target sequence is the input shifted by one position. With batch size 32 and sequence length 256, one step has 32 × 256 prediction targets.*\n\nThe GPT model produces logits.\n\nFor our 65-character vocabulary, every sequence position produces 65 scores.\n\nThese aren't probabilities yet.\n\nWe apply softmax to turn the scores into a probability distribution:\n\nFor training, we use cross-entropy loss.\n\nFor a single correct target token:\n\nIf the model assigns a high probability to the correct next character, the loss is low.\n\nIf it assigns a very low probability, the loss is high.\n\n*Logits become probabilities through softmax, and cross-entropy is −log of the probability given to the correct character, averaged over the target positions.*\n\nFor a batch of next-character predictions, we average the loss across the target positions.\n\nI implemented stable softmax cross-entropy using a log-sum-exp approach to avoid numerical overflow.\n\nThis was important because numerical stability isn't optional when you're training a model.\n\nA mathematically correct formula can still produce problematic numerical values if implemented carelessly.\n\nThis was a milestone.\n\nWe loaded the actual Shakespeare dataset, built the tokeniser, prepared a batch and passed it through our GPT model.\n\nThe first real training experiment used:\n\n```\nVocabulary size: 65\nBatch size: 32\nSequence length: 256\n```\n\nThe model processed actual Shakespeare passages.\n\nWe also decoded the input and target IDs back into text to verify that the dataset was aligned correctly.\n\nThen we ran the first training step.\n\nThe output included:\n\n```\nRunning one training step...\n\nLoss: 5.4462\n\nPASS: First Shakespeare training step completed.\n```\n\nThat number wasn't something to celebrate as a good language model yet.\n\nIt was something more fundamental:\n\nThe entire forward pass, loss calculation, backpropagation and parameter update had worked on real text.\n\nThe model had completed an actual learning step.\n\nAfter all the individual mathematical experiments, seeing the components work together was a satisfying moment.\n\nGetting one training step to work was only the beginning.\n\nThe next question was whether the model could actually learn.\n\nI ran an experiment where the model repeatedly trained on the same fixed batch.\n\nThis was a separate run, which is why its starting loss differs from the single training step above.\n\nThe results:\n\n```\nINITIAL LOSS: 4.9393\n\nStep 10  | Loss: 4.6922\nStep 20  | Loss: 4.4776\nStep 30  | Loss: 4.3167\nStep 40  | Loss: 4.1928\nStep 50  | Loss: 4.0918\nStep 60  | Loss: 4.0043\nStep 70  | Loss: 3.9256\nStep 80  | Loss: 3.8529\nStep 90  | Loss: 3.7848\nStep 100 | Loss: 3.7202\n\nFINAL LOSS: 3.7139\n```\n\nThe loss decreased from approximately 4.94 to 3.71.\n\n*Loss on one fixed batch went from 4.9393 at the start to 3.7202 at step 100, with a final loss of 3.7139. Because it was the same batch every step, this doesn't show generalisation.*\n\nThat was encouraging.\n\nBut there was an important catch.\n\nWe were repeatedly training on the same batch.\n\nThe model was getting better at predicting that particular batch, but that didn't prove it had learned patterns that generalised to unseen Shakespeare passages.\n\nThis is where training loss and validation loss become important.\n\nA model can become very good at the examples it has already seen without becoming equally good at new examples.\n\nThat's overfitting.\n\nTo investigate this, we split the dataset into two parts:\n\nThe separation is important.\n\nDuring validation, we perform a forward pass and calculate the loss. We do not call backpropagation or the optimizer.\n\nThis lets us ask two different questions:\n\nIf training loss decreases while validation loss starts increasing, that can be a sign of overfitting.\n\n*Training steps run forward, loss, backprop and the optimizer. Validation runs only the forward pass and loss, so the weights don't change.*\n\nI wanted the project to include this distinction from the beginning because a decreasing training loss alone is not enough to claim that a language model is learning well.\n\nAnother practical requirement appeared once training started working.\n\nIf I trained the model and then closed the program, I didn't want to lose the learned weights.\n\nI implemented checkpoint saving.\n\nA checkpoint stores the model's configuration and its trainable parameter values.\n\nIn simplified form:\n\n``` js\nconst checkpoint = {\n  config,\n  parameters: model.parameters().map(parameter => ({\n    shape: parameter.shape,\n    data: parameter.data\n  }))\n};\n```\n\nWhy save the shape as well as the data?\n\nBecause a parameter's dimensions are part of the model architecture.\n\nA matrix with shape `[65, 32]` is not interchangeable with one having shape `[32, 65]`, even if both contain the same number of values.\n\nI also implemented checkpoint loading, including checks for parameter counts, tensor shapes and data lengths.\n\nThen came the verification.\n\nWe trained Model A, saved its weights, created a completely fresh Model B, and loaded the checkpoint into it.\n\nBoth models received the same input.\n\nThe result:\n\n```\nCHECKPOINT VERIFICATION\n\nModel A loss: 5.11065033\nModel B loss: 5.11065033\n\nMaximum output difference: 0\n\nPASS: Model weights saved and restored correctly.\n```\n\nThis was a particularly useful test.\n\nIt demonstrated that we could take the learned parameters from one model instance and restore them into another without changing the model's predictions.\n\nThat is the foundation for a model that can be trained once and reused later.\n\n*Model A's config and parameters (shape + data) are saved and loaded into a fresh Model B. Both give a loss of 5.11065033 with a maximum output difference of 0.*\n\nTraining and generation are two different processes.\n\nDuring training, the model receives an input sequence and learns to predict the next character at every position.\n\nDuring generation, we start with a prompt and repeatedly ask the model for the next character.\n\nThe process looks like this:\n\n```\nPrompt: \"ROMEO:\"\n       |\n       v\nConvert prompt to token IDs\n       |\n       v\nPass tokens through GPT\n       |\n       v\nRead logits at final position\n       |\n       v\nApply temperature and softmax\n       |\n       v\nSample next character\n       |\n       v\nAppend character to prompt\n       |\n       v\nRepeat\n```\n\nOne parameter that makes generation interesting is temperature.\n\nBefore sampling, we divide logits by temperature:\n\nThen apply softmax.\n\nA lower temperature makes the probability distribution more concentrated, favouring higher-scoring characters.\n\nA higher temperature makes it more spread out, allowing more varied choices.\n\nThis doesn't make the model more intelligent. It changes how we sample from its predictions.\n\n*Generation is a loop: encode the prompt, run GPT, read the final position's logits, apply temperature and softmax, sample, append and repeat. Lower T concentrates the distribution, and higher T spreads it out.*\n\nThere is also a context-window limit.\n\nOur model can only process a configured maximum number of characters at once. As generation continues, we keep the most recent part of the context within that limit.\n\nThis is one of the practical differences between the simple idea of next-character prediction and a working autoregressive generation system.\n\nIt's important to be realistic about what we've built.\n\nThis is a small, character-level GPT trained on a limited Shakespeare corpus.\n\nIt is not a ChatGPT replacement.\n\nIt doesn't have the scale, knowledge, instruction-following abilities or reasoning capabilities of a modern large language model.\n\nIt learns statistical patterns in the training text and uses those patterns to predict the next character.\n\nBut that's exactly why it is useful as a learning project.\n\nWe can inspect its tensors. We can examine its weights. We can follow the forward pass. We can understand the attention calculation. We can see how the loss changes when the model learns.\n\nThere isn't a pretrained model hiding all those details from us.\n\nThe next stage is to move beyond running isolated training scripts.\n\nI want the final project to behave like an actual application.\n\nThe plan is to build a Node.js server that:\n\nThe intention is simple:\n\nTrain once, save the model, restart the server, and continue using the trained model without starting from random weights every time.\n\nI also want to publish the project on GitHub so other developers can explore the implementation and experiment with it.\n\nWhen I started on 30 July 2026, I had a question about how AI models work.\n\nI thought I would probably spend some time reading about neural networks and Transformers.\n\nInstead, the journey took me through:\n\n`NaN` values.\nThe biggest lesson wasn't that I could now build a large language model.\n\nI couldn't, and that was never the goal.\n\nThe lesson was that a technology that once seemed like a black box became understandable when I broke it into smaller pieces and implemented those pieces myself.\n\nI also learned that knowing a formula and understanding an implementation are two very different things.\n\nYou can read about attention in five minutes. Understanding the tensor shapes, gradients, masking and numerical stability takes much more work.\n\nAnd that work is worth it.\n\nI started this project because I didn't want to remain someone who only knows how to use AI tools.\n\nI wanted to understand what happens behind them.\n\nBuilding a tiny GPT didn't answer every question I have about AI. In fact, it gave me many more questions to explore.\n\nBut it changed how I look at these models.\n\nThe next time I use an LLM, I won't see only a prompt and a response. I'll think about tokenisation, embeddings, attention, learned weights, probability distributions and all the computation happening underneath.\n\nAnd that's what I wanted from this journey.\n\n**Not to build the biggest model. To understand the model.**\n\n**Source code:** [github.com/Roshan190/tiny-shakespeare-gpt-js](https://github.com/Roshan190/tiny-shakespeare-gpt-js)\n\nThanks for following Beyond the Prompt. This is the end of the first journey, not the end of my exploration of AI.\n\n*Project journey started: 30 July 2026.*", "url": "https://wpnews.pro/news/from-zero-to-shakespeare-gpt-i-built-a-language-model-in-javascript", "canonical_source": "https://dev.to/roshanshikhare/from-zero-to-shakespeare-gpt-i-built-a-language-model-in-javascript-431b", "published_at": "2026-11-16 04:19:33+00:00", "updated_at": "2026-11-16 04:20:58.443149+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "neural-networks", "ai-research"], "entities": ["JavaScript", "Andrej Karpathy", "Tiny Shakespeare", "char-rnn", "Transformer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/from-zero-to-shakespeare-gpt-i-built-a-language-model-in-javascript", "markdown": "https://wpnews.pro/news/from-zero-to-shakespeare-gpt-i-built-a-language-model-in-javascript.md", "text": "https://wpnews.pro/news/from-zero-to-shakespeare-gpt-i-built-a-language-model-in-javascript.txt", "jsonld": "https://wpnews.pro/news/from-zero-to-shakespeare-gpt-i-built-a-language-model-in-javascript.jsonld"}}