By the time I reached this stage, I had spent a considerable amount of time learning how neural networks work.
I had implemented neurons, trained MLPs, worked through backpropagation, experimented with activation functions, and eventually built the components of a Transformer.
But there was still one thing missing.
I hadn't brought everything together into a language model that could learn from actual text.
That was the goal from the beginning.
I wanted to take everything I had learned and build a tiny GPT from scratch using JavaScript.
Not a chatbot powered by an API. Not a wrapper around a pretrained model.
An actual neural network with its own trainable weights, attention mechanism, training loop and text-prediction pipeline.
And for the dataset, I chose Shakespeare.
I needed a real text corpus to train the model.
I didn't want to keep testing everything with small artificial examples. I wanted the model to learn from actual sequences of language.
Shakespeare was a good starting point for a few reasons:
I used the Tiny Shakespeare corpus from Andrej Karpathy's "char-rnn" project.
The dataset contains roughly 1.1 million characters and, with our character-level tokeniser, produces a vocabulary of 65 unique characters.
That meant our model had 65 possible character IDs to work with.
This was the first time our learning experiments were connected to a meaningful language dataset.
One thing I had learned from the earlier experiments was that putting everything into one enormous file would make the project difficult to understand.
I wanted the architecture to reflect the actual concepts we were learning.
So I separated the implementation into modules.
tiny-shakespeare-gpt/
β
βββ data/
β βββ shakespeare.txt
β βββ nextTokenDataset.js
β βββ prepareData.js
β
βββ tokenizer/
β βββ tokenizer.js
β
βββ tensor/
β βββ Tensor.js
β βββ operations.js
β βββ autograd.js
β βββ random.js
β
βββ layers/
β βββ embedding.js
β βββ positionalEmbedding.js
β βββ linear.js
β βββ layerNorm.js
β βββ gelu.js
β βββ causalAttention.js
β βββ selfAttentionHead.js
β βββ multiHeadAttention.js
β βββ feedForward.js
β βββ transformerBlock.js
β
βββ models/
β βββ gptModel.js
β
βββ training/
β βββ loss.js
β βββ optimizer.js
β βββ trainStep.js
β βββ trainLoop.js
β βββ checkpoint.js
β
βββ generation/
βββ generate.js
Each folder had a responsibility.
The tokeniser handled text. The tensor module handled numerical operations and gradients. The layers contained the neural-network components. The model assembled the architecture. The training module handled learning.
It might seem like ordinary software organisation, but it mattered a lot here.
When something went wrong, I could investigate a particular operation or layer instead of debugging one massive file.
The project is split into one folder per concept: data, tokenizer, tensor, layers, models, training and generation.
This was one of the most important implementation decisions.
Machine-learning frameworks normally provide tensors and automatic differentiation.
I wasn't using one.
So I needed my own basic Tensor structure.
At its simplest, a Tensor stores:
Conceptually:
class Tensor {
constructor(data, shape) {
this.data = data;
this.shape = shape;
this.grad = new Array(data.length).fill(0);
this.parent = [];
this.backward = () => {};
}
}
The parent references allow us to track the computation graph.
When we perform an operation, we create a new Tensor and record which earlier tensors contributed to it.
For example:
A ----\
Add ----> C
B ----/
If we want to calculate gradients for C, we need to propagate them back to A and B.
I implemented operations such as addition, multiplication, matrix multiplication, transpose, exponential, logarithm and square root, with their corresponding backward functions.
Then I implemented a topological traversal of the computation graph so that backward operations could execute in the correct order.
This is the foundation of automatic differentiation.
Each Tensor records its parents. The forward pass builds the graph (A + B β C), and backward functions run in topological order to send gradients from C back to A and B.
It was also a very different experience from calling a framework's backward() function without knowing what happened underneath.
With tensors and gradients available, I could assemble the actual model components.
The implementation included:
Embedding: Converts token IDs into learned vectors.
Positional embedding: Adds learned position information to the token representations.
Linear layers: Apply trainable weight matrices and biases.
LayerNorm: Normalises feature values and learns scale and shift parameters.
GELU: Adds the nonlinearity used in the feed-forward network.
Causal attention: Calculates attention while preventing access to future positions.
Multi-head attention: Combines multiple attention heads.
Feed-forward network: Applies learned transformations to each token representation.
Transformer block: Combines attention, residual connections, normalisation and feed-forward layers.
The most interesting part was not writing each class independently.
It was making sure their dimensions and gradients connected correctly.
For example, our multi-head attention had to combine the outputs of individual heads and project the concatenated representation back into the model dimension.
The Transformer block then had to preserve the input shape so that residual connections worked.
A mistake in any of these interfaces could break the entire model.
Now came the actual language-modelling task.
We don't need to teach the model to generate an entire paragraph in one operation.
Instead, we train it to predict the next character.
Consider:
Input: HELLO
Target: ELLO!
For a sequence of input characters:
H E L L O
The targets are the next characters:
E L L O !
Every position in the sequence contributes a prediction task.
This is called next-token prediction.
Our dataset helper creates input and target sequences by shifting the token IDs by one position.
In simplified JavaScript:
function createNextTokenPairs(tokenIds, start, length) {
const inputs = [];
const targets = [];
for (let i = 0; i < length; i++) {
inputs.push(tokenIds[start + i]);
targets.push(tokenIds[start + i + 1]);
}
return { inputs, targets };
}
Our actual batching implementation also samples multiple random starting positions and combines them into batches.
For example, with:
Batch size: 32
Sequence length: 256
The model receives 32 sequences, each containing 256 characters.
The positional embedding matrix is sized to the configured maximum sequence length, so here it has 256 rows.
It predicts the next character at every position.
This gives us 32 Γ 256 prediction targets in one training step.
And because we use causal attention, the model cannot look at future characters when making those predictions.
Next-token prediction: the target sequence is the input shifted by one position. With batch size 32 and sequence length 256, one step has 32 Γ 256 prediction targets.
The GPT model produces logits.
For our 65-character vocabulary, every sequence position produces 65 scores.
These aren't probabilities yet.
We apply softmax to turn the scores into a probability distribution:
For training, we use cross-entropy loss.
For a single correct target token:
If the model assigns a high probability to the correct next character, the loss is low.
If it assigns a very low probability, the loss is high.
Logits become probabilities through softmax, and cross-entropy is βlog of the probability given to the correct character, averaged over the target positions.
For a batch of next-character predictions, we average the loss across the target positions.
I implemented stable softmax cross-entropy using a log-sum-exp approach to avoid numerical overflow.
This was important because numerical stability isn't optional when you're training a model.
A mathematically correct formula can still produce problematic numerical values if implemented carelessly.
This was a milestone.
We loaded the actual Shakespeare dataset, built the tokeniser, prepared a batch and passed it through our GPT model.
The first real training experiment used:
Vocabulary size: 65
Batch size: 32
Sequence length: 256
The model processed actual Shakespeare passages.
We also decoded the input and target IDs back into text to verify that the dataset was aligned correctly.
Then we ran the first training step.
The output included:
Running one training step...
Loss: 5.4462
PASS: First Shakespeare training step completed.
That number wasn't something to celebrate as a good language model yet.
It was something more fundamental:
The entire forward pass, loss calculation, backpropagation and parameter update had worked on real text.
The model had completed an actual learning step.
After all the individual mathematical experiments, seeing the components work together was a satisfying moment.
Getting one training step to work was only the beginning.
The next question was whether the model could actually learn.
I ran an experiment where the model repeatedly trained on the same fixed batch.
This was a separate run, which is why its starting loss differs from the single training step above.
The results:
INITIAL LOSS: 4.9393
Step 10 | Loss: 4.6922
Step 20 | Loss: 4.4776
Step 30 | Loss: 4.3167
Step 40 | Loss: 4.1928
Step 50 | Loss: 4.0918
Step 60 | Loss: 4.0043
Step 70 | Loss: 3.9256
Step 80 | Loss: 3.8529
Step 90 | Loss: 3.7848
Step 100 | Loss: 3.7202
FINAL LOSS: 3.7139
The loss decreased from approximately 4.94 to 3.71.
Loss on one fixed batch went from 4.9393 at the start to 3.7202 at step 100, with a final loss of 3.7139. Because it was the same batch every step, this doesn't show generalisation.
That was encouraging.
But there was an important catch.
We were repeatedly training on the same batch.
The model was getting better at predicting that particular batch, but that didn't prove it had learned patterns that generalised to unseen Shakespeare passages.
This is where training loss and validation loss become important.
A model can become very good at the examples it has already seen without becoming equally good at new examples.
That's overfitting.
To investigate this, we split the dataset into two parts:
The separation is important.
During validation, we perform a forward pass and calculate the loss. We do not call backpropagation or the optimizer.
This lets us ask two different questions:
If training loss decreases while validation loss starts increasing, that can be a sign of overfitting.
Training steps run forward, loss, backprop and the optimizer. Validation runs only the forward pass and loss, so the weights don't change.
I wanted the project to include this distinction from the beginning because a decreasing training loss alone is not enough to claim that a language model is learning well.
Another practical requirement appeared once training started working.
If I trained the model and then closed the program, I didn't want to lose the learned weights.
I implemented checkpoint saving.
A checkpoint stores the model's configuration and its trainable parameter values.
In simplified form:
const checkpoint = {
config,
parameters: model.parameters().map(parameter => ({
shape: parameter.shape,
data: parameter.data
}))
};
Why save the shape as well as the data?
Because a parameter's dimensions are part of the model architecture.
A matrix with shape [65, 32] is not interchangeable with one having shape [32, 65], even if both contain the same number of values.
I also implemented checkpoint , including checks for parameter counts, tensor shapes and data lengths.
Then came the verification.
We trained Model A, saved its weights, created a completely fresh Model B, and loaded the checkpoint into it.
Both models received the same input.
The result:
CHECKPOINT VERIFICATION
Model A loss: 5.11065033
Model B loss: 5.11065033
Maximum output difference: 0
PASS: Model weights saved and restored correctly.
This was a particularly useful test.
It demonstrated that we could take the learned parameters from one model instance and restore them into another without changing the model's predictions.
That is the foundation for a model that can be trained once and reused later.
Model A's config and parameters (shape + data) are saved and loaded into a fresh Model B. Both give a loss of 5.11065033 with a maximum output difference of 0.
Training and generation are two different processes.
During training, the model receives an input sequence and learns to predict the next character at every position.
During generation, we start with a prompt and repeatedly ask the model for the next character.
The process looks like this:
Prompt: "ROMEO:"
|
v
Convert prompt to token IDs
|
v
Pass tokens through GPT
|
v
Read logits at final position
|
v
Apply temperature and softmax
|
v
Sample next character
|
v
Append character to prompt
|
v
Repeat
One parameter that makes generation interesting is temperature.
Before sampling, we divide logits by temperature:
Then apply softmax.
A lower temperature makes the probability distribution more concentrated, favouring higher-scoring characters.
A higher temperature makes it more spread out, allowing more varied choices.
This doesn't make the model more intelligent. It changes how we sample from its predictions.
Generation is a loop: encode the prompt, run GPT, read the final position's logits, apply temperature and softmax, sample, append and repeat. Lower T concentrates the distribution, and higher T spreads it out.
There is also a context-window limit.
Our model can only process a configured maximum number of characters at once. As generation continues, we keep the most recent part of the context within that limit.
This is one of the practical differences between the simple idea of next-character prediction and a working autoregressive generation system.
It's important to be realistic about what we've built.
This is a small, character-level GPT trained on a limited Shakespeare corpus.
It is not a ChatGPT replacement.
It doesn't have the scale, knowledge, instruction-following abilities or reasoning capabilities of a modern large language model.
It learns statistical patterns in the training text and uses those patterns to predict the next character.
But that's exactly why it is useful as a learning project.
We can inspect its tensors. We can examine its weights. We can follow the forward pass. We can understand the attention calculation. We can see how the loss changes when the model learns.
There isn't a pretrained model hiding all those details from us.
The next stage is to move beyond running isolated training scripts.
I want the final project to behave like an actual application.
The plan is to build a Node.js server that:
The intention is simple:
Train once, save the model, restart the server, and continue using the trained model without starting from random weights every time.
I also want to publish the project on GitHub so other developers can explore the implementation and experiment with it.
When I started on 30 July 2026, I had a question about how AI models work.
I thought I would probably spend some time reading about neural networks and Transformers.
Instead, the journey took me through:
NaN values.
The biggest lesson wasn't that I could now build a large language model.
I couldn't, and that was never the goal.
The lesson was that a technology that once seemed like a black box became understandable when I broke it into smaller pieces and implemented those pieces myself.
I also learned that knowing a formula and understanding an implementation are two very different things.
You can read about attention in five minutes. Understanding the tensor shapes, gradients, masking and numerical stability takes much more work.
And that work is worth it.
I started this project because I didn't want to remain someone who only knows how to use AI tools.
I wanted to understand what happens behind them.
Building a tiny GPT didn't answer every question I have about AI. In fact, it gave me many more questions to explore.
But it changed how I look at these models.
The next time I use an LLM, I won't see only a prompt and a response. I'll think about tokenisation, embeddings, attention, learned weights, probability distributions and all the computation happening underneath.
And that's what I wanted from this journey.
Not to build the biggest model. To understand the model.
Source code: github.com/Roshan190/tiny-shakespeare-gpt-js
Thanks for following Beyond the Prompt. This is the end of the first journey, not the end of my exploration of AI.
Project journey started: 30 July 2026.