cd /news/large-language-models/building-an-llm-from-scratch-with-bo… · home › topics › large-language-models › article
[ARTICLE · art-174128] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Building an LLM from Scratch with Bob: A Didactic, 103,360-Parameter Walkthrough

A developer built an LLM From Scratch Lab using Bob, a self-contained GPT-style decoder-only model of roughly 103,360 trainable parameters that implements tokenizers, causal multi-head self-attention, LayerNorm and autoregressive sampling by hand in plain PyTorch tensors and autograd rather than framework primitives. The project runs locally on laptop CPU or Apple Silicon MPS and is wrapped in an interactive Streamlit lab that lets learners inspect live tensors, tweak hyperparameters and watch training in real time, organized into ten progressive sub-tasks with animated visualizations driven by real computed values.

key takeaway

A hand-written GPT-style decoder with d_model=64, 4 heads and 2 layers totals about 103,360 parameters and trains on a laptop CPU or Apple Silicon MPS, making every attention and tokenization step inspectable.

by read8 min views3 publishedNov 22, 2026
Building an LLM from Scratch with Bob: A Didactic, 103,360-Parameter Walkthrough
Image: Dev.to

A didactical walkthrough to LLM creation.

#

Introduction

Understanding how Large Language Models (LLMs) work under the hood is not a new topic, yet it remains one of the most rewarding journeys a student or software engineer can undertake. Whenever I hold mentoring discussions with students who are eager to look past API wrappers and black-box abstractions, I routinely point them toward a few foundational books that shaped my own understanding of transformer architectures, tokenization, and neural network training (you will find my recommended reading list in the Links chapter at the end of this post).

Reading theory is essential, but nothing cements an architecture in your head quite like building it piece by piece with zero high-level shortcuts. That is why I decided to use Bob to implement a clean, self-contained, and deeply didactic LLM From Scratch Lab. Rather than relying on monolithic framework primitives like nn.TransformerEncoder or nn.MultiheadAttention, every core mechanism—from character and Byte-Pair Encoding (BPE) tokenizers to causal multi-head self-attention, LayerNorm, and autoregressive sampling—is written by hand using plain PyTorch tensors and autograd.

With a default configuration (d_model=64, n_heads=4, n_layers=2, max_seq_len=64, and weight tying enabled), the entire GPT-style decoder-only model weighs in at roughly 103,360 trainable parameters. It runs completely locally on a laptop CPU or Apple Silicon GPU (MPS), wrapped in an interactive Streamlit lab that lets learners inspect live tensors, tweak hyperparameters, and watch the model learn in real time.

#

Implementation

The implementation is quite comprehensive, and rather than walking through every single boilerplate file in the repository, I want to highlight how the project is structured for clarity. The codebase separates the pure PyTorch engine (llm/ and training/) from the interactive Streamlit presentation layer ( ui/ and pages/), ensuring that the mathematical core remains completely framework-agnostic and easy to read.

To make the learning curve approachable, the entire LLM lifecycle is broken down into ten progressive sub-tasks.

Each sub-task pairs plain-language intuition and mathematical formulas with live visualizers and custom-rendered animated GIFs (animgen/).

Every animation is driven directly by real numbers computed by the running code—using consistent color coding (orange for queries, blue for keys and tokens, green for values, purple for positions, and red for loss) so learners can visually trace matrix shapes and data flow at every stage.

Below is a walkthrough of the core building blocks inside llm/, accompanied by key excerpts from the hand-written Python modules.

class CharTokenizer:
    """Maps every distinct character of a corpus to an integer id."""

    kind = "char"

    def __init__(self, chars: Iterable[str] = ()):
        self.itos: list[str] = [UNK] + sorted(set(chars) - {UNK})
        self.stoi: dict[str, int] = {t: i for i, t in enumerate(self.itos)}

    def encode(self, text: str) -> list[int]:
        """Text -> list of ids. Unknown characters map to ``UNK_ID``."""
        return [self.stoi.get(c, UNK_ID) for c in text]

#

Tokenization: From Raw Text to Integer IDs (tokenizer.py)

Language models only operate on numbers, so the first sub-task maps raw text into a sequence of integer IDs drawn from a fixed vocabulary. The project implements two tokenizers sharing a unified encode / decode interface: a character-level tokenizer (CharTokenizer) and a Byte-Pair Encoding tokenizer ( BPETokenizer) that iteratively merges the most frequent adjacent symbol pairs.

#

Token Embeddings: Learnable Lookup Tables (embeddings.py)

Once text is converted into integer IDs, each ID must be mapped to a dense vector of size d_model. Instead of hiding behind nn.Embedding, TokenEmbedding explicitly allocates an nn.Parameter matrix of shape (vocab_size, d_model) and uses standard tensor indexing so learners can see that an embedding lookup is simply selecting a row.

class TokenEmbedding(nn.Module):
    """Learnable lookup table ``(vocab_size, d_model)``."""

    def __init__(
        self,
        vocab_size: int,
        d_model: int,
        init: str = "normal",
        std: float = 0.02,
        scale: bool = False,
    ) -> None:
        super().__init__()
        self.vocab_size = vocab_size
        self.d_model = d_model
        self.scale = scale
        self.weight = nn.Parameter(torch.empty(vocab_size, d_model))
        init_embedding_(self.weight, init, std)

    def forward(self, ids: torch.Tensor) -> torch.Tensor:
        """``(...,) int ids -> (..., d_model)`` by indexing rows of the table."""
        if ids.numel() and (ids.min() < 0 or ids.max() >= self.vocab_size):
            raise IndexError(f"token id out of range [0, {self.vocab_size})")
        out = self.weight[ids]  # advanced indexing = the whole "layer"
        return out * math.sqrt(self.d_model) if self.scale else out

#

Positional Encoding: Breaking Permutation Symmetry (positional.py)

Self-attention is inherently permutation-invariant—without position signals, "dog bites man" and "man bites dog" look identical to the attention mechanism. To inject sequence order, the lab supports both GPT-2 style learned positional tables and the classic fixed sinusoidal waves from Vaswani et al. (2017):

def sinusoidal_table(max_len: int, d_model: int, base: float = 10000.0) -> torch.Tensor:
    """Return the ``(max_len, d_model)`` sinusoidal matrix."""
    position = torch.arange(max_len, dtype=torch.float32).unsqueeze(1)  # (L, 1)
    even = torch.arange(0, d_model, 2, dtype=torch.float32)  # (ceil(d/2),)
    div_term = torch.exp(-math.log(base) * even / d_model)  # (ceil(d/2),)
    angles = position * div_term  # (L, ceil(d/2))
    table = torch.zeros(max_len, d_model)
    table[:, 0::2] = torch.sin(angles)
    table[:, 1::2] = torch.cos(angles[:, : d_model // 2])
    return table

#

Causal Multi-Head Self-Attention (attention.py)

This is the heart of the transformer. Every token projects into a Query (what it is looking for), a Key (what it advertises), and a Value (the payload it carries). Keeping q_proj, k_proj, and v_proj as three separate linear layers keeps their roles explicit, while a lower-triangular causal mask prevents tokens from peeking into the future during next-token prediction:

def scaled_dot_product_attention(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    mask: Optional[torch.Tensor] = None,
    dropout: Optional[nn.Module] = None,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Return ``(output, weights)``."""
    scores = attention_scores(q, k, mask)
    weights = torch.softmax(scores, dim=-1)  # normalise each query's row
    weights = torch.nan_to_num(weights, nan=0.0)  # fully-masked rows -> zeros
    used = dropout(weights) if dropout is not None else weights
    return used @ v, weights

#

Normalization & Position-Wise Feed-Forward Networks (layers.py)

While attention mixes information across tokens, the position-wise Feed-Forward Network (FFN) processes each token independently, expanding its representation to 4 * d_model, applying a smooth GELU non-linearity, and projecting back. Wrapping sub-layers with hand-written LayerNorm stabilizes activation scales across depth:

class LayerNorm(nn.Module):
    """Layer normalisation over the last dimension, written by hand."""

    def __init__(self, d_model: int, eps: float = 1e-5) -> None:
        super().__init__()
        self.d_model = d_model
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(d_model))  # gamma
        self.bias = nn.Parameter(torch.zeros(d_model))  # beta

    def stages(self, x: torch.Tensor) -> dict[str, torch.Tensor]:
        """Return every intermediate quantity (for the educational breakdown)."""
        mean = x.mean(dim=-1, keepdim=True)
        var = ((x - mean) ** 2).mean(dim=-1, keepdim=True)  # biased, like nn.LayerNorm
        normalized = (x - mean) / torch.sqrt(var + self.eps)
        output = normalized * self.weight + self.bias
        return {"mean": mean, "var": var, "normalized": normalized, "output": output}

#

Assembling the Decoder-Only GPT & Autoregressive Generation (transformer.py)

Finally, GPT stacks n_layers pre-norm TransformerBlock modules, applies a final LayerNorm, and projects back to vocabulary logits (tying the output head weights to the token embedding matrix to save parameters). Notice how return_details=True exposes every intermediate tensor—queries, attention weights, hidden states, and logits—so the UI and animation engine can inspect the network's internal reasoning:

class TransformerBlock(nn.Module):
    """One pre-norm decoder block: attention sub-layer then feed-forward sub-layer."""

    def __init__(self, cfg: GPTConfig) -> None:
        super().__init__()
        self.ln1 = LayerNorm(cfg.d_model)
        self.attn = MultiHeadAttention(cfg.d_model, cfg.n_heads, cfg.dropout, causal=True)
        self.ln2 = LayerNorm(cfg.d_model)
        self.ffn = FeedForward(cfg.d_model, cfg.ffn_mult * cfg.d_model, cfg.dropout, cfg.activation)

    def forward(self, x: torch.Tensor, return_details: bool = False):
        if not return_details:
            x = x + self.attn(self.ln1(x))
            return x + self.ffn(self.ln2(x))

During inference, GPT.generate() runs an autoregressive loop—cropping context to max_seq_len, computing next-token logits, shaping the probability distribution via temperature, top-kk, or nucleus (top-pp) sampling, and appending the sampled token ID:

        try:
            for _ in range(max_new_tokens):
                context = ids[:, -self.cfg.max_seq_len :]
                logits = self(context)[:, -1, :]
                probs = next_token_distribution(logits, temperature, top_k, top_p)
                nxt = torch.multinomial(probs.float().cpu(), 1, generator=generator).to(ids.device)
                ids = torch.cat([ids, nxt], dim=1)
        finally:
            self.train(was_training)

#

Conclusion

By pairing clean, dependency-isolated architecture with hands-on visual tooling, Bob turns the abstract mathematics of a GPT-style language model into an inspectable, local sandbox. Starting from a simple uv virtual environment and zero external cloud dependencies, students and developers can explore every layer of the stack:

  • Transparent Core Engine (llm/ & training/) : Every critical operation—character and BPE tokenization, embedding table indexing, sinusoidal and learned positional encodings, causal multi-head attention, pre-norm residual blocks, hand-written cross-entropy loss, and optimizers (SGD, Adam, AdamW)—is implemented explicitly in plain PyTorch and verified against numerical reference tests.
  • Interactive Learning & Live Training (ui/ & pages/) : Across nine interactive Streamlit lesson pages, learners can switch between plain-language explanations, mathematical formulas, and real source code, train their ~103,360-parameter model on custom.txt or.md corpora, monitor live loss and gradient-norm curves, and experiment with temperature, top-kk, and top-pp sampling.
  • Automated Documentation & Visual Storytelling (docsgen/ & animgen/) : Every one of the ten sub-tasks is backed by static HTML/Markdown guides and looping Pillow-rendered animations generated directly from live model executions, ensuring that every diagram, matrix dimension, and metric reflects real code rather than static mockups.

Whether you are mentoring students taking their first steps into generative AI or refreshing your own intuition on transformer internals, building a compact model from scratch remains the clearest way to demystify how modern LLMs actually think.

Thanks for reading ✅

#

Links and References

Disclaimer: the book links this section point to manning.com. I have no financial, sponsorship, affiliate, or other formal affiliation with Manning Publications or manning.com—these links are shared purely as independent recommendations for readers and students.

── more in #large-language-models 4 stories · sorted by recency
── more on @bob 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/building-an-llm-from…] indexed:0 read:8min 2026-11-22 · —