Building an LLM from Scratch with Bob: A Didactic, 103,360-Parameter Walkthrough A developer built an LLM From Scratch Lab using Bob, a self-contained GPT-style decoder-only model of roughly 103,360 trainable parameters that implements tokenizers, causal multi-head self-attention, LayerNorm and autoregressive sampling by hand in plain PyTorch tensors and autograd rather than framework primitives. The project runs locally on laptop CPU or Apple Silicon MPS and is wrapped in an interactive Streamlit lab that lets learners inspect live tensors, tweak hyperparameters and watch training in real time, organized into ten progressive sub-tasks with animated visualizations driven by real computed values. A didactical walkthrough to LLM creation. Introduction Understanding how Large Language Models LLMs work under the hood is not a new topic, yet it remains one of the most rewarding journeys a student or software engineer can undertake. Whenever I hold mentoring discussions with students who are eager to look past API wrappers and black-box abstractions, I routinely point them toward a few foundational books that shaped my own understanding of transformer architectures, tokenization, and neural network training you will find my recommended reading list in the Links chapter at the end of this post . Reading theory is essential, but nothing cements an architecture in your head quite like building it piece by piece with zero high-level shortcuts. That is why I decided to use Bob to implement a clean, self-contained, and deeply didactic LLM From Scratch Lab . Rather than relying on monolithic framework primitives like nn.TransformerEncoder or nn.MultiheadAttention , every core mechanism—from character and Byte-Pair Encoding BPE tokenizers to causal multi-head self-attention, LayerNorm, and autoregressive sampling—is written by hand using plain PyTorch tensors and autograd. With a default configuration d model=64 , n heads=4 , n layers=2 , max seq len=64 , and weight tying enabled , the entire GPT-style decoder-only model weighs in at roughly 103,360 trainable parameters . It runs completely locally on a laptop CPU or Apple Silicon GPU MPS , wrapped in an interactive Streamlit lab that lets learners inspect live tensors, tweak hyperparameters, and watch the model learn in real time. Implementation The implementation is quite comprehensive, and rather than walking through every single boilerplate file in the repository, I want to highlight how the project is structured for clarity. The codebase separates the pure PyTorch engine llm/ and training/ from the interactive Streamlit presentation layer ui/ and pages/ , ensuring that the mathematical core remains completely framework-agnostic and easy to read. To make the learning curve approachable, the entire LLM lifecycle is broken down into ten progressive sub-tasks . Each sub-task pairs plain-language intuition and mathematical formulas with live visualizers and custom-rendered animated GIFs animgen/ . Every animation is driven directly by real numbers computed by the running code—using consistent color coding orange for queries, blue for keys and tokens, green for values, purple for positions, and red for loss so learners can visually trace matrix shapes and data flow at every stage. Below is a walkthrough of the core building blocks inside llm/ , accompanied by key excerpts from the hand-written Python modules. class CharTokenizer: """Maps every distinct character of a corpus to an integer id.""" kind = "char" def init self, chars: Iterable str = : Vocabulary order is deterministic: