# I Built a Tool-Calling LLM From Absolute Zero — Here’s Everything I Learned > Source: > Published: 2026-09-18 20:31:01+00:00 There’s a moment, about two weeks into training your first language model from scratch, when the loss curve finally bends downward and the model — this pile of random numbers you initialized yourself — outputs a grammatically correct sentence for the first time. It’s a strange kind of magic. Not because it’s mysterious — you wrote every line of code that produced it — but because it isn’t. You know *exactly* why it works. You watched it learn. That feeling is the entire reason I built [**Pebble**](https://github.com/viditraj/pebble): an open-source, 25-million-parameter language model that you build entirely from scratch, and that — by the end — can call tools like a calculator or a weather API, the same way GPT-4 or Claude do. This article is the guide I wish existed when I started. No hand-waving, no “attention is basically like a search engine” analogies that fall apart the moment you ask a follow-up question. Just the actual mechanics of how a modern LLM is built, trained, and made useful — with enough detail that you could implement it yourself this weekend. Let’s address the obvious objection first: you are never going to out-train GPT-5 in your bedroom. That’s not the point. Here’s the point. There is a massive, unhelpful gap in how LLMs are taught online: Nothing sits in the middle. Nothing says: *here is a complete, working, modern LLM — small enough to read in an afternoon, real enough that training it teaches you the actual mechanics.* That gap is where Pebble lives. And once you’ve built one small model end-to-end — tokenizer, transformer, training loop, alignment, deployment — every paper you read afterward, every “how does ChatGPT actually work” thread on X, every offhand comment about “KV-cache memory pressure” in a job interview, suddenly clicks into a system you already understand, not a black box you’re told to trust. By the end of this build, you will have personally implemented: Let’s go through each piece — what it is, why it exists, and how it fits together. Neural networks don’t understand text. They understand numbers. The first thing any LLM needs is a way to convert "Hello, world!" into a list of integers, and back again. The naive approaches both fail in instructive ways: The solution the entire industry converged on is **Byte-Pair Encoding (BPE)**: start with individual bytes, and iteratively merge the most frequent adjacent pairs into new tokens. Common words like “the” collapse into a single token. Rare or novel words get split into meaningful subword chunks — "unhappiness" might become ["un", "happiness"], and the model learns from thousands of examples that "un-" means negation, generalizing that pattern to words it's never seen. ``` tokenizer = BPETokenizer.train(corpus, vocab_size=16384)tokens = tokenizer.encode("Hello, world!") # [1762, 44, 1452, 33]text = tokenizer.decode(tokens) # "Hello, world!" ``` Building this from raw bytes — not importing tiktoken — is where you actually internalize *why* LLMs are bad at counting letters in a word (the model never sees letters, it sees merged subword chunks) and why vocabulary size is a real design decision, not a hyperparameter you copy from a config file. This is the part most tutorials get stuck teaching as if it were still 2019. Here’s what’s actually inside a current-generation open model — and what Pebble implements piece by piece. Four components separate a modern architecture from GPT-2’s original design, and each one solves a specific, concrete problem: **Rotary Position Embeddings (RoPE).** Transformers process all tokens in parallel, which means they have no inherent sense of word order — “dog bites man” and “man bites dog” would look identical without positional information. Older models added a separate position vector to each token embedding. RoPE instead *rotates* the query and key vectors by an angle proportional to their position, which has an elegant side effect: the dot product between two rotated vectors naturally encodes their *relative* distance, not just their absolute position. That relative-distance property is a big part of why modern models generalize better to longer sequences than they were trained on. **Grouped Query Attention (GQA).** Standard multi-head attention gives every attention head its own Key and Value projections. With 8 heads, that’s 8 separate K/V matrices to cache during generation — and that cache is often the single biggest memory cost at inference time. GQA has multiple query heads *share* a smaller set of K/V heads — Pebble uses 8 query heads but only 4 KV heads — cutting KV-cache memory roughly in half with almost no quality loss. It’s the exact mechanism that lets LLaMA-2–70B run on 2 GPUs instead of 4. **SwiGLU.** The feed-forward layer between attention blocks used to use ReLU or GELU. SwiGLU replaces this with a *gated* activation — one linear projection controls how much of another linear projection passes through — which consistently improves downstream performance at the same parameter count. It costs a bit more compute per layer; every major open model released since 2023 has decided that trade is worth it. **RMSNorm.** LayerNorm normalizes activations using both their mean and variance. RMSNorm found that the mean-centering step is mostly unnecessary — normalizing by root-mean-square alone gets you the same training stability for meaningfully less compute. It’s a small change that adds up across dozens of layers and millions of training steps. Put together, this is the Pebble-25M spec: Small enough to train in a weekend. Architecturally identical, layer for layer, to the model that’s answering your prompts on a much larger scale right now. Here’s the entire idea behind pretraining, stripped of mysticism: **predict the next token, measure how wrong you were, adjust the weights very slightly, and repeat fifty thousand times.** That’s genuinely it. The complexity isn’t in the concept — it’s in making that loop fast and stable enough to actually converge on the hardware you have. ``` training: batch_size: 32 gradient_accumulation_steps: 4 # effective batch = 128 max_lr: 3e-4 min_lr: 3e-5 warmup_steps: 1000 total_steps: 50000 weight_decay: 0.1 grad_clip: 1.0 precision: bf16 ``` A few things you learn only by actually running this loop on constrained hardware: And here’s the detail that surprises almost everyone the first time they profile their own training run: the model weights are *not* what fills your GPU memory. The activations — the intermediate values computed at every layer, for every token, for every item in your batch — dwarf everything else combined. This is precisely why gradient checkpointing and batch size tuning matter more than parameter count when you’re working within an 8GB budget. It’s a detail you’ll never internalize from a diagram; you only learn it when your own training run OOMs and you have to figure out why. A freshly pretrained model is a very good autocomplete engine. It is not an assistant. Ask it a question and it’s just as likely to continue the sentence with another question as it is to answer yours — because all it learned to do was predict plausible next tokens from raw internet text. Two more training stages turn “very good autocomplete” into “actually useful”: **Supervised Fine-Tuning (SFT)** trains the model on structured conversations — a system prompt, a user question, an assistant response — while *masking the loss* so the model only learns from its own turns, not from copying the user’s input. This is what teaches the model to follow instructions and adopt a chat format at all. ``` { "messages": [ {"role": "system", "content": "You are Pebble. Use tools when helpful."}, {"role": "user", "content": "What's the weather in Tokyo?"}, {"role": "assistant", "tool_calls": [{"name": "weather", "args": {"city": "Tokyo"}}]}, {"role": "tool", "content": "72F, partly cloudy"}, {"role": "assistant", "content": "It's 72F and partly cloudy in Tokyo right now."} ]} ``` **Direct Preference Optimization (DPO)** goes one step further: instead of just training on “correct” examples, it trains on *pairs* — a preferred response and a rejected one — and nudges the model’s probability mass toward the preferred one. This is what teaches Pebble to prefer calling the calculator over confidently hallucinating an answer to 47 * 89. The loss function is almost suspiciously simple for what it accomplishes: ``` loss = -log_sigmoid(beta * (log_ratio_chosen - log_ratio_rejected)) ``` No separate reward model, no reinforcement learning loop — just a clever reformulation that turns preference learning into something closer to a classification problem. This is the part that makes Pebble more than a toy, and it’s simpler than it looks from the outside. During SFT, the model learns to emit special tokens — <|tool_call|> and <|end|> — when a question calls for a function rather than a free-text answer. The inference engine watches for these tokens as they're generated, and when it sees one, it pauses generation, parses out the function name and arguments, actually executes the function, formats the result, and feeds it back into the model's context so generation can resume with real information instead of a guess. ``` You: What's 47 * 89?Pebble: <|tool_call|> calculator(expression="47 * 89") <|end|> <|result|> 4183 <|end|> 47 multiplied by 89 is 4,183. ``` That’s the entire trick behind tool use in every major model you’ve used — GPT-4, Claude, Gemini. Structured output tokens, a parser watching for them, and a function call slotted into the conversation. There’s no separate “reasoning module” bolted on. It’s the same next-token-prediction machinery, just fine-tuned to sometimes predict a function call instead of a sentence. A trained model that only runs on the GPU it was trained on isn’t very useful. The final stage shrinks Pebble from roughly 50MB to about 12MB through INT4 post-training quantization — converting weights from 16-bit floats to 4-bit integers with a calibrated scale — then exports to GGUF format so it runs anywhere llama.cpp does, laptop CPU included. ``` python scripts/export.py --checkpoint checkpoints/sft/latest.pt --format gguf --bits 4./llama-cli -m pebble-25m-q4.gguf -p "Hello, Pebble!" ``` The quality loss at this parameter count and bit-width is small enough that you genuinely won’t notice it in casual use — which is itself a useful, hands-on lesson in why quantization is such a standard part of the deployment pipeline for every model you interact with day to day. Twenty-five million parameters will not write your essays or pass a bar exam, and that was never the goal. What training one from scratch gives you is something most people who use LLMs every day never get: an internal, mechanical model of what’s actually happening when you send a prompt. Once you’ve built this yourself: That’s the trade Pebble is built around: give up scale, keep everything else. Every layer is real. Every line is yours. And the gap between “using an LLM” and “understanding one” — the gap that no amount of prompting ChatGPT about transformers will ever fully close — gets a lot smaller. The full code, curriculum, and configs are open-source on GitHub: **github.com/viditraj/pebble** Clone it, train the tokenizer, and by tomorrow you’ll have watched a pile of random weights learn to speak. That’s a better way to understand a transformer than any explainer video, including this one. ``` git clone https://github.com/viditraj/pebble.gitcd pebblepip install -r requirements.txtpython scripts/train_tokenizer.py --vocab-size 16384 --data data/raw/ ``` If this helped you understand LLMs a little better, a star on the repo goes a long way — and if you build something interesting on top of Pebble, I’d genuinely love to see it. *Pebble is MIT-licensed and built on ideas from Karpathy’s nanoGPT and Zero to Hero series, Sebastian Raschka’s “Build a Large Language Model (From Scratch),” and the LLaMA, RoFormer, and DPO papers. Full citations are in the repo.* [I Built a Tool-Calling LLM From Absolute Zero — Here’s Everything I Learned](https://pub.towardsai.net/i-built-a-tool-calling-llm-from-absolute-zero-heres-everything-i-learned-bad8ece610d3) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.