Most "I built an LLM" posts show you a loss curve going down and stop there.
This one is different, because my from-scratch model lost, and I think the why is the most useful thing I can teach you.
Over the last few weeks I built two language models on the same task (empathetic, supportive dialogue) and compared them head to head:
Around both I built a serving layer (FastAPI, OpenAI-compatible endpoints, Docker, Prometheus) and a verification harness so every number in this post can be reproduced from the repo.
Repo: github.com/bharqav/SLM
By the end of this article you'll understand:
Let's start with the machine itself.
Strip away the hype and a language model does exactly one thing:
Given a sequence of tokens, produce a probability distribution over the next token.
Everything else (chat, code completion, "reasoning") is that one operation applied repeatedly. Here's the full pipeline.
Neural networks eat numbers, not strings. A tokenizer splits text into chunks called tokens and maps each to an integer ID.
My scratch model uses GPT-2's byte-pair encoding (BPE) via tiktoken, with a vocabulary of 50,257 tokens. Qwen uses its own tokenizer with a vocabulary of 151,936.
import tiktoken
enc = tiktoken.get_encoding("gpt2")
ids = enc.encode("I have been feeling really down lately")
print(ids) # a list of integers
print(enc.decode(ids)) # back to the original text
BPE starts from bytes and repeatedly merges the most frequent pairs, so common words become single tokens and rare words get split into pieces. That's why a model can handle any string, including made-up words and code.
Each token ID indexes a row in a giant lookup table called the embedding matrix. For my 757M model that's 50,257 x 1,536, about 77 million parameters, just to turn IDs into vectors.
But there's a problem. Attention (coming up next) doesn't know word order by itself. "Dog bites man" and "man bites dog" would look identical. So we also inject position information.
256 x 1,536, added to the token vectors.
The model is a stack of identical transformer blocks. Mine has 24 of them. Each block has two sub-layers wrapped in residual connections:
x = x + Attention( LayerNorm(x) )
x = x + MLP( LayerNorm(x) )
Two ideas hide in those two lines.
Residual connections (x = x + ...) mean each layer only has to learn a correction to what's already flowing through. Gradients also get a clean highway back to early layers, which is what makes 24+ layer networks trainable at all.
Pre-norm (normalizing before each sub-layer, not after) keeps activations well-scaled and training stable, especially as depth grows.
For every token, attention asks: which earlier tokens matter for predicting what comes next?
Each token's vector is projected into three vectors:
Attention scores are softmax(Q @ K^T / sqrt(d)), and the output is the score-weighted sum of the values. Here's the heart of my implementation, simplified:
class CausalSelfAttention(nn.Module):
def __init__(self, n_embd, n_head, dropout):
super().__init__()
self.n_head = n_head
self.c_attn = nn.Linear(n_embd, 3 * n_embd) # Q, K, V in ONE matmul
self.c_proj = nn.Linear(n_embd, n_embd) # output projection
self.dropout = dropout
def forward(self, x):
B, T, C = x.shape
q, k, v = self.c_attn(x).split(C, dim=2)
q, k, v = (t.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
for t in (q, k, v))
y = F.scaled_dot_product_attention(
q, k, v, is_causal=True,
dropout_p=self.dropout if self.training else 0.0,
)
y = y.transpose(1, 2).contiguous().view(B, T, C)
return self.c_proj(y)
Two details matter here.
Causal masking. During training, the model sees a whole sequence at once, so we must stop token t from peeking at token t+1, or it would just copy the answer. is_causal=True applies a lower-triangular mask so each position only attends to itself and the past.
Multiple heads. With 1,536 dimensions and 24 heads, each head works in a 64-dimensional subspace. Different heads can specialize: one tracks bracket matching, another subject-verb agreement, another something no human has labeled. We run them in parallel and concatenate.
Attention mixes information between tokens. The MLP then processes each token's vector independently: expand to 4x the width (1,536 → 6,144), apply a nonlinearity (GELU in my model), project back down. A large share of a transformer's parameters, and much of its stored "knowledge", live in these MLPs.
After the final block and a last LayerNorm, each position's vector is multiplied by the embedding matrix again, this time in reverse, to produce one score (a logit) per vocabulary token. Softmax turns logits into probabilities.
Sharing the same matrix for input embedding and output projection is called weight tying. It saves ~77M parameters and acts as a regularizer, because the embedding space has to be useful for both reading and predicting.
Training data is a long stream of tokens. For every position, the target is simply the next token. The loss is cross-entropy between the predicted distribution and that target:
logits = model(x) # (B, T, vocab)
loss = F.cross_entropy(
logits.view(-1, logits.size(-1)),
targets.view(-1),
ignore_index=-100,
)
loss.backward()
optimizer.step()
That's it. No labels beyond the text itself. This is why LLM pretraining scales: the internet is the dataset.
A handy way to read the loss: perplexity = exp(loss). My best 24-layer empathy checkpoint hit a validation loss of 3.9259, which is a perplexity of about 50.7. Loosely: the model is as uncertain as if it were choosing between ~51 equally likely tokens at each step.
To generate text, you feed in a prompt, get a distribution over the next token, sample one, append it, and repeat. How you sample matters enormously:
def sample_next(logits, generated_ids, temperature=0.7, top_k=20, rep_penalty=1.2):
for t in set(generated_ids):
logits[t] = logits[t] / rep_penalty if logits[t] > 0 else logits[t] * rep_penalty
logits = logits / temperature # <1 = sharper, >1 = flatter
topk_vals, topk_idx = torch.topk(logits, top_k) # keep only the k best
probs = F.softmax(topk_vals, dim=-1)
return topk_idx[torch.multinomial(probs, 1)]
Notice something wasteful: when generating token 101, the keys and values for tokens 1-100 haven't changed. A KV cache stores them so each new token costs one token's worth of compute instead of re-processing the whole prefix.
Qwen (via Hugging Face) uses a KV cache. My from-scratch model does not, and as you'll see in the benchmarks, that single omission matters a lot.
I deliberately followed GPT-2's design so every piece would be something I could fully explain. Two sizes:
Shared settings: 256-token context, 4x MLP expansion, GELU, pre-norm LayerNorm with bias, dropout 0.1, tied embeddings, learned positions, and gradient checkpointing, which trades extra compute for roughly 40% less activation memory.
One initialization detail worth knowing, straight from the GPT-2 paper. The output projections inside each block (c_proj) are initialized with a smaller standard deviation, 0.02 / sqrt(2 * n_layer). Every block adds to the residual stream, so without this scaling the stream's variance grows with depth and early training gets chaotic.
You can sanity-check the parameter count with back-of-envelope math: each block has roughly 12 x d^2 parameters (4d² for attention's Q/K/V/output, 8d² for the MLP). For d = 1,536 that's ~28.3M per block, x 24 blocks = ~680M. Add the 77M embedding table and you land right at ~757M.
My first pretraining ran on CPU only, float32, 8 threads, at about 215 tokens per second on a ~2.5M-token code-and-prose corpus. Two runs totaling roughly 2h44m got the 253M model to a best validation loss of 4.2803.
That number is not good. It's the kind of model that produces fluent-looking nonsense. But it was a complete, working end-to-end loop, and everything after it built on that foundation.
Next I trained the 24-layer, 757M configuration on empathetic dialogue (EmpatheticDialogues + ESConv, about 2.1M train tokens) on an RTX 5060 with bfloat16 mixed precision. Effective batch: 32 sequences x 256 tokens = 8,192 tokens per step.
Validation loss over time:
Good progress. But when I actually read the outputs, something was badly wrong.
The model's replies were non-sequiturs. It would answer "I feel like nobody understands me" with something fluent and completely unrelated. The loss curve looked healthy. The behavior was broken.
Digging into the corpus, I found two structural defects:
Lesson: a falling validation loss only tells you the model is getting better at the distribution you gave it. It says nothing about whether that distribution was the right one.
The fix was a new preprocessing script that rebuilds the corpus with an explicit format:
User: <text>
Assistant: <text>
with consecutive same-role turns merged, Alpaca instruction/response pairs added in the same format (so the model can answer general questions, not just offer support), over-long answers dropped, and a 1M-token slice of code appended raw. The model tells the two modes apart by prompt shape: a bare code prefix continues as code, a User:/Assistant: prefix continues as dialogue. Dialogue and Alpaca splits are disjoint, so validation is leak-free.
The clean corpus came to ~5.93M tokens: 22,364 dialogues, 44,481 QA pairs, and the 1M code tokens.
I also used --init-from to load only the model weights and reset the optimizer. When your data distribution changes, resuming old optimizer state means carrying momentum tuned for a different dataset.
At inference there's a matching trick. The model happily keeps writing after it finishes its reply, inventing the next User: turn. So I cut the generated text at the first User: or Assistant: marker after the answer, and train with an EOS token at the end of each sequence so sampling can stop cleanly.
Even with clean structure, the small model had to invent its own style from very little data. So I tried distillation: run Qwen2.5-0.5B-Instruct over 20,000 Alpaca instructions (temperature 0.3), then mix those answers into the training corpus. Now the 253M model learns to imitate a stronger teacher's answers.
Validation loss across these runs on the clean corpus fell from 4.0473 to 3.2289 (the CLI/API default checkpoint, at step 8,000) and to 2.9197 for the later v3 checkpoint.
An important caveat: those numbers are not comparable with the 3.9259 from the empathy-only run. Different corpus, different validation set, different task mix. Validation loss is only comparable between models evaluated on the same data with the same tokenizer. I'll come back to this, because it nearly misled me.
At this point I had a working scratch pipeline and a decent understanding of the whole stack. So why bring in Qwen?
Because the honest question was: what would it take to build the best possible empathetic assistant under my constraints?
Here's the gap I was up against.
Scale of pretraining. My scratch model saw millions of tokens. Qwen2.5 was pretrained on trillions. That's several orders of magnitude more data, and that's where grammar, world knowledge, code ability, and instruction-following come from. Fine-tuning on 23K dialogues can steer a model with all that capability. It can't conjure it.
Architecture upgrades. Qwen isn't just "bigger GPT-2". Its design (0.5B parameters, 24 layers, hidden size 896) includes:
Fit for the size bracket. At 494M parameters, Qwen2.5-0.5B-Instruct sits between my 253M and 757M scratch models. Same order of magnitude, so the comparison is meaningful, even if it isn't size-parity.
And importantly, I framed this correctly in the repo: it is not a claim that one model is a distilled or compressed version of the other. It's a side-by-side of two real training pipelines on the same task.
Full fine-tuning updates every weight, which needs optimizer state (several times the model size) in GPU memory. QLoRA makes it fit on a consumer card with two ideas.
1. LoRA (Low-Rank Adaptation). Freeze the original weight matrix W and learn a small update expressed as the product of two skinny matrices:
W' = W + (alpha / r) * B @ A
where A is r x d_in and B is d_out x r, with a tiny rank r. Instead of training a 896 x 896 matrix (802,816 values), a rank-8 adapter trains 8 x 896 + 896 x 8 = 14,336 values.
2. Quantization. Store the frozen base weights in 4-bit NF4 format with double quantization, and compute in bf16. The base model shrinks to a fraction of its memory footprint, while the small adapter trains in higher precision.
Let me show the arithmetic on my adapter, because it's a satisfying check. Qwen's q_proj maps 896 → 896 and v_proj maps 896 → 128 (2 KV heads x 64 dims). At rank 8, per layer:
That matches the 540,672 scalars in my saved adapter exactly, about 0.11% of the 494M base model, and the whole adapter file is ~2.1 MB.
A second, much smaller pass used 770 rows of hand-written steering dialogues mixed with replay data from stage 1: 96 steps, 175 seconds, learning rate 5e-5.
Here's where I have to be honest, because the repo is.
Stage 2 did not measurably improve validation loss. Stage 1 ended at 2.7650; stage 2 at 2.7655. And the stage-2 validation set was the same 461-row file used in stage 1, so it isn't an untouched hold-out anyway. Validation loss was never going to capture what steering is for (policy and behavior), but I also have no automated behavioral gate in the repo that proves it worked. My stored behavioral probe records predate the latest retrain and no script reproduces them. I've listed that as a known gap rather than pretend otherwise.
I also hit a subtle config surprise: the stage-2 run was launched with --init-adapter, and my training script ignores --lora-r and friends in that path, the stage-1 config instead. So the trained stage-2 adapter is rank-8 over q_proj/ v_proj, even though its log records the CLI values (r=16, alpha=32). Always verify what your adapter config actually is, not what you typed on the command line.
For deployment I merge the adapter back into the base weights and save a single fp16 checkpoint (988,097,536 bytes). The merged model loads as a plain AutoModelForCausalLM: no PEFT wrapper, no adapter step at load time, and zero extra inference cost, since the architecture is identical to the base.
model = AutoModelForCausalLM.from_pretrained(
"slm-ft/merged_fp16/", torch_dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("slm-ft/merged_fp16/")
All benchmarks: CPU, float32, 8 threads, identical prompts, max_new_tokens=40, temperature 0.7, top-k 20, top-p 0.9, repetition penalty 1.2, seed 1337. Reproducible from benchmarks/generation_matrix.json.
Generation throughput:
Memory after load (RSS):
Reading these correctly:
Speed is where my scratch models can compete. Quality is where they can't.
def preprocess(text): produced "You have an appointment?" and class ChatMessage: produced nothing at all. That's the predictable result of training only on empathy data.
Bottom line: neither from-scratch model matches Qwen's output quality on these prompts. The project's quality-oriented path is the QLoRA fine-tune. The scratch models are the educational path, and the 16L one is a perfectly reasonable fast model for short supportive replies.
A 757M model losing to a 494M one sounds backwards until you separate the three things that determine quality: parameters, data, and compute. My 757M model had more parameters but trained on roughly a million times less data than Qwen saw, for a few thousand steps on a single consumer GPU. Parameters are capacity; data and compute are what fill that capacity. A large empty model is just a large empty model.
That's the real reason fine-tuning wins at small scale: you rent someone else's trillions of training tokens and spend your own tiny budget only on the last mile.
A model nobody can call is a notebook. So I built the serving layer properly.
FastAPI server (slm_api.py) with:
/generate plus /v1/completions and /v1/chat/completions, with SSE streaming variants, so existing OpenAI clients can point at it/health, /model/info, and a Prometheus /metrics endpoint tracking request counts, latency histograms, tokens generated, and model-loaded state
Deployment: a slim Docker image (CPU-only PyTorch wheels, because the default Linux wheel bundles several GB of CUDA runtime I wasn't going to use), non-root user, health checks, and a production docker-compose stack with Prometheus and resource limits.
Verification harness: one command, python harness.py, runs the test suite, a setup-artifact check, the live comparison, and a dependency audit, then exits non-zero if any step fails. There's also seeded generation (seed=1337), pinned dependency constraints, pre-commit hooks, and a held-out perplexity evaluation script.
I added the harness after a mistake, which brings me to the part I think matters most.
I kept a "provenance note" every time a number in my own docs turned out to be wrong. There were several:
None of these were catastrophic, because each got caught by the same habit: every number in the docs must trace to an artifact in the repo, and every claim must say what it does and doesn't prove. If you take one engineering lesson from this post, take that one. It's more valuable than any architecture trick.
Everything (the model code, training scripts, QLoRA pipeline, benchmarks, API server, Docker setup, and harness) is open source under the MIT license:
License note: the training data includes EmpatheticDialogues (CC BY-NC-4.0) and ESConv (CC BY-NC-SA), so models trained on it carry non-commercial constraints. Check the repo's License section before using any weights commercially.
If you build your own, I'd love to hear what broke first. In my experience it's never the architecture; it's the data.
Questions or corrections? Drop a comment, or open an issue on the repo. If a number in this post doesn't match the artifacts, the artifacts win.