I Trained a 757M-Parameter LLM From Scratch, Then Fine-Tuned Qwen Instead. Here's Everything I Learned. A developer built a 757M-parameter GPT-style language model from scratch and fine-tuned Qwen on the same empathetic dialogue task, reporting that the from-scratch model lost the head-to-head comparison. The work includes a FastAPI/OpenAI-compatible serving layer with Docker and Prometheus, plus a verification harness so results can be reproduced from the public repo. Most "I built an LLM" posts show you a loss curve going down and stop there. This one is different, because my from-scratch model lost , and I think the why is the most useful thing I can teach you. Over the last few weeks I built two language models on the same task empathetic, supportive dialogue and compared them head to head: Around both I built a serving layer FastAPI, OpenAI-compatible endpoints, Docker, Prometheus and a verification harness so every number in this post can be reproduced from the repo. Repo: github.com/bharqav/SLM https://github.com/bharqav/SLM By the end of this article you'll understand: Let's start with the machine itself. Strip away the hype and a language model does exactly one thing: Given a sequence of tokens, produce a probability distribution over the next token. Everything else chat, code completion, "reasoning" is that one operation applied repeatedly. Here's the full pipeline. Neural networks eat numbers, not strings. A tokenizer splits text into chunks called tokens and maps each to an integer ID. My scratch model uses GPT-2's byte-pair encoding BPE via tiktoken , with a vocabulary of 50,257 tokens. Qwen uses its own tokenizer with a vocabulary of 151,936. python import tiktoken enc = tiktoken.get encoding "gpt2" ids = enc.encode "I have been feeling really down lately" print ids a list of integers print enc.decode ids back to the original text BPE starts from bytes and repeatedly merges the most frequent pairs, so common words become single tokens and rare words get split into pieces. That's why a model can handle any string, including made-up words and code. Each token ID indexes a row in a giant lookup table called the embedding matrix . For my 757M model that's 50,257 x 1,536 , about 77 million parameters, just to turn IDs into vectors. But there's a problem. Attention coming up next doesn't know word order by itself. "Dog bites man" and "man bites dog" would look identical. So we also inject position information . 256 x 1,536 , added to the token vectors. The model is a stack of identical transformer blocks . Mine has 24 of them. Each block has two sub-layers wrapped in residual connections: x = x + Attention LayerNorm x x = x + MLP LayerNorm x Two ideas hide in those two lines. Residual connections x = x + ... mean each layer only has to learn a correction to what's already flowing through. Gradients also get a clean highway back to early layers, which is what makes 24+ layer networks trainable at all. Pre-norm normalizing before each sub-layer, not after keeps activations well-scaled and training stable, especially as depth grows. For every token, attention asks: which earlier tokens matter for predicting what comes next? Each token's vector is projected into three vectors: Attention scores are softmax Q @ K^T / sqrt d , and the output is the score-weighted sum of the values. Here's the heart of my implementation, simplified: python class CausalSelfAttention nn.Module : def init self, n embd, n head, dropout : super . init self.n head = n head self.c attn = nn.Linear n embd, 3 n embd Q, K, V in ONE matmul self.c proj = nn.Linear n embd, n embd output projection self.dropout = dropout def forward self, x : B, T, C = x.shape q, k, v = self.c attn x .split C, dim=2 reshape to batch, heads, time, head dim q, k, v = t.view B, T, self.n head, C // self.n head .transpose 1, 2 for t in q, k, v is causal=True applies the lower-triangular mask y = F.scaled dot product attention q, k, v, is causal=True, dropout p=self.dropout if self.training else 0.0, y = y.transpose 1, 2 .contiguous .view B, T, C return self.c proj y Two details matter here. Causal masking. During training, the model sees a whole sequence at once, so we must stop token t from peeking at token t+1 , or it would just copy the answer. is causal=True applies a lower-triangular mask so each position only attends to itself and the past. Multiple heads. With 1,536 dimensions and 24 heads, each head works in a 64-dimensional subspace. Different heads can specialize: one tracks bracket matching, another subject-verb agreement, another something no human has labeled. We run them in parallel and concatenate. Attention mixes information between tokens . The MLP then processes each token's vector independently : expand to 4x the width 1,536 → 6,144 , apply a nonlinearity GELU in my model , project back down. A large share of a transformer's parameters, and much of its stored "knowledge", live in these MLPs. After the final block and a last LayerNorm, each position's vector is multiplied by the embedding matrix again , this time in reverse, to produce one score a logit per vocabulary token. Softmax turns logits into probabilities. Sharing the same matrix for input embedding and output projection is called weight tying . It saves ~77M parameters and acts as a regularizer, because the embedding space has to be useful for both reading and predicting. Training data is a long stream of tokens. For every position, the target is simply the next token. The loss is cross-entropy between the predicted distribution and that target: logits = model x B, T, vocab loss = F.cross entropy logits.view -1, logits.size -1 , targets.view -1 , ignore index=-100, loss.backward optimizer.step That's it. No labels beyond the text itself. This is why LLM pretraining scales: the internet is the dataset. A handy way to read the loss: perplexity = exp loss . My best 24-layer empathy checkpoint hit a validation loss of 3.9259, which is a perplexity of about 50.7. Loosely: the model is as uncertain as if it were choosing between ~51 equally likely tokens at each step. To generate text, you feed in a prompt, get a distribution over the next token, sample one, append it, and repeat. How you sample matters enormously: python def sample next logits, generated ids, temperature=0.7, top k=20, rep penalty=1.2 : penalize tokens we've already produced for t in set generated ids : logits t = logits t / rep penalty if logits t 0 else logits t rep penalty logits = logits / temperature <1 = sharper, 1 = flatter topk vals, topk idx = torch.topk logits, top k keep only the k best probs = F.softmax topk vals, dim=-1 return topk idx torch.multinomial probs, 1 Notice something wasteful: when generating token 101, the keys and values for tokens 1-100 haven't changed. A KV cache stores them so each new token costs one token's worth of compute instead of re-processing the whole prefix. Qwen via Hugging Face uses a KV cache. My from-scratch model does not , and as you'll see in the benchmarks, that single omission matters a lot. I deliberately followed GPT-2's design so every piece would be something I could fully explain. Two sizes: Shared settings: 256-token context, 4x MLP expansion, GELU, pre-norm LayerNorm with bias, dropout 0.1, tied embeddings, learned positions, and gradient checkpointing, which trades extra compute for roughly 40% less activation memory. One initialization detail worth knowing, straight from the GPT-2 paper. The output projections inside each block c proj are initialized with a smaller standard deviation, 0.02 / sqrt 2 n layer . Every block adds to the residual stream, so without this scaling the stream's variance grows with depth and early training gets chaotic. You can sanity-check the parameter count with back-of-envelope math: each block has roughly 12 x d^2 parameters 4d² for attention's Q/K/V/output, 8d² for the MLP . For d = 1,536 that's ~28.3M per block, x 24 blocks = ~680M. Add the 77M embedding table and you land right at ~757M. My first pretraining ran on CPU only , float32, 8 threads, at about 215 tokens per second on a ~2.5M-token code-and-prose corpus. Two runs totaling roughly 2h44m got the 253M model to a best validation loss of 4.2803 . That number is not good. It's the kind of model that produces fluent-looking nonsense. But it was a complete, working end-to-end loop, and everything after it built on that foundation. Next I trained the 24-layer, 757M configuration on empathetic dialogue EmpatheticDialogues + ESConv, about 2.1M train tokens on an RTX 5060 with bfloat16 mixed precision. Effective batch: 32 sequences x 256 tokens = 8,192 tokens per step. Validation loss over time: Good progress. But when I actually read the outputs , something was badly wrong. The model's replies were non-sequiturs. It would answer "I feel like nobody understands me" with something fluent and completely unrelated. The loss curve looked healthy. The behavior was broken. Digging into the corpus, I found two structural defects: Lesson: a falling validation loss only tells you the model is getting better at the distribution you gave it. It says nothing about whether that distribution was the right one. The fix was a new preprocessing script that rebuilds the corpus with an explicit format: User: