cd /news/large-language-models/training-a-small-gpt-style-language-… · home › topics › large-language-models › article
[ARTICLE · art-170822] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Training a Small GPT-Style Language Model from Scratch with PyTorch, Hugging Face, and CUDA

A developer built a roughly 1.06-million-parameter GPT-style causal language model from scratch using PyTorch, Hugging Face Transformers, Datasets, and Tokenizers, and NVIDIA CUDA, training it on a 300-record subset of TinyStories. The decoder-only model (3 layers, 4 attention heads, hidden size 128, context length 128, 3,512-token byte-level BPE vocabulary) cut training loss from 7.5319 to 6.6421 and validation loss from 7.0656 to 6.3688 over two epochs, and the saved checkpoint reloaded successfully via AutoModelForCausalLM.from_pretrained(). The developer notes generated text remains repetitive and grammatically inconsistent, framing the run as a validation of the end-to-end pipeline rather than a capable model.

by read3 min views9 publishedNov 16, 2026

Introduction

Understanding large language models becomes much more practical when you move beyond pretrained checkpoints and explore the training process yourself.

I recently extended a PyTorch Transformer project with a generative language-model training workflow using Hugging Face Datasets, Tokenizers, Transformers, and CUDA.

The result is a small, approximately 1.06-million-parameter GPT-style causal language model trained on a subset of TinyStories.

This article documents the architecture, training pipeline, model serialization, experimental results, and limitations.

The project contains two distinct Transformer workflows:

A custom PyTorch Transformer encoder for binary text classification.

A new decoder-only GPT-style causal language model for next-token prediction.

The generative model uses Hugging Face's GPT2Config and GPT2LMHeadModel, initialized with random weights rather than pretrained GPT-2 weights.

Technology stack

Python and PyTorch

Hugging Face Transformers

Hugging Face Datasets

Hugging Face Tokenizers

NVIDIA CUDA

AdamW optimizer

Safetensors and Hugging Face Hub

The experimental configuration uses three decoder layers, four attention heads, a hidden dimension of 128, and a maximum context length of 128 tokens.

I used the TinyStories dataset, available at:

https://huggingface.co/datasets/roneneldan/TinyStories The pipeline loads examples using Hugging Face Datasets streaming, filters unusable text, shuffles the selected corpus, and divides the samples into training and validation sets.

For this run, I used: 270 training records

30 validation records

A fixed random seed

This deliberately small corpus was selected to validate the end-to-end implementation rather than to produce a capable general-purpose language model.

The pipeline trains a byte-level BPE tokenizer directly on the training texts.

It includes padding, unknown, beginning-of-sequence, and end-of-sequence tokens.

The resulting vocabulary contains 3,512 tokens.

Training the tokenizer only on the training split keeps the held-out validation texts separate from vocabulary learning.

The model is initialized using Hugging Face's GPT-2 architecture with randomly initialized weights.

Its essential configuration includes:

Hidden size: 128

Decoder layers: 3

Attention heads: 4

Context length: 128

Vocabulary size: 3,512

Trainable model parameters: 1,060,992

Unlike an encoder-based classification model, a causal language model predicts subsequent tokens using a causal attention mechanism.

The training data is tokenized into input sequences with corresponding next-token learning targets, while padding positions are excluded from the loss.

Training runs on an NVIDIA GPU through PyTorch CUDA.

The workflow uses AdamW, cross-entropy-based causal language-model loss, gradient clipping, and validation after each epoch.

The best-performing checkpoint is selected using validation loss.

Training results

Metric

Epoch 1

Epoch 2

Training loss

7.5319

6.6421

Validation loss

7.0656

6.3688

Both reported losses decreased during the two-epoch experiment.

These are preliminary results, not evidence of general-purpose language understanding or performance on standardized language-model benchmarks.

The training program exports the model configuration, tokenizer, and weights into a directory compatible with Hugging Face Transformers.

The saved model was successfully reloaded using AutoModelForCausalLM.from_pretrained() and AutoTokenizer.from_pretrained().

The generated text is still repetitive and grammatically inconsistent, which is expected given the small model capacity and extremely limited training.

Nevertheless, the experiment validates the complete technical path from dataset ingestion through inference.

Tokenizer design matters. Vocabulary size and tokenization affect sequence representation and the model's output space.

Architecture matters. A Transformer encoder designed for classification is not directly interchangeable with a decoder-only generative model.

Validation matters. Saving the checkpoint with the lowest validation loss provides a more useful selection criterion than relying solely on training loss.

GPU memory matters. Smaller architectures, shorter contexts, and conservative batch sizes make experimentation possible on consumer hardware.

Reproducibility matters. Model configurations, tokenizer files, dependency information, dataset provenance, and known limitations should be documented.

My next development priorities include:

Increasing training data and token coverage.

Comparing model depth, hidden dimensions, and context lengths.

Adding held-out perplexity and generation-quality evaluation.

Measuring GPU memory consumption and training throughput.

Exploring mixed-precision training and memory-efficient attention where hardware supports them.

Improving model documentation and reproducibility.

Conclusion

This experiment helped connect Transformer architecture concepts with the practical workflow of training a generative language model.

The current checkpoint is a small proof of concept, not a production-grade model. Its value is in demonstrating the engineering steps required to create, validate, save, and reload a causal language model.

Hugging Face model:

https://huggingface.co/sudhanshu42844/my-tiny-llm Feedback on the training pipeline, model evaluation, and efficient experimentation with small language models is welcome.

── more in #large-language-models 4 stories · sorted by recency
── more on @pytorch 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/training-a-small-gpt…] indexed:0 read:3min 2026-11-16 · —