Introduction
Understanding large language models becomes much more practical when you move beyond pretrained checkpoints and explore the training process yourself.
I recently extended a PyTorch Transformer project with a generative language-model training workflow using Hugging Face Datasets, Tokenizers, Transformers, and CUDA.
The result is a small, approximately 1.06-million-parameter GPT-style causal language model trained on a subset of TinyStories.
This article documents the architecture, training pipeline, model serialization, experimental results, and limitations.
The project contains two distinct Transformer workflows:
A custom PyTorch Transformer encoder for binary text classification.
A new decoder-only GPT-style causal language model for next-token prediction.
The generative model uses Hugging Face's GPT2Config and GPT2LMHeadModel, initialized with random weights rather than pretrained GPT-2 weights.
Technology stack
Python and PyTorch
Hugging Face Transformers
Hugging Face Datasets
Hugging Face Tokenizers
NVIDIA CUDA
AdamW optimizer
Safetensors and Hugging Face Hub
The experimental configuration uses three decoder layers, four attention heads, a hidden dimension of 128, and a maximum context length of 128 tokens.
I used the TinyStories dataset, available at:
https://huggingface.co/datasets/roneneldan/TinyStories The pipeline loads examples using Hugging Face Datasets streaming, filters unusable text, shuffles the selected corpus, and divides the samples into training and validation sets.
For this run, I used: 270 training records
30 validation records
A fixed random seed
This deliberately small corpus was selected to validate the end-to-end implementation rather than to produce a capable general-purpose language model.
The pipeline trains a byte-level BPE tokenizer directly on the training texts.
It includes padding, unknown, beginning-of-sequence, and end-of-sequence tokens.
The resulting vocabulary contains 3,512 tokens.
Training the tokenizer only on the training split keeps the held-out validation texts separate from vocabulary learning.
The model is initialized using Hugging Face's GPT-2 architecture with randomly initialized weights.
Its essential configuration includes:
Hidden size: 128
Decoder layers: 3
Attention heads: 4
Context length: 128
Vocabulary size: 3,512
Trainable model parameters: 1,060,992
Unlike an encoder-based classification model, a causal language model predicts subsequent tokens using a causal attention mechanism.
The training data is tokenized into input sequences with corresponding next-token learning targets, while padding positions are excluded from the loss.
Training runs on an NVIDIA GPU through PyTorch CUDA.
The workflow uses AdamW, cross-entropy-based causal language-model loss, gradient clipping, and validation after each epoch.
The best-performing checkpoint is selected using validation loss.
Training results
Metric
Epoch 1
Epoch 2
Training loss
7.5319
6.6421
Validation loss
7.0656
6.3688
Both reported losses decreased during the two-epoch experiment.
These are preliminary results, not evidence of general-purpose language understanding or performance on standardized language-model benchmarks.
The training program exports the model configuration, tokenizer, and weights into a directory compatible with Hugging Face Transformers.
The saved model was successfully reloaded using AutoModelForCausalLM.from_pretrained() and AutoTokenizer.from_pretrained().
The generated text is still repetitive and grammatically inconsistent, which is expected given the small model capacity and extremely limited training.
Nevertheless, the experiment validates the complete technical path from dataset ingestion through inference.
Tokenizer design matters. Vocabulary size and tokenization affect sequence representation and the model's output space.
Architecture matters. A Transformer encoder designed for classification is not directly interchangeable with a decoder-only generative model.
Validation matters. Saving the checkpoint with the lowest validation loss provides a more useful selection criterion than relying solely on training loss.
GPU memory matters. Smaller architectures, shorter contexts, and conservative batch sizes make experimentation possible on consumer hardware.
Reproducibility matters. Model configurations, tokenizer files, dependency information, dataset provenance, and known limitations should be documented.
My next development priorities include:
Increasing training data and token coverage.
Comparing model depth, hidden dimensions, and context lengths.
Adding held-out perplexity and generation-quality evaluation.
Measuring GPU memory consumption and training throughput.
Exploring mixed-precision training and memory-efficient attention where hardware supports them.
Improving model documentation and reproducibility.
Conclusion
This experiment helped connect Transformer architecture concepts with the practical workflow of training a generative language model.
The current checkpoint is a small proof of concept, not a production-grade model. Its value is in demonstrating the engineering steps required to create, validate, save, and reload a causal language model.
Hugging Face model:
https://huggingface.co/sudhanshu42844/my-tiny-llm Feedback on the training pipeline, model evaluation, and efficient experimentation with small language models is welcome.