{"slug": "built-gpt-2-on-custom-deep-learning-framework-i-built-from-scratch-in-c", "title": "Built GPT-2 on Custom Deep Learning Framework I built from scratch in C++", "summary": "Developer Akshay Muchaklavya built Forge, a custom deep learning framework written from scratch in C++, and used it to implement GPT-2 that matches Hugging Face's transformers token-for-token under greedy decoding. The framework includes core primitives like Linear, LayerNorm, SelfAttention, optimizers, and AVX2 kernels, and supports loading real pretrained weights via safetensors. Forge is available on GitHub and requires CMake 3.20+, a C++20 compiler, OpenBLAS, and a CPU with AVX2 support.", "body_md": "[Overview](#overview)[Installation](#installation)[Tensor](#tensor-documentation)[Linear Layer](#linear-layer-documentation)[Activations](#activations)[Loss functions](#loss-functions-documentation)[Optimizers](#optimizers-documentation)[LayerNorm](#layernorm)[Self-Attention](#self-attention)[Embeddings](#embeddings)[Tokenizer](#tokenizer)[Parameters extraction & Model Load/Save (safetensors)](#reflection-based-parameters--model-loadsave-safetensors)\n\nForge implements core deep learning abstractions and primitives, enabling you to build, train, and optimize neural network models efficiently. The framework is designed for simplicity, making it suitable for learning use cases.\n\nMost people learn deep learning by calling `model.fit()`\n\nand trusting PyTorch got the internals right. I wanted to know *why* it's right - how a tensor actually sits in memory, why GEMM dominates a forward pass, why attention needs a causal mask and what breaks silently if you get a transpose wrong.\n\nSo Forge is a deep learning framework built from scratch. Every primitive here - Linear, LayerNorm, SelfAttention, Optimizers, and some of the the AVX2 kernels underneath - exists because I wrote it myself and verified it against a real reference.\n\nThe proof: Forge's GPT-2, loaded with real pretrained weights, matches Hugging Face's `transformers`\n\n**token-for-token** under greedy decoding. Not close - exact. That's what convinces me this actually taught me how modern AI works, not just how to produce something that looks like it does.\n\n- CMake 3.20+\n- A C++20 compiler\n- Windows: MinGW-w64 (tested with the toolchain bundled in CLion)\n- Linux: GCC or Clang\n\n- OpenBLAS\n- A CPU with AVX2 support (any x86-64 CPU from roughly the last 10 years)\n\nEigen, reflect-cpp and ctti are fetched automatically via CMake's `FetchContent`\n\n-\nno manual setup needed for either.\n\n-\nInstall OpenBLAS. If you don't already have it, grab a prebuilt release from the\n\n[OpenBLAS releases page](https://github.com/OpenMathLib/OpenBLAS/releases)and extract it somewhere, e.g.`C:/Libs/OpenBLAS`\n\n. -\nClone the repo:\n\n```\n   git clone https://github.com/muchlakshay/Forge\n   cd Forge\n```\n\n- Configure and build:\n\n```\n   cmake -B build -G \"MinGW Makefiles\" -DCMAKE_BUILD_TYPE=Release -DOPENBLAS_ROOT=\"C:/Libs/OpenBLAS\"\n   cmake --build build\n```\n\nIf OpenBLAS is somewhere other than `C:/Libs/OpenBLAS`\n\n, point `-DOPENBLAS_ROOT`\n\nat wherever you extracted it.\n\n`Forge`\n\nbuilds as a static library at`build/libForge.a`\n\n, with headers under`core/`\n\nand`primitives/`\n\nin the repo itself.\n\n- Install OpenBLAS and a compiler toolchain:\n\n```\n   sudo apt install build-essential cmake libopenblas-dev\n```\n\n- Clone the repo:\n\n```\n   git clone https://github.com/muchlakshay/Forge\n   cd Forge\n```\n\n- Configure and build:\n\n```\n   cmake -B build -DCMAKE_BUILD_TYPE=Release\n   cmake --build build\n```\n\n`OPENBLAS_ROOT`\n\ndefaults to `/usr`\n\n, which matches where `apt`\n\ninstalls it -\nno extra flag needed unless you built OpenBLAS from source somewhere custom.\n\n`Forge`\n\nbuilds as a static library at`build/libForge.a`\n\n.\n\nForge includes `gpt2`\n\nand `mnist`\n\ntest executables that demonstrate the\nframework in action. They're off by default\nso a plain build only produces the library. To build them too:\n\n```\ncmake -B build -DFORGE_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release\ncmake --build build\n```\n\nThis adds `gpt2`\n\n/`mnist`\n\n(Linux) or `gpt2.exe`\n\n/`mnist.exe`\n\n(Windows) to\n`build/`\n\n. Prebuilt version of these for windows are also available on the\n[Releases page](https://github.com/muchlakshay/Forge/releases/tag/0.1) if you'd rather skip building them yourself.\n\nSince `Forge`\n\nlinks OpenBLAS as a shared library rather than statically, any\nexecutable you link against `Forge`\n\nneeds the OpenBLAS runtime library\navailable at runtime, not just at link time:\n\n**Windows:** copy`libopenblas.dll`\n\n(found under`OPENBLAS_ROOT/bin`\n\n) into the same folder as your built executable. Without it, the executable will fail to launch with a missing-DLL error.**Linux:** either install OpenBLAS system-wide (`sudo apt install libopenblas0`\n\n), or ensure`libopenblas.so`\n\n/`libopenblas.so.0`\n\nis somewhere on your`LD_LIBRARY_PATH`\n\n.\n\nThe `tests/CMakeLists.txt`\n\nin this repo already handles this automatically\nfor the `gpt2`\n\n/`mnist`\n\nexecutables via a post-build copy step on Windows -\nif you're linking your own executable against `Forge`\n\noutside that setup,\nyou'll need to do this step yourself.\n\n| Platform | Arch | Tested OS Version | Most Thoroughly Tested |\n|---|---|---|---|\n| Windows | x64 | Windows 11 Pro version 25H2 | yes |\n| Linux | x64 | Ubuntu 26.04 (via WSL2) | yes |\n\nCurrently, Forge uses **Eigen** as its underlying math library for tensor operations. However, **Eigen's abstraction and lazy evaluation (expression templates) introduce significant performance bottlenecks**, especially during backpropagation.\n\n**Performance Optimization Plan:**\n\n- Replace Eigen with\n**OpenBLAS** for optimized linear algebra operations - Implement\n**custom vectorized element-wise kernels** for backward pass computations - These optimizations will dramatically improve training speed and throughput\n\n**Multi-dimensional tensors** supporting up to**4 dimensions** for handling batch data, spatial dimensions, and feature channels- Optimized for deep learning use cases (not a general-purpose tensor library)\n- Core abstraction layer for all neural network computations\n\n**CPU radix tree-based memory allocator** for efficient memory allocation and deallocation- Optimized for the allocation patterns typical in deep learning workflows\n\n**Linear Layer**: Fully connected layer with learnable weights and biases** Layer Normalization**: Normalization across features** Embedding Layer**: Token/index embedding lookups** Sinusoidal Encoding**: Positional encodings for sequence models** Multi-Head Self Attention**: Core component for transformer-based architectures- BPE Tokenizer\n\n- ReLU\n- GELU\n- Sigmoid\n- Tanh\n- Softmax\n\n- Cross-Entropy Loss, with fused softmax (for multi-class classification)\n- Binary Cross-Entropy Loss, with fused softmax)\n- Mean Squared Error (MSE)\n\n**SGD**: Standard stochastic gradient descent** SGD with Momentum**: Accelerated gradient-based optimization** Adam**: Adaptive learning rate optimizer** AdamW**: Adam with weight decay regularization\n\n**CPU Backend**(with Eigen-based math, performance optimizations in progress)\n\n**CUDA Backend**(GPU acceleration for NVIDIA devices)** Optimized kernels**(OpenBLAS + custom vectorized kernels)\n\nForge is ideal for:\n\n- Learning deep learning systems implementation from first principles\n- Experimenting with neural network architectures\n\nThe **Tensor** class is the core abstraction in Forge for multi-dimensional data. It encapsulates:\n\n**Data storage** via CPU memory allocator (radix tree-based)**Shape and stride information** for multi-dimensional indexing**Automatic differentiation** support for backpropagation**Data type (dtype)** and**device** information\n\n```\nclass Tensor {\n    Device m_device;                          // CPU or GPU\n    Dtype m_dtype;                            // float32, float64, int32, int16, etc.\n    std::shared_ptr<StorageAbstract> m_storage; // Actual data buffer\n    std::vector<std::size_t> m_shape;         // Dimensions: (batch, height, width, channels)\n    std::vector<std::size_t> m_strides;       // Row-major strides for efficient indexing\n    std::size_t m_size;                       // Total element count\n    bool m_need_grads;                        // Requires gradients during backward?\n    DispatchKey m_dispatch_key;               // CPU/CUDA dispatch key\n    mutable std::shared_ptr<NodeAbstract> m_node;   // Autograd computation graph node\n    std::shared_ptr<Tensor> m_grads;          // Accumulated gradients tensor\n};\n#include \"Forge.h\"\nusing namespace Forge;\n\n// Create empty tensor\nTensor t;\n\n// Create tensor with shape (batch=32, seq_len=128, features=64)\nstd::vector<std::size_t> shape = {32, 128, 64};\nTensor embeddings(shape, Dtype::float32, true, Device::CPU);\n// Create batch of 10 samples with 5 features, initialized to zero\nTensor zeros = Tensor::Zeros({10, 5}, true);\n\n// Create ones matrix for bias initialization\nTensor ones = Tensor::Ones({1, 64}, true);\n\n// Create constant tensor with value 0.5\nTensor half_tensor = Tensor::Constant({8, 8}, 0.5f, true);\n\n// Create range tensor: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\nTensor indices = Tensor::Range(0, 10, 1, true);\n// Convert numpy/C array to Forge tensor\nfloat data[] = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f};\nTensor from_data = Tensor::FromHostPtr(data, {2, 3}, true);\n\n// Direct initializer list initialization\nTensor matrix({2, 3});\nmatrix = {{1.0f, 2.0f, 3.0f}, \n          {4.0f, 5.0f, 6.0f}};\n// Create flat tensor\nTensor flat = Tensor::Range(0, 24, 1, true);  // [0, 1, 2, ..., 23]\n\n// Reshape to (2, 3, 4) - batch=2, height=3, width=4\nTensor reshaped = flat.reshape(2, 3, 4);\n// Elements remain the same, just different view\n\n// Another reshape: (2, 3, 4) -> (6, 4)\nTensor flattened = reshaped.reshape(6, 4);\nTensor original = Tensor::Ones({3, 3}, true);\n\n// Deep clone - independent copy\nTensor independent_copy = original.clone();\n\n// Copy data from one tensor to another\nTensor destination({3, 3});\ndestination.copy(original);\nTensor batch = Tensor::Zeros({32, 128, 64});  // batch_size=32\n\n// Get first sample from batch\nTensor first_sample = batch[0];  // Shape: (128, 64)\n\n// Get tenth sample\nTensor tenth_sample = batch[9];  // Shape: (128, 64)\n```\n\nForge integrates with Eigen for efficient element-wise operations:\n\n```\n// Create a weight matrix for linear layer\nTensor weights({64, 32}, true);  // 64 input features, 32 output features\nweights = 0.1f;\n\n// Get Eigen map for direct element access\nauto w_map = weights.as_eigen<float, 2>();\n\n// Set specific elements\nw_map(0, 0) = 0.5f;   // weights[0][0]\nw_map(10, 5) = -0.3f; // weights[10][5]\n\n// Element-wise operations via Eigen\nw_map = w_map * 2.0f;  // Scale all weights by 2\nw_map = w_map + 0.01f; // Add small constant (regularization)\nTensor t = Tensor::Range(0, 60, 1);\nt = t.reshape(3, 4, 5);\n\n// Get shape information\nconst auto& shape = t.shape();           // {3, 4, 5}\nstd::cout << \"Shape: [\" << shape[0] << \", \" << shape[1] << \", \" << shape[2] << \"]\\n\";\n\n// Get memory layout\nconst auto& strides = t.strides();       // {20, 5, 1} for row-major layout\n\n// Data type and device\nDtype dtype = t.dtype();                 // Dtype::float32\nDevice device = t.device();              // Device::CPU\n\n// Memory information\nstd::size_t total_elements = t.size();   // 60\nvoid* data_ptr = t.data();               // Raw pointer to buffer\n\n// Gradient requirement\nbool requires_grad = t.need_grads();     // true or false\n// Input features\nTensor x {Tensor::Ones({32, 10}, true)};  // 32 samples, 10 features\nLinear linear {10, 32};\n\n// Forward pass (simplified)\nTensor output = linear(x);  \n\n//trigger a backward pass\noutput.backward()\n\n//access gradients of x\nstd::cout<<x.gradients();\nTensor output = Linear(x);\n\n// Backward with graph retention (for multiple backward passes)\noutput.backward(true);  // keep_graph=true\n\n// Use gradients...\nauto grad1 = x.gradients().clone();\n\n// Clear and do another backward\nx.clear_grads();\noutput.backward(true);  // Second backward pass\n\n// Now gradients accumulated again\n```\n\nThe **Linear** layer is a fully connected (dense) layer that applies an affine transformation to the input. It's one of the fundamental building blocks in neural networks, implementing the operation:\n\n```\noutput = input @ weights^T + bias\n```\n\nWhere:\n\n**input** has shape`(batch_size, ..., input_features)`\n\n(supports up to 4D tensors)**weights** has shape`(input_features, output_features)`\n\n**bias** has shape`(1, 1, 1, output_features)`\n\n(broadcasted)**output** has shape`(batch_size, ..., output_features)`\n\n```\nclass Linear {\n    Tensor m_weights;              // Shape: (input_size, output_size)\n    Tensor m_bias;                 // Shape: (1, 1, 1, output_size)\n    DispatchKey m_dispatch_key;    // CPU or GPU dispatch key\n    std::size_t m_input_size;      // Number of input features\n    std::size_t m_output_size;     // Number of output features\n    bool m_using_bias;             // Use bias or not\n    Dtype m_dtype;                 // Data type (float32, float64, etc.)\n    Device m_device;               // Device (CPU or GPU)\n};\n#include \"Forge.h\"\nusing namespace Forge;\n\n// Create a linear layer: 128 input features -> 64 output features\n// Uses Xavier Normal initialization by default\nLinear linear_1(128, 64);\n\n// With custom data type\nLinear linear_2(256, 128, Initializers::he_normal, Dtype::float32);\n\n// Without bias term\nLinear linear_no_bias(100, 50, Initializers::xavier_normal, Dtype::float32, Device::CPU, false);\nLinear(\n    std::size_t input_size,                    // Number of input features\n    std::size_t output_size,                   // Number of output features\n    Initializers initializer = Initializers::xavier_normal,  // Weight initialization\n    Dtype dtype = Dtype::float32,              // Data type\n    Device device = Device::CPU,               // Device (CPU/GPU)\n    bool bias = true                           // Use bias term?\n);\n// Xavier Normal: Good for networks with sigmoid/tanh activations\nLinear layer_xavier(128, 64, Initializers::xavier_normal);\n\n// He Normal: Recommended for ReLU-based networks\nLinear layer_he(128, 64, Initializers::he_normal);\n\n// Xavier Uniform: Uniform distribution variant\nLinear layer_xu(128, 64, Initializers::xavier_uniform);\n\n// He Uniform: He initialization with uniform distribution\nLinear layer_hu(128, 64, Initializers::he_uniform);\n// Create layer and input\nLinear fc(784, 128);\nTensor input({32, 784});  // batch_size=32, features=784\n\n// Forward pass using operator()\nTensor output = fc(input);  // Returns shape: (32, 128)\n\nstd::cout << \"Input shape: \";\nfor (auto s : input.shape()) std::cout << s << \" \";\nstd::cout << \"\\nOutput shape: \";\nfor (auto s : output.shape()) std::cout << s << \" \";\nstd::cout << \"\\n\";\njs\nLinear layer(64, 32);\n\n// Get read-only references\nconst auto& weights = layer.weights();     // Shape: (64, 32)\nconst auto& bias = layer.bias();           // Shape: (1, 1, 1, 32)\n\n// Get mutable references for optimization\nauto& weights_mut = layer.weights();\nauto& bias_mut = layer.bias();\n\n// Inspect dimensions\nstd::size_t in_size = layer.input_size();   // 64\nstd::size_t out_size = layer.output_size(); // 32\n\n// Print shapes\nstd::cout << \"Weights shape: \" << weights.shape()[0] << \" x \" << weights.shape()[1] << \"\\n\";\nstd::cout << \"Bias shape: \" << bias.size() << \" elements\\n\";\nLinear layer(512, 256);\n\n// Query layer metadata\nstd::cout << \"Layer configuration:\\n\";\nstd::cout << \"  Input size: \" << layer.input_size() << \"\\n\";\nstd::cout << \"  Output size: \" << layer.output_size() << \"\\n\";\nstd::cout << \"  Data type: \" << static_cast<int>(layer.dtype()) << \"\\n\";\nstd::cout << \"  Device: \" << static_cast<int>(layer.device()) << \"\\n\";\n\n// Weights storage\nconst auto& w = layer.weights();\nstd::cout << \"  Weights total elements: \" << w.size() << \"\\n\";\nstd::cout << \"  Bias total elements: \" << layer.bias().size() << \"\\n\";\n```\n\nThis module provides the activation functions used inside models. Each one is a small stateless functor - construct it (or use a temporary) and call it like a function on a `Tensor`\n\n. Gradients are wired up automatically whenever the input requires them, so you never call a grads function directly; it runs as part of the normal `.backward()`\n\npass.\n\n``` js\nTensor operator()(const Tensor& input) const;\n```\n\nReturns a new tensor of the same shape as `input`\n\n, with ReLU applied element-wise.\n\n```\nRelu(x) = max(x, 0)\nForge::Relu relu;\n\nTensor x = ...;     // any shape\nTensor y = relu(x); // same shape as x\ny.backward();\njs\nTensor operator()(const Tensor& input) const;\n```\n\nReturns a new tensor of the same shape as `input`\n\n, with the sigmoid (logistic) function applied element-wise.\n\n```\nSigmoid(x) = 1 / (1 + exp(-x))\nForge::Sigmoid sigmoid;\n\nTensor x = ...;        // any shape\nTensor y = sigmoid(x); // same shape as x, values in (0, 1)\ny.backward();\njs\nTensor operator()(const Tensor& input) const;\n```\n\nReturns a new tensor of the same shape as `input`\n\n, with the hyperbolic tangent applied element-wise.\n\n```\nTanh(x) = tanh(x)\nForge::Tanh tanh_;\n\nTensor x = ...;      // any shape\nTensor y = tanh_(x); // same shape as x, values in (-1, 1)\ny.backward();\njs\nTensor operator()(const Tensor& input) const;\n```\n\nReturns a new tensor of the same shape as `input`\n\n, with Leaky ReLU applied element-wise.\n\n```\nLeakyRelu(x) = x        if x > 0\n             = a * x    otherwise        # a = 0.01, fixed internally\nForge::LeakyRelu leaky_relu;\n\nTensor x = ...;            // any shape\nTensor y = leaky_relu(x);  // same shape as x\ny.backward();\n```\n\nThe negative slope `a`\n\nis hardcoded at `0.01`\n\n- there's currently no constructor parameter to change it.\n\n``` js\nTensor operator()(const Tensor& input) const;\n```\n\nReturns a new tensor of the same shape as `input`\n\n, with GELU applied element-wise, using the tanh-based approximation (the same one used in GPT-style implementations).\n\n```\nGelu(x) = 0.5 * x * (1 + tanh(k * (x + c * x^3)))\n\n  k = sqrt(2 / pi) = 0.7978845608028654\n  c = 0.044715\nForge::Gelu gelu;\n\nTensor x = ...;     // any shape\nTensor y = gelu(x); // same shape as x\ny.backward();\n```\n\nThis is the tanh approximation, not the exact erf-based formula (`0.5 * x * (1 + erf(x / sqrt(2)))`\n\n). Results are very close but not bit-identical to an exact GELU implementation.\n\n``` js\nTensor operator()(const Tensor& input) const;\n```\n\nComputes softmax along the **last axis** of a **4D** tensor, returning a tensor of the same shape. This is the same shape convention used by attention scores, e.g. `(batch, heads, seq_len, seq_len)`\n\n.\n\nFor each row `x`\n\nalong the last axis:\n\n```\nSoftmax(x)_i = exp(x_i - max(x)) / sum_j(exp(x_j - max(x)))\n```\n\nThe `- max(x)`\n\nsubtraction is purely for numerical stability and doesn't change the result.\n\n```\nForge::Softmax softmax;\n\nTensor scores = ...;            // shape (batch, heads, seq_len, seq_len)\nTensor probs = softmax(scores); // same shape, normalized over the last axis\nprobs.backward();\n```\n\n`Softmax`\n\nspecifically reduces over axis index `3`\n\n(the last of four dims) - passing a 2D or 3D tensor won't normalize correctly. If you need softmax over a different rank/axis, reshape first.\n\n**All six are stateless**- there's nothing to configure or store, so`Forge::Relu{}(x)`\n\nworks just as well as keeping a named instance around.and work on any tensor shape/rank, always preserving it.`Relu`\n\n,`Sigmoid`\n\n,`Tanh`\n\n,`LeakyRelu`\n\n, and`Gelu`\n\nare purely element-wise`Softmax`\n\nis the one exception, requiring a 4D input as noted above.\n\nLoss functions quantify the difference between predictions and ground truth.\n\n```\nMSE mse;\nTensor loss = mse(predictions, targets);\n```\n\n**Definition:** `L = (1/N) * Σ(predictions_i - targets_i)²`\n\n**Use Cases:** Regression, reconstruction tasks\n\n**API:**\n\n``` js\nTensor operator()(const Tensor& predictions, const Tensor& targets);\nCrossEntropy ce;\nTensor loss = ce(logits, targets);  // targets: one-hot encoded\n```\n\n**Definition:** `L = -(1/N) * Σ targets_i * log(softmax(logits_i))`\n\n**Use Cases:** Multi-class classification\n\n**Note:** Softmax is fused internally for numerical stability. Pass logits, not probabilities.\n\n**API:**\n\n``` js\nTensor operator()(const Tensor& predictions, const Tensor& ground_truth);\nBinaryCrossEntropy bce;\nTensor loss = bce(logits, targets);  // targets: 0 or 1\n```\n\n**Definition:** `L = -(1/N) * Σ [targets_i * log(sigmoid(logits_i)) + (1 - targets_i) * log(1 - sigmoid(logits_i))]`\n\n**Use Cases:** Binary classification, multi-label classification\n\n**Note:** Sigmoid is fused internally for numerical stability. Pass logits, not probabilities.\n\n**API:**\n\n``` js\nTensor operator()(const Tensor& predictions, const Tensor& ground_truth);\n```\n\nThis module provides the public-facing optimizers used to train models. Each optimizer owns the parameters it was constructed with, along with any internal state (momentum/moment buffers), and updates those parameters in place each time `update()`\n\nis called. Internally, `update()`\n\ndispatches to a device-specific backend (currently only CPU supported).\n\nTwo optimizers are currently available:\n\n- stochastic gradient descent, with optional momentum`Forge::SGD`\n\n- adaptive moment estimation, with optional weight decay`Forge::Adam`\n\n```\nexplicit SGD(std::vector<Parameter> parameters, float lr = 0.01f, float momentum_coef = 0.0f);\n```\n\n| Argument | Meaning |\n|---|---|\n`parameters` |\nThe parameters this optimizer will update. Must all share the same device and dtype; duplicates are silently removed. |\n`lr` |\nLearning rate (step size). Default `0.01` . |\n`momentum_coef` |\nMomentum coefficient, must be in `[0, 1]` . `0.0` (default) = plain SGD. `> 0.0` = momentum SGD. |\n\nIf `momentum_coef > 0`\n\n, a zero-initialized velocity buffer is allocated internally for each parameter - no manual buffer setup needed.\n\n| Method | Meaning |\n|---|---|\n`update()` |\nApplies one optimization step to all owned parameters, using their currently accumulated gradients. |\n`clear_grads()` |\nZeroes the gradients of all owned parameters. Call after `update()` , before the next backward pass. |\n`setLearningRate(lr)` |\nUpdates the learning rate. |\n`setMomentumCoef(momentum_coef)` |\nUpdates the momentum coefficient (must be in `[0, 1]` ). |\n`parameters()` |\nRead-only access to the owned parameters. |\n`learningRate()` |\nReturns the current learning rate. |\n`momentumCoef()` |\nReturns the current momentum coefficient. |\n\nFor each parameter `p`\n\nwith gradient `g`\n\n:\n\n-\n**Without momentum**(`momentum_coef == 0`\n\n):\n\n```\np = p - lr * g\n```\n\n-\n**With momentum**(`momentum_coef > 0`\n\n), using internal velocity buffer`V`\n\n:\n\n```\nV = momentum_coef * V + g\np = p - lr * V\n```\n\n`V`\n\naccumulates an exponentially-weighted sum of past gradients, smoothing the descent direction and accelerating convergence along consistent gradient directions.\n\n```\nForge::SGD optimizer(model.parameters(), /*lr=*/0.01f, /*momentum_coef=*/0.9f);\nForge::MSE loss_fn;\n\nfor (auto& batch : batches) {\n    optimizer.clear_grads();\n    pred = model(batch)\n    auto loss = loss_fn(ground_truth, pred);\n    loss.backward();\n    optimizer.update();\n}\njs\nexplicit Adam(const std::vector<Parameter>& parameters, float lr = 0.01f, float beta_1 = 0.9f,\n    float beta_2 = 0.999f, float decay_factor = 0.01f);\n```\n\n| Argument | Meaning |\n|---|---|\n`parameters` |\nThe parameters this optimizer will update. Must all share the same device and dtype; duplicates are silently removed. |\n`lr` |\nLearning rate. Default `0.01` . |\n`beta_1` |\nExponential decay rate for the first moment estimate, must be in `[0, 1]` . Default `0.9` . |\n`beta_2` |\nExponential decay rate for the second moment estimate, must be in `[0, 1]` . Default `0.999` . |\n`decay_factor` |\nWeight-decay coefficient, applied only to parameters whose `need_decay` flag is set. Default `0.01` . |\n\nZero-initialized first- and second-moment buffers are allocated internally for each parameter. The step counter used for bias correction (`epoch`\n\n) is managed internally, starting at `1`\n\nand incrementing automatically on every `update()`\n\ncall - no need to track it yourself.\n\n| Method | Meaning |\n|---|---|\n`update()` |\nApplies one Adam step to all owned parameters, then advances the internal step counter. |\n`clear_grads()` |\nZeroes the gradients of all owned parameters. Call after `update()` , before the next backward pass. |\n`reset()` |\nResets the internal step counter back to `1` (e.g. when restarting training). Does not reset the moment buffers. |\n`setLearningRate(lr)` |\nUpdates the learning rate. |\n`setBeta_1(beta_1)` |\nUpdates beta_1 (must be in `[0, 1]` ). |\n`setBeta_2(beta_2)` |\nUpdates beta_2 (must be in `[0, 1]` ). |\n`setDecayFactor(decay_rate)` |\nUpdates the weight-decay coefficient (must be in `[0, 1]` ). |\n`parameters()` |\nRead-only access to the owned parameters. |\n`firstMoment()` |\nRead-only access to the internal first-moment buffers (`M` ). |\n`secondMoment()` |\nRead-only access to the internal second-moment buffers (`V` ). |\n`beta_1()` / `beta_2()` |\nCurrent beta values. |\n`learningRate()` |\nCurrent learning rate. |\n`epoch()` |\nCurrent internal step counter. |\n`decayFactor()` |\nCurrent weight-decay coefficient. |\n\nFor each parameter `p`\n\nwith gradient `g`\n\n, at step `t = epoch`\n\n:\n\n```\nM = beta_1 * M + (1 - beta_1) * g           # first moment\nV = beta_2 * V + (1 - beta_2) * g^2          # second moment\n\nM_hat = M / (1 - beta_1^t)                  # bias-corrected first moment\nV_hat = V / (1 - beta_2^t)                  # bias-corrected second moment\n\np = p - lr * M_hat / (sqrt(V_hat) + e)       # e = 1e-8, fixed internally\n\nif need_decay:\n    p = p - lr * decay_factor * p           # weight decay, applied after the Adam step\nForge::Adam optimizer(model.parameters(), /*lr=*/0.001f, /*beta_1=*/0.9f,\n    /*beta_2=*/0.999f, /*decay_factor=*/0.01f);\nForge::MSE loss_fn;\n\nfor (auto& batch : dataloader) {\n    optimizer.clear_grads();\n    pred = model(batch)\n    auto loss = loss_fn(ground_truth, pred);\n    loss.backward();\n    optimizer.update();\n}\n```\n\n-\n**Construction validates inputs.** Constructing either optimizer with an empty parameter list, or with parameters spread across mixed devices/dtypes, throws`std::invalid_argument`\n\n. Duplicate parameters are detected and removed automatically, no need to de-duplicate yourself. -\nto`momentum_coef`\n\n,`beta_1`\n\n,`beta_2`\n\n, and`decay_factor`\n\nare all range-checked`[0, 1]`\n\n, both at construction and whenever set via their setters. -\n**Per-parameter weight decay:** decay is opt-in via each`Parameter`\n\n's`need_decay`\n\nflag, not a global switch - set this when building your model if you want decay to skip certain parameters (e.g. biases/LayerNorm scales). -\n- it starts at`Adam::epoch()`\n\nis managed for you`1`\n\nand increments on every`update()`\n\ncall. Use`reset()`\n\nif you need to restart bias correction from scratch (e.g. after loading a fresh set of parameters into an existing optimizer). -\n- remember to call it once per step (typically right before`clear_grads()`\n\nis separate from`update()`\n\n`loss.backward()`\n\n), or gradients will keep accumulating across steps.\n\n`Forge::LayerNorm`\n\nimplements layer normalization over the last dimension of a 3D input tensor (`batch, seq_len, d_model`\n\n), with learnable per-feature scale (`gamma`\n\n) and shift (`beta`\n\n) parameters. Like the optimizers, the actual math runs through a device-specific backend (CPU, GPU, ...) resolved internally - callers just construct a `LayerNorm`\n\nand call it like a function.\n\n``` js\nLayerNorm(std::size_t d_model, Dtype dtype, const Device& device, bool need_grads = true);\n```\n\n| Argument | Meaning |\n|---|---|\n`d_model` |\nSize of the last dimension to normalize over (the feature dimension). |\n`dtype` |\nData type for `gamma` , `beta` , and the computation. |\n`device` |\nDevice the parameters live on (e.g. CPU, GPU). |\n`need_grads` |\nWhether `gamma` /`beta` are trainable (accumulate gradients). Default `true` . |\n\nOn construction, `gamma`\n\nis initialized to ones and `beta`\n\nto zeros, both with shape `[d_model]`\n\n.\n\n``` js\nTensor operator()(const Tensor& input);\n```\n\nApplies layer normalization to `input`\n\nand returns a new tensor of the same shape.\n\n`input`\n\nmust be a 3D tensor shaped`(batch, seq_len, d_model)`\n\n, where the last dimension matches the`d_model`\n\nthis`LayerNorm`\n\nwas constructed with - otherwise it throws`std::invalid_argument`\n\n.`input`\n\nmust reside on the same device as the`LayerNorm`\n\ninstance - otherwise it throws`std::invalid_argument`\n\n.- If\n`input`\n\n,`gamma`\n\n, or`beta`\n\nrequire gradients, the necessary autograd node is attached automatically - no manual backward wiring needed.\n\n| Method | Meaning |\n|---|---|\n`gamma()` |\nMutable reference to the learnable scale parameter, shape `[d_model]` . |\n`beta()` |\nMutable reference to the learnable shift parameter, shape `[d_model]` . |\n`d_model()` |\nReturns the configured feature dimension size. |\n`d_device()` |\nReturns the device this `LayerNorm` operates on. |\n`dtype()` |\nReturns the configured data type. |\n\nFor each row `x`\n\nalong the last dimension (i.e. each `(batch, seq_len)`\n\nposition, a vector of length `d_model`\n\n):\n\n```\nmean = mean(x)                          # average over d_model\nvar  = mean((x - mean)^2)               # variance over d_model\n\nx_norm = (x - mean) / sqrt(var + eps)   # eps = 1e-5, fixed internally\n\ny = gamma * x_norm + beta               # gamma, beta broadcast over (batch, seq_len)\n```\n\n`gamma`\n\nand `beta`\n\nare applied element-wise per feature (broadcast across `batch`\n\nand `seq_len`\n\n), so every position in the sequence is rescaled/shifted the same way along the feature axis.\n\n```\nForge::LayerNorm ln(/*d_model=*/512, Dtype::Float32, Device::CPU);\n\nTensor x = ...; // shape (batch, seq_len, 512)\nTensor y = ln(x); // normalized output, same shape (batch, seq_len, 512)\n\ny.backward(); // gradients flow back into x, ln.gamma(), and ln.beta() automatically\n\n// gamma/beta are ordinary learnable parameters - hand them to an optimizer like any other:\nForge::Adam optimizer(ln.parameters(), /*lr=*/0.001f);\n```\n\n**Input must be 3D.**`LayerNorm`\n\nexpects`(batch, seq_len, d_model)`\n\n- reshape lower- or higher-rank tensors to this layout before calling it.**Normalization is over the last axis only.** Each`(batch, seq_len)`\n\nrow is normalized independently across its`d_model`\n\nfeatures; there is no cross-sequence or cross-batch normalization.and is not currently configurable from the constructor.`eps`\n\nis fixed at`1e-5`\n\nEven if you pass`need_grads`\n\nonly controls`gamma`\n\n/`beta`\n\n.`need_grads = false`\n\n, the output tensor will still require gradients if the`input`\n\nyou call it with requires gradients - gradient tracking is`need_grads OR input.need_grads()`\n\n.**Device/shape mismatches throw immediately**(`std::invalid_argument`\n\n) rather than silently broadcasting or casting - construct one`LayerNorm`\n\nper device/`d_model`\n\ncombination you need.\n\n`Forge::SelfAttention`\n\nimplements multi-head scaled dot-product self-attention with an optional causal mask, followed by a linear output projection. As with the other modules, the actual math runs through a device-specific backend resolved internally - you only ever interact with the `SelfAttention`\n\nobject itself.\n\n```\nSelfAttention(std::size_t d_model, std::size_t Q_K_dims, std::size_t V_dims, std::size_t heads, bool mask,\n    Device device, Dtype dtype = Dtype::float32, Initializers initializer = Initializers::he_normal);\n```\n\n| Argument | Meaning |\n|---|---|\n`d_model` |\nInput/output feature dimension. |\n`Q_K_dims` |\nDimension of the query/key projection, per head. |\n`V_dims` |\nDimension of the value projection, per head. |\n`heads` |\nNumber of attention heads. |\n`mask` |\nWhether this instance uses causal masking. If `true` , you must call `createMask(seq_len)` before the first forward pass. |\n`device` |\nDevice the weights live on. |\n`dtype` |\nData type for weights and computation. Default `float32` . |\n`initializer` |\nInitialization scheme used for the query/key/value projection weights. Default `he_normal` . |\n\nOn construction, the query/key projection weights are allocated with shape `(heads, d_model, Q_K_dims)`\n\n, the value projection weights with shape `(heads, d_model, V_dims)`\n\n, and all three are initialized using `initializer`\n\n. An internal `Linear`\n\nlayer (no bias) projects the concatenated multi-head output, `heads * V_dims`\n\n, back down to `d_model`\n\n.\n\n``` js\nTensor operator()(const Tensor& input) const;\n```\n\nRuns the forward pass and returns a tensor of shape `(batch, seq_len, d_model)`\n\n.\n\n`input`\n\nis expected as`(batch, seq_len, d_model)`\n\n(rank > 3 throws`std::invalid_argument`\n\n).- If this instance was constructed with\n`mask = true`\n\n, you must have called`createMask(seq_len)`\n\nfirst - otherwise it throws`std::runtime_error`\n\n. The mask's size must also match the input's sequence length, or it throws`std::invalid_argument`\n\n. - Gradients are wired up automatically for\n`input`\n\nand the query/key/value weights whenever any of them require gradients - no manual backward bookkeeping needed.\n\n```\nvoid createMask(std::size_t seq_len);\n```\n\nBuilds a `(seq_len, seq_len)`\n\ncausal mask and stores it internally. Only valid for instances constructed with `mask = true`\n\n(otherwise throws `std::runtime_error`\n\n). Call this once before the first forward pass, and again whenever `seq_len`\n\nchanges.\n\n| Method | Meaning |\n|---|---|\n`query()` |\nRead-only access to the query projection weights, shape `(heads, d_model, Q_K_dims)` . |\n`key()` |\nRead-only access to the key projection weights, shape `(heads, d_model, Q_K_dims)` . |\n`value()` |\nRead-only access to the value projection weights, shape `(heads, d_model, V_dims)` . |\n`linear()` |\nMutable access to the internal output-projection `Linear` layer (its own weights are also learnable - see the `Linear` module's docs). |\n`mask()` |\nRead-only access to the stored causal mask tensor. |\n`using_mask()` / `useMask()` |\nWhether this instance was constructed with masking enabled (both return the same flag). |\n`heads()` |\nNumber of attention heads. |\n`d_model()` |\nConfigured model dimension. |\n`device()` |\nDevice the weights live on. |\n`dtype()` |\nConfigured data type. |\n`dispatch_key()` |\nInternal device-routing key - not generally needed by callers. |\n\nShapes: `input`\n\nis `(batch, seq_len, d_model)`\n\n; `query_W`\n\n/`key_W`\n\nare `(heads, d_model, Q_K_dims)`\n\n; `value_W`\n\nis `(heads, d_model, V_dims)`\n\n. `@`\n\ndenotes matrix multiplication, batched over `batch`\n\nand `heads`\n\n.\n\nFor each head `h`\n\n:\n\n```\nQ = input @ query_W[h]                  # (batch, seq_len, Q_K_dims)\nK = input @ key_W[h]                    # (batch, seq_len, Q_K_dims)\nV = input @ value_W[h]                  # (batch, seq_len, V_dims)\n\nscores = (Q @ K^T) / sqrt(Q_K_dims)     # (batch, seq_len, seq_len)\n\nif using_mask:\n    scores = scores + mask              # mask has -10000 at future positions (j > i)\n\nattn = softmax(scores, axis=-1)         # (batch, seq_len, seq_len)\nhead_out = attn @ V                     # (batch, seq_len, V_dims)\n```\n\nThe per-head outputs are concatenated along the last axis and projected back to `d_model`\n\n:\n\n```\nconcat = concat_heads(head_out)         # (batch, seq_len, heads * V_dims)\noutput = concat @ W_out^T               # (batch, seq_len, d_model), W_out = linear() weights, no bias\n// Encoder-style self-attention, no mask\nForge::SelfAttention attn(/*d_model=*/512, /*Q_K_dims=*/64, /*V_dims=*/64, /*heads=*/8,\n    /*mask=*/false, Device::CPU);\n\nTensor x = ...; // shape (batch, seq_len, 512)\nTensor y = attn(x); // shape (batch, seq_len, 512)\ny.backward();\n// Decoder-style self-attention, causal mask\nForge::SelfAttention attn(/*d_model=*/512, /*Q_K_dims=*/64, /*V_dims=*/64, /*heads=*/8,\n    /*mask=*/true, Device::CPU);\n\nattn.createMask(seq_len); // required before the first forward pass\n\nTensor y = attn(x);\ny.backward();\n\n// query/key/value projection weights, plus the internal Linear's own weights,\n// are all ordinary learnable parameters - register them with your optimizer\nForge::Adam optimizer(attn.parameters(), /*lr=*/0.0001f);\n```\n\nForgetting it throws`createMask`\n\nmust be called before forward when`mask = true`\n\n.`std::runtime_error(\"create a mask prior to forward pass\")`\n\n.**The mask is fixed at the sequence length you build it for.** If your sequence length changes between batches, call`createMask(new_seq_len)`\n\nagain.**Heads up:** the input-vs-mask length check in`operator()`\n\ncompares`input.shape()[0]`\n\nagainst the mask's sequence length. For a`(batch, seq_len, d_model)`\n\ninput,`shape()[0]`\n\nis the batch size, not`seq_len`\n\n- it looks like it should be comparing against`shape()[1]`\n\ninstead. As written, this check only behaves correctly when`batch_size == seq_len`\n\n, so don't rely on it to catch a real mismatch.**The output projection has no bias**-`linear()`\n\nis constructed with bias disabled.**Weight initialization applies only to Q/K/V.** The internal`Linear`\n\nlayer initializes itself independently (see the`Linear`\n\nmodule's own docs for its default scheme).\n\n`Forge::Embedding`\n\nimplements a learned token embedding lookup table -\nmapping integer token IDs to dense `d_model`\n\n-dimensional vectors, the\nfirst layer in a typical transformer.\n\n```\nForge::Embedding embd(\n    d_model,               // embedding dimension\n    vocab_size,            // number of distinct tokens\n    Dtype::float32,        // must be float32 or float64 - int32 is rejected at construction\n    /*need_grads=*/true,\n    Initializers::xavier_normal,\n    Device::CPU\n);\n\nTensor out = embd(seq_ids);  // seq_ids: 1-D int32 Tensor of token IDs -> (seq_len, d_model)\n```\n\n`seq_ids`\n\nmust be a 1-D`int32`\n\ntensor - any other dtype or rank throws`std::invalid_argument`\n\n.- The embedding table itself (\n`m_embeddings`\n\n) must be`float32`\n\n(for now) - constructing with`Dtype::int32`\n\nthrows immediately. - Dispatches through\n`primitive_dispatcher()`\n\nto the correct backend (currently only CPU) implementation (`EmbeddingsImplAbstract`\n\n) based on the output tensor's dispatch key, keeping the op backend-agnostic at the call site. - When\n`need_grads`\n\nis true, a gradient node (`EmbeddingsGradsAbstract`\n\n) is automatically attached to the autograd graph after the forward pass. `all_embeddings()`\n\nexposes the underlying weight tensor directly (e.g. for weight tying with an output projection).`parameters()`\n\nreturns the embedding weight wrapped as a`Parameter`\n\n, named`embd.<n>.w`\n\n, for use with an optimizer.\n\n`SimpleTokenizer`\n\nimplements a from-scratch Byte Pair Encoding (BPE)\ntokenizer with GPT-2-style pre-tokenization - the same word-splitting\nregex GPT-2 uses (handling contractions, letter runs, digit runs, and\nwhitespace separately) before BPE merges are applied within each chunk.\n\n```\nSimpleTokenizer tok(SimpleTokenizer::Type::BPE);\ntok.on_file(\"corpus.txt\", /*max_vocab=*/8000);\n```\n\n- Starts from a base vocabulary of 3 special tokens (\n`<pad>`\n\n,`<bos>`\n\n,`<eos>`\n\n) plus all 256 raw byte values, then iteratively merges the most frequent adjacent token pair in the corpus until`max_vocab`\n\nis reached or no pair occurs more than once. `max_vocab`\n\nmust be at least 300 (`MIN_VOCAB_SIZE`\n\n) - lower throws`std::invalid_argument`\n\n.\n\n```\ntok.load(\"vocab.bin\");\n```\n\nVocabularies are stored in a simple binary format (`save`\n\n/`load`\n\n) - useful\nfor loading a pretrained vocabulary (e.g. reconstructed GPT-2 merge rules)\nwithout retraining from a corpus.\n\n``` php\nForge::Tensor ids = tok.encode(\"he went to the\");   // -> 1-D int32 Tensor of token IDs\nStringVec tokens  = tok.decode(ids);           // -> token strings, one per ID\n```\n\n`encode`\n\nreturns a`Forge::Tensor`\n\n(int32), ready to feed directly into`Embedding::operator()`\n\n.`decode`\n\nthrows`std::invalid_argument`\n\nif it encounters a token ID outside the loaded vocabulary.- An optional\n`transformation`\n\ncallback can be passed to`encode`\n\nto preprocess each pre-tokenized chunk (e.g. lowercasing) before BPE merges are applied.\n\nForge uses [reflect-cpp](https://github.com/getml/reflect-cpp) to automatically\ndiscover a model's trainable parameters from its struct layout, with no manual\nregistration required - any member exposing a `parameters()`\n\nmethod is picked\nup automatically, including nested submodules.\n\nAny aggregate struct whose members expose `parameters() -> std::vector<Parameter>`\n\n(e.g. `Linear`\n\n, `Embedding`\n\n, `LayerNorm`\n\n) is automatically reflectable:\n\n```\nstruct GPT2Block {\n    Forge::SelfAttention attn;\n    Forge::LayerNorm ln1;\n    Forge::Linear    mlp_fc;\n    Forge::LayerNorm ln2;\n};\n```\n\nNo base class, no manual parameter registration - `rfl::to_view`\n\nintrospects\nthe struct's members directly.\n\n```\nauto params = Forge::extract_parameters(model);   // flat std::vector<Parameter>\nForge::Adam optimizer(params, /*lr=*/...);\n```\n\nWalks every member satisfying the `HasParameters`\n\nconcept and collects their\n`Parameter`\n\ns into one flat list - this is what feeds the optimizer without\nyou having to hand-list every layer's weights.\n\n```\nForge::save(model, \"weights.safetensors\");\n```\n\nWrites an actual [safetensors](https://github.com/huggingface/safetensors)-format\nfile: an 8-byte header length, a JSON header describing each tensor's dtype,\nshape, and byte offset (padded to an 8-byte boundary), followed by the raw\ntensor data. Parameter names are built as `<member_name>.<param_name>`\n\n(recursing into nested submodules via `get_state_dict`\n\n), matching the naming\nconvention each layer's own `parameters()`\n\nalready establishes.\n\n```\nForge::load(model, \"weights.safetensors\");\n```\n\nReads the file, matches tensor names against the model's own state dict, and copies loaded data directly into each parameter in place. Throws if the number of tensors doesn't match the model's parameter count - a basic architecture-mismatch guard. This is the mechanism used to load real GPT-2 weights directly into Forge's GPT-2 implementation.\n\n`Forge::load_safetensors(filename)`\n\nis also available standalone, returning\na raw `std::map<std::string, Tensor>`\n\nwithout requiring a matching model\nstruct - useful for inspecting a checkpoint's contents directly.", "url": "https://wpnews.pro/news/built-gpt-2-on-custom-deep-learning-framework-i-built-from-scratch-in-c", "canonical_source": "https://github.com/muchlakshay/Forge", "published_at": "2026-08-17 22:04:46+00:00", "updated_at": "2026-08-17 22:11:08.988375+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "developer-tools"], "entities": ["Akshay Muchaklavya", "Forge", "GPT-2", "Hugging Face", "OpenBLAS", "CMake", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/built-gpt-2-on-custom-deep-learning-framework-i-built-from-scratch-in-c", "markdown": "https://wpnews.pro/news/built-gpt-2-on-custom-deep-learning-framework-i-built-from-scratch-in-c.md", "text": "https://wpnews.pro/news/built-gpt-2-on-custom-deep-learning-framework-i-built-from-scratch-in-c.txt", "jsonld": "https://wpnews.pro/news/built-gpt-2-on-custom-deep-learning-framework-i-built-from-scratch-in-c.jsonld"}}