{"slug": "tensor-is-the-might", "title": "Tensor is the might", "summary": "A developer is building a complete tensor library from scratch in C, inspired by Fabrice Bellard's libnc, to support neural network computations. The library abstracts tensors as flat arrays with shape and stride metadata, and includes reference counting for memory management. This work aims to provide an open-source foundation for accelerated tensor operations.", "body_md": "# Tensor is the might\n\nEvery good abstraction solves a problem, and this post will cover everything I know so far about a brilliant math abstraction - tensors.\n\nNeural networks, from a simple 2-layer [MLP](https://en.wikipedia.org/wiki/Multilayer_perceptron) to GPT-5, all boil down to the same thing: floating-point numbers flowing through a graph of operations. This post builds a complete, accelerated tensor library from scratch in C. It is heavily inspired by Bellard’s [libnc](https://bellard.org/libnc/), which unfortunately has not been open-sourced yet.\n\nA tensor is nothing but a flat array of numbers, plus some metadata telling you how to interpret those numbers as a multi-dimensional object. We all learned that 2D arrays can be better represented as 1D array plus a number of rows/columns - this is essentially what a tensor is.\n\nBut going beyond two dimensions - we might need some other metadata, such as a generalised shape:\n\n```\nfloat data[32 * 3 * 28 * 28]; // 32 images, 3 channels, 28x28 pixels\nint shape[4] = {32, 3, 28, 28}; // shape of the tensor\nint ndim = 4; // number of dimensions\n```\n\nHaving a shape we can figure out that an element at position `[n,c,h,w]`\n\nin a 4D tensor lives at offset `data[n*(3*28*28)+c*(28*28)+h*28+w]`\n\n, if we keep our tensor in a *row-major* C-order format (often the default in most tensor libraries today).\n\nNow, calculating each time an index of an element like this is inefficient, so we can precompute the *strides* once the shape is known. Strides tell how many elements to skip if we want to advance for one element in the given dimension. We could also group all shape-related fields together:\n\n```\nstruct ut_shape {\n  int ndim; // number of dimensions\n  int nelem; // number of elements\n  int shape[4]; // shape of the tensor\n  int strides[4]; // strides for each dimension\n};\n\nstruct ut_tensor {\n  struct ut_shape shape; // shape of the tensor\n  float *data; // pointer to the data\n  ... // more fields to come later\n};\n```\n\nWe can add some helpers to create a shape and get flat index from a multi-dimensional index:\n\n```\nut_shape ut_shape_new(int ndims, int* dims) {\n  ut_shape s = {.ndims = ndims, .nelems = 1};\n  for (int i = 0; i < ndim; i++) {\n    s.shape[i] = dims[i];\n    s.nelem *= dims[i];\n  }\n  return s;\n}\n\nint ut_index(ut_shape s, const int* idx) {\n  int flat = 0, stride = 1;\n  for (int i = s.ndim - 1; i >= 0; i--) {\n    flat += idx[i] * stride;\n    stride *= s.shape[i];\n  }\n  return flat;\n}\n\nut_shape s = ut_shape_new(3, (int[]){2, 3, 4});\nassert(s.nelem == 24);\nassert(s.ndim  == 3);\n// element [1][2][3] should be at offset 1*12 + 2*4 + 3 = 23\nassert(ut_index(s, (int[]){1, 2, 3}) == 23);\n```\n\nTensors are usually dynamically allocated, so we should provide a way to create and destroy them, nothing but a wrapper on top of `malloc`\n\n/`free`\n\n.\n\nSometimes we want to create a tensor that shares the same data with another tensor, for example to get a “view” of a part of a tensor, to transpose a tensor without copying data, or to modify its shape (flatten). In this case we need to track ownership of the data. It’d be also useful to “retain” tensors, so that they could outlive their original scope, like during iterative training to avoid the same allocation over and over. So, we add a reference counter field to the tensor struct:\n\n```\nstruct ut_tensor {\n  struct ut_shape shape; // shape of the tensor\n  float *data; // pointer to the data\n  int refcount; // reference count for shared ownership, free when reaches 0\n  struct ut_tensor *owner; // pointer to the owner tensor if this tensor is a view\n};\n```\n\nWe can optimise memory management further, adding arena allocator or memory pool to avoid frequent `malloc`\n\n/`free`\n\ncalls, but what we already have is a good start.\n\nHowever, our tensors are hardly useful without operations on them.\n\n## Elementwise\n\nThe most basic operations on tensors are elementwise – a single loop over all elements, applying a function to each element (unary) or a pair of elements (binary). We can implement them just like that:\n\n``` js\nstatic void ew_neg(float* out, const float* a, int n) {\n  for (int i = 0; i < n; i++) out[i] = -a[i];\n}\n\n// ...more unary ops...\n\nstatic void ew_relu(float* out, const float* a, int n) {\n  for (int i = 0; i < n; i++) out[i] = fmaxf(0.f, a[i]);\n}\nstatic void ew_add(float* out, const float* a, const float* b, int n) {\n  for (int i = 0; i < n; i++) out[i] = a[i] + b[i];\n}\n\n// ...more binary ops...\n\nstatic ut_tensor* ew_unary(ut_tensor* a, void (*fn)(float*, const float*, int)) {\n  ut_tensor* out = ut_alloc(a->shape.ndim, a->shape.shape);\n  fn(out->data, a->data, a->shape.nelem);\n  return out;\n}\nstatic ut_tensor* ew_binary(ut_tensor* a, ut_tensor* b,\n                            void (*fn)(float*, const float*, const float*, int)) {\n  ut_tensor* out = ut_alloc(a->shape.ndim, a->shape.shape);\n  fn(out->data, a->data, b->data, a->shape.nelem);\n  return out;\n}\n\nut_tensor* ut_neg(ut_tensor* a) { return ew_unary(a, ew_neg); }\nut_tensor* ut_exp(ut_tensor* a) { return ew_unary(a, ew_exp); }\nut_tensor* ut_sigmoid(ut_tensor* a) { return ew_unary(a, ew_sigmoid); }\nut_tensor* ut_tanh(ut_tensor* a) { return ew_unary(a, ew_tanh); }\nut_tensor* ut_relu(ut_tensor* a) { return ew_unary(a, ew_relu); }\nut_tensor* ut_add(ut_tensor* a, ut_tensor* b) { return ew_binary(a, b, ew_add); }\nut_tensor* ut_sub(ut_tensor* a, ut_tensor* b) { return ew_binary(a, b, ew_sub); }\nut_tensor* ut_mul(ut_tensor* a, ut_tensor* b) { return ew_binary(a, b, ew_mul); }\nut_tensor* ut_div(ut_tensor* a, ut_tensor* b) { return ew_binary(a, b, ew_div); }\nut_tensor* ut_scale(ut_tensor* a, float s) {\n  ut_tensor* out = ut_alloc(a->shape.ndim, a->shape.shape);\n  for (int i = 0; i < a->shape.nelem; i++) out->data[i] = a->data[i] * s;\n  return out;\n}\n```\n\nIt’d be nice to `assert()`\n\nthat the shapes of two tensors are the same before doing a binary operation. Alternatively, at this point we might decide that we want to support “broadcasting” – extending the smaller tensor to match the shape of the larger tensor. For example a tensor of shape `[2, 3, 4]`\n\nand a tensor of shape `[3, 4]`\n\ncan be added together by “stretching” the second tensor along the first dimension. While being a common feature in many tensor libraries, I decided to leave it out for now. Most models I’m aiming for have their underlying tensors perfectly aligned. If not - we can either duplicate data explicitly aligning tensor shapes, or adding broadcasting index calculation later.\n\nIt might be tempting at this point to implement more operations, matrix multiplication, convolutions, etc. But this is the moment where we have to ask – do we want all operations to run on CPU?\n\n## Golden Processing Unit (GPU)\n\nLooking at modern GPU prices, it is clear that they probably have a significant value when it comes to crunching arrays of numbers. Thus, instead of limiting our library to a CPU, we might consider offloading some operations to a GPU.\n\nThis is where things get interesting, because we now need to manage memory across both CPU and GPU, transfer data efficiently, learn how to write GPU kernels, and much more. To make things worse, there is no single “GPU accelerator” – there are many vendors, each with their own APIs and quirks: CUDA, OpenCL, WebGPU, Metal, Vulkan, etc.\n\nI tried many times to get the most out of CPU-only tensors, but even with BLAS, LAPACK, OpenMP – I could not reach the same magnitude of performance as GPU-accelerated frameworks.\n\nTo keep things manageable, I decided to start with Metal. On the one hand it limits us to Apple devices, but on the other hand it’s a fairly modern API and simplifies things due to “unified memory” of Apple Silicon.\n\nMetal uses its own language to define GPU kernels (MSL), which is similar to modern C++, and it compiles kernels at runtime, so we can define them as strings in our C code:\n\n``` js\nstatic const char *shaderSource =\n    \"#include <metal_stdlib>\\n\"\n    \"using namespace metal;\\n\"\n    \"\\n\"\n    \"kernel void relu(device const float *in  [[buffer(0)]],\\n\"\n    \"                  device float       *out [[buffer(1)]],\\n\"\n    \"                  uint id [[thread_position_in_grid]]) {\\n\"\n    \"    float val = in[id];\\n\"\n    \"    out[id] = fmax(val, 0.0f);\\n\"\n    \"}\\n\";\n```\n\nTo make this kernel run on GPU we need to create a device, build a command queue (GPUs are asynchronous), compile the kernel, create buffers for input and output, and finally dispatch the kernel to run on GPU. Here’s how a simple GPU-accelerated ReLU operation might look like:\n\n```\n// prepare device, queue, and compile kernel\nid<MTLDevice> device = MTLCreateSystemDefaultDevice();\nid<MTLCommandQueue> queue = [device newCommandQueue];\nNSError *err = nil;\nid<MTLLibrary> library = [device newLibraryWithSource:[NSString stringWithUTF8String:shaderSource] options:nil error:&err];\nid<MTLFunction> reluFn = [library newFunctionWithName:@\"relu\"];\nid<MTLComputePipelineState> pipeline = [device newComputePipelineStateWithFunction:reluFn error:&err];\n\n// prepare data, dispatch kernel\nfloat input[8]  = {-2.0, -1.0, 0.0, 1.0, 2.0, -0.5, 3.0, -3.0};\nfloat output[8] = {0};\nid<MTLBuffer> bufIn = [device newBufferWithBytes:input length:sizeof(input) options:MTLResourceStorageModeShared];\nid<MTLBuffer> bufOut = [device newBufferWithLength:sizeof(output) options:MTLResourceStorageModeShared];\nid<MTLCommandBuffer> cmdBuf = [queue commandBuffer];\nid<MTLComputeCommandEncoder> enc = [cmdBuf computeCommandEncoder];\n[enc setComputePipelineState:pipeline];\n[enc setBuffer:bufIn  offset:0 atIndex:0];\n[enc setBuffer:bufOut offset:0 atIndex:1];\nMTLSize gridSize = MTLSizeMake(8, 1, 1);\nMTLSize tgSize  = MTLSizeMake(pipeline.maxTotalThreadsPerThreadgroup, 1, 1);\n[enc dispatchThreads:gridSize threadsPerThreadgroup:tgSize];\n[enc endEncoding];\n[cmdBuf commit];\n[cmdBuf waitUntilCompleted];\n// copy results back\nmemcpy(output, bufOut.contents, sizeof(output));\n```\n\nWe can wrap the first part of this code into a “Metal context” struct, that is created once and is kept alive for the lifetime of the whole library. Then we can add helpers to allocate buffers, read and write them and dispatch kernels. While the official Metal API is in Objective-C or Swift, we can use low-level ObjC [runtime functions](https://developer.apple.com/documentation/objectivec/objective-c-runtime) to call it from C, to keep the library pure (`objC_msgSend`\n\nall over the code).\n\nNow every operation on tensors has to be implemented twice – as a naïve CPU implementation, and as a GPU kernel. Depending on the origin of the tensor we would call one of the implementations. Note that this also means that tensors carry the “device” field, and we must specify the device when the tensor is allocated – CPU tensors would only have a normal data buffer, but Metal tensors would also have a pointer to a Metal buffer object.\n\nIt’s also becoming important that we need to track when GPU and CPU buffers diverge. An easy way is to keep “dirty” flags for CPU/GPU buffers and provide “sync” functions to copy data from one buffer to another.\n\nEven though some operations can or should be done on a GPU, in some cases to simplify things we still create it on a CPU first, then do the processing and send/copy data back to GPU.\n\nAll in all, we’d need to implement the following:\n\n``` js\nstatic const char *_mtl_src = \"...\"; // MSL source code for all kernels\nvoid *ut_mtl_init(); // create Metal context as a singleton\nvoid ut_mtl_dispatch(void *ctx, char *kernel, void **bufs, int nbufs, void *bytes, int blen, int n); // dispatch kernel with buffers\nvoid *ut_mtl_buf_alloc(void *ctx, void *data, int len); // allocate Metal buffer\nvoid ut_mtl_buf_free(void *ctx, void *buf); // free Metal buffer\nvoid ut_mtl_buf_read(void *ctx, void *buf, void *data, int len); // read Metal buffer to CPU\nvoid ut_mtl_buf_write(void *ctx, void *buf, void *data, int len); // write CPU buffer to Metal\n\nvoid ut_sync_cpu(ut_tensor *t); // sync CPU buffer from GPU\nvoid ut_sync_gpu(ut_tensor *t); // sync GPU buffer from CPU\nvoid ut_to_device(ut_tensor *t, ut_dev device); // move tensor to specified device (CPU/GPU)\n```\n\nDone with element-wise ops and their GPU kernel equivalents we can move on to the core of all neural networks - matrix multiplication.\n\n## Matmul\n\nA made-up on the spot statistics tell me that 80% of FLOPs in any neural network are matrix multiplications. Getting them right and fast is crucial for any deep learning framework. There are many ways to implement matrix multiplication, the most efficient one depends on the size of the matrices, the hardware, and the memory layout.\n\nFirst, let’s consider the simplest case of multiplying two 2D matrices A and B, where A is of shape `[M, K]`\n\nand B is of shape `[K, N]`\n\n. The result C will be of shape `[M, N]`\n\n. The naïve implementation could look like this:\n\n``` js\nvoid gemm(const float *A, const float *B, float *C,\n          int m, int n, int k, bool ta, bool tb) {\n    for (int i = 0; i < m; i++)\n        for (int j = 0; j < n; j++) {\n            float sum = 0.f;\n            for (int l = 0; l < k; l++) {\n                float a = ta ? A[l*m + i] : A[i*k + l];\n                float b = tb ? B[j*k + l] : B[l*n + j];\n                sum += a * b;\n            }\n            C[i*n + j] = sum;\n        }\n}\n```\n\nNeedless to say this would be terribly slow - three nested loops and no cache optimisation. On Apple Silicon we can use a direct replacement, `cblas_sgemm`\n\nfrom the Accelerate framework. It uses AMX (matrix co-processor) and happens to be 50..100x faster for any matrix above 32x32:\n\n```\ncblas_sgemm(CblasRowMajor,\n            ta ? CblasTrans : CblasNoTrans,\n            tb ? CblasTrans : CblasNoTrans,\n            m, n, k, 1.f,\n            A, ta ? m : k,\n            B, tb ? k : n,\n            0.f, C, n);\n```\n\nBut we can go even further and use Metal with GPU matmul for large matrices. Metal comes with `MPSMatrix`\n\nclass and a few methods to do the multiplication. But it’s only a 2D matrices, so we’d have to implement our own kernel to handle more than 2D tensors.\n\n``` js\nkernel void bmatmul(device\n  const float * a, device\n  const float * b_, device float * c,\n    constant int * p, uint idx[[thread_position_in_grid]]) {\n  int M = p[1], N = p[2], K = p[3];\n  int tot = p[0] * M * N;\n  if ((int) idx >= tot) return;\n  int n = (int) idx % N, t = (int) idx / N, m = t % M, bat = t / M;\n  float s = 0;\n  int ao = bat * M * K + m * K, bo = bat * K * N + n;\n  for (int k = 0; k < K; k++) s += a[ao + k] * b_[bo + k * N];\n  c[idx] = s;\n}\n```\n\nThe overall `ut_matmul`\n\nfunction would now look like this:\n\n```\nut_tensor *ut_matmul(ut_tensor *a, ut_tensor *b) {\n    // special case: 2D x 2D\n    if (a->shape.ndim == 2 && b->shape.ndim == 2) {\n        int n = ..., m = ..., k = ...; // dimensions: a: [m, k], b: [k, n]\n        if (a->dev == UT_METAL || b->dev == UT_METAL) {\n            // GPU matmul\n            ut_tensor *out = ut_alloc(2, (int[]){m, n}, UT_METAL);\n            ut_mtl_matmul(a, b, out);\n            return out;\n        } else {\n            // CPU matmul\n            ut_tensor *out = ut_alloc(2, (int[]){m, n}, UT_CPU);\n            cblas_sgemm(CblasRowMajor,\n                        CblasNoTrans, CblasNoTrans,\n                        m, n, k, 1.f,\n                        a->data, k,\n                        b->data, n,\n                        0.f, out->data, n);\n            return out;\n        }\n    }\n    // special case: 3D x 3D (batches)\n    if (a->shape.ndim == 3 && b->shape.ndim == 3) {\n        int B, n, m, k; // dimensions: a: [B, m, k], b: [B, k, n]\n        // similarly, for Metal: call a custom kernel, for CPU - sgemm in a loop for each batch\n    }\n}\n```\n\nOne thing missing is matmul for transposed matrices. Earlier we found a way to “transpose” a matrix without copying the data (only updating the strides), but for multiplication we should handle it in a special way, too. Fortunately, all implementations seem to support transposition with little code changes (`ta`\n\nand `tb`\n\nflags in `cblas_sgemm`\n\n, and a few extra lines in the Metal kernel).\n\n## Autograd by hand\n\nAt this point we can perform arithmetics on tensors and adding new operations becomes mostly straightforward. It’s a good foundation for building neural networks, but we are still missing one crucial feature – proper differentiation.\n\nWhen a network “learns” it adjusts its parameters to minimize a loss function. To do this, we need to compute the gradients of the loss function with respect to the parameters. This is where automatic differentiation usually comes in.\n\nIn “big” frameworks like PyTorch or Tensorflow a differentiation engine records a computation graph during the forward pass, and then walks it in reverse to compute the gradients. I’m going to do a simpler thing, manual backpropagation. Every layer of a network becomes a struct with “forward” and “backward” functions and gradients are pre-allocated tensors of the same shape as each trainable parameter.\n\nThe forward pass takes an input tensor and produces an output tensor. The backward pass takes the gradient of the loss with respect to the output tensor, and computes the gradient of the loss with respect to the input tensor and the parameters. It may not be as user-friendly as autograd, but for most well-known network architectures it’s easier to implement in C.\n\nLet’s start with a Linear layer. It consists of a weights matrix NxM (where N is number of input features and M is number of output features), and optionally a bias vector:\n\n```\ntypedef struct ut_linear {\n  ut_tensor* weight;  // [in, out]\n  ut_tensor* bias;    // [out]\n  int nin, nout;\n  bool has_bias;\n} ut_linear;\n```\n\nForward pass is quite simple: a matrix multiplication followed by adding a bias vector (later we might consider fusing these two operations into one to speed things up a little):\n\n```\nut_tensor *ut_linear_forward(ut_linear *l, ut_tensor *x) {\n    ut_tensor *out = ut_matmul(x, l->weight);   // [B, in] @ [in, out]\n    if (l->bias)\n        for (int b = 0; b < out->shape.shape[0]; b++)\n            for (int j = 0; j < l->nout; j++)\n                out->data[b * l->nout + j] += l->bias->data[j];\n    return out;\n}\n```\n\nIn practice the forward pass would be a more verbose due to metal/non-metal branches, but the logic remains simple. The backward pass is a bit harder, as we need to compute gradients with respect to the input and the parameters.\n\nA great explanation of matmul backpropagation can be found in [this article](https://robotchinwag.com/posts/linear-layer-deriving-the-gradient-for-the-backward-pass/). For my mathematically deprived brain it was easier to read how pytorch implemented it:\n\n```\n# forward(x: Tensor, w: Tensor): Tensor\ny = x@w\n\n# backward(dldy: Tensor): Tensor\ndldx = dldy @ w.T\ndldw = x.T @ dldy\n```\n\nIf we include bias, then `y=x@w+b`\n\nand `dldb`\n\nis just `dldy`\n\n. Bias is a plus operation, so gradient is kept as-is. In case of mini-batch training we should sum output gradients across the batch dimension and that would be our bias gradient.\n\n```\nut_tensor *ut_linear_backward(ut_linear *l,\n                                ut_tensor *x,\n                                ut_tensor *grad_out,\n                                ut_tensor *dW, ut_tensor *db) {\n    // dW += x.T @ grad_out\n    ut_tensor *xT    = ut_transpose(x, 0, 1);\n    ut_tensor *dW_b  = ut_matmul(xT, grad_out);\n    for (int i = 0; i < dW->shape.nelem; i++) dW->data[i] += dW_b->data[i];\n\n    // db += colsum(grad_out)\n    for (int b = 0; b < grad_out->shape.shape[0]; b++)\n        for (int j = 0; j < l->nout; j++)\n            db->data[j] += grad_out->data[b * l->nout + j];\n\n    // dx = grad_out @ W.T \n    ut_tensor *WT = ut_transpose(l->weight, 0, 1);\n    ut_tensor *dx = ut_matmul(grad_out, WT);\n\n    ut_free(xT); ut_free(dW_b); ut_free(WT);\n    return dx;\n}\n```\n\nAnd this is where things can be optimised further. In backward pass we do a materialising transposition of the weight matrix, essentially copying the whole tensor. Even with GPU acceleration it’s still not optimal. We can implement a special case of matmul that takes a transposed matrix as input, and avoid the extra allocation and copy, `ut_matmul_t`\n\n.\n\nFinally, we are at the point where we can implement and train a simple fully-connected neural network.\n\n## Loss\n\nTo learn something you should be able to tell right from wrong. This is where loss functions come into play. They measure how far off the predictions are from the actual labels. For classification tasks a common loss function is cross-entropy, so that’s where we start from.\n\nA network tries to predict the probability distribution of classes for each input sample. The cross-entropy loss measures the difference between the predicted distribution and the true distribution (usually a one-hot encoded vector).\n\nA loss can be calculated like this:\n\n``` js\nfloat cross_entropy_loss(ut_tensor *logits, const int *labels, int B) {\n    float total = 0.f;\n    int C = logits->shape.shape[1];\n    for (int b = 0; b < B; b++) {\n        float *row = logits->data + b * C;\n        float mx = row[0];\n        for (int j = 1; j < C; j++) if (row[j] > mx) mx = row[j];\n        float sum = 0.f;\n        for (int j = 0; j < C; j++) sum += expf(row[j] - mx);\n        total += logf(sum) + mx - row[labels[b]];\n    }\n    return total / (float)B;\n}\n```\n\nWe subtract `max(row)`\n\nbefore exp() to avoid overflow, keeping every exponential below 1, preventing cases like `exp(89)`\n\n, which easily overflows float32 range.\n\nSo, the result is a number, the lower it is - the better the network is trained. But we also should use this value to compute the gradient of the loss with respect to the logits, which will be then used in backpropagation:\n\n```\n// Backward: d_logits[b][j] = softmax[b][j] - (j == labels[b])\nut_tensor *cross_entropy_backward(ut_tensor *logits,\n                                  const int *labels, int B) {\n    int C = logits->shape.shape[1];\n    ut_tensor *d = ut_like(logits);\n    for (int b = 0; b < B; b++) {\n        float *row  = logits->data + b * C;\n        float *drow = d->data      + b * C;\n        float mx = row[0];\n        for (int j = 1; j < C; j++) if (row[j] > mx) mx = row[j];\n        float sum = 0.f;\n        for (int j = 0; j < C; j++) sum += expf(row[j] - mx);\n        float ls = logf(sum) + mx;\n        for (int j = 0; j < C; j++) drow[j] = expf(row[j] - ls) / (float)B;\n        drow[labels[b]] -= 1.f / (float)B;\n    }\n    return d;\n}\n```\n\nWhile this is working, we can go further and implement a “fused” version of cross-entropy loss - the one that calculates the loss and the gradient in a single pass, which is more efficient. Fused operations is a common optimisation technique in deep learning frameworks, and a proper tensor library uses them a lot.\n\n## Optimiser\n\nSpeaking of optimisations, we need to implement an optimiser to update the model parameters based on the gradients computed during backpropagation.\n\nThe gradients computed during backpropagation tell us which direction makes the loss *worse*. To improve the model we move parameters into the opposite direction, scaled by some “learning rate” factor. So, the gradients describe which how impactful the parameters are on the less and which way to move them to make loss smaller, while the optimiser decides how to actually update each parameter: adjusting them too little will take longer to train, adjusting too much might miss the “sweet spot” and make the network learning unstable.\n\nThe simplest optimiser rule is stochastic gradient descent (SGD): adjust each parameter by a gradient multiplied by a small number (learning rate).\n\nWe can slightly improve it by adding a “momentum”. Instead of following the gradient directly, we accumulate a velocity vector that takes into account the previous updates. This smoothes out oscillations and accelerates through flat regions.\n\nAdditionally, we can use gradient clipping: if any gradient component exceeds a certain threshold – we clamp it. This prevents a few bad input samples from destroying the rest of our training.\n\n```\nvoid ut_sgd_step(ut_sgd* o, float clip) {\n for each parameter p, gradient g:\n   g = clamp(g, -clip, clip)\n   if momentum > 0: v = mom*v - lr*g;  p += v\n   else:            p -= lr*g\n   zero gradient for next iteration\n}\n```\n\nThere are better optimisers, such as Adam, but SGD is probably the simplest to implement first.\n\n## MNIST\n\nMNIST dataset is the “hello world” of deep learning. It consists of 60,000 training images and 10,000 test images of handwritten digits (0-9), each image is 28x28 pixels in grayscale. The task is to classify each image into one of the 10 digit classes.\n\nTime to test our tensor library on a practical task!\n\nData format of the dataset is trivial - raw bytes for images and separate single-byte labels. For training we’ll need an array of shuffled indices to use random batches. Otherwise, it’s pretty straightforward:\n\n``` php\n// Model: 784 -> 128 -> ReLU -> 10 (known to have good results)\nut_linear fc1 = ut_linear_alloc(784, 128, true, UT_CPU);\nut_linear fc2 = ut_linear_alloc(128, 10, true, UT_CPU);\n\n// Optimiser: SGD + momentum\nut_tensor* params[] = {fc1.weight, fc1.bias, fc2.weight, fc2.bias};\nut_sgd opt = ut_sgd_alloc(params, 4, 0.01f, 0.9f);\n\n// Batch of 64, 15 epochs, 60k / 64 = 937 batches\nint B = 64, epochs = 15, batches = train.n / B;\nut_tensor* grad_logits = ut_alloc(2, (int[]){B, 10}, UT_CPU);\nint* idx = malloc((size_t)train.n * sizeof(int));\nfor (int i = 0; i < train.n; i++) idx[i] = i;\n\nfor (int ep = 0; ep < epochs; ep++) {\n  shuffle(idx, train.n);\n  for (int bi = 0; bi < batches; bi++) {\n    // pack batch from shuffled indices\n    float bx[B * 784]; int bl[B];\n    for (int i = 0; i < B; i++) {\n      int ii = idx[bi * B + i];\n      memcpy(bx + i * 784, train.imgs + ii * 784, 784 * sizeof(float));\n      bl[i] = train.labels[ii];\n    }\n    ut_tensor* x = ut_from_data(2, (int[]){B, 784}, bx, UT_CPU);\n\n    // forward\n    ut_tensor *h1 = ut_linear_forward(&fc1, x);\n    ut_tensor *h1r = ut_relu(h1);\n    ut_tensor *logits = ut_linear_forward(&fc2, h1r);\n    // loss + grad\n    float loss = ut_cross_entropy(logits, bl, grad_logits);\n    // backward\n    ut_tensor* dh1r = ut_linear_backward(&fc2, h1r, grad_logits, opt.grads[2], opt.grads[3]);\n    ut_tensor* dh1 = ut_relu_backward(dh1r, h1);\n    ut_linear_backward(&fc1, x, dh1, opt.grads[0], opt.grads[1]);\n    ut_sgd_step(&opt, 5.0f);\n    ut_free(x); ut_free(h1); ut_free(h1r); ut_free(logits); ut_free(dh1r); ut_free(dh1);\n  }\n}\n```\n\nRunning on a rusty MacBook M1, I get 0.8 seconds per epoch, and within 5 epochs it reaches 99% accuracy on train set and ~97% on test set. For a small dataset like this there is no difference between CPU and GPU in terms of performance, but on larger models GPU would pay off.\n\nFor comparison: PyTorch achieves same accuracy, but takes almost 2 seconds per epoch. On other, larger models, I noticed that C is usually 2-4x faster than PyTorch, especially if most of the operations get fused properly.\n\n## What’s next?\n\nThis is just the bare minimum. For a proper tensor library we’ll need many more layers and operations, but adding them is a trivial (although, boring) process: add CPU implementation, add GPU kernel, make a dispatch wrapper (either call unary/binary dispatch, or a custom one), add backward pass.\n\nFor example, adding a `gelu`\n\n(or any other activation layer) this is roughly 20 lines of code.\n\nThe primary goal for me was to build some ground layer to train tinyML/edgeAI models on MacBook, and so far I like it. The [full library](https://github.com/zserge/utensil) has convolutional layers (for image processing), pooling, batch normalization, dropout, and a few more optimisers and loss functions. It can load/store tensors into raw binary files and performs well on toy datasets, like CIFAR, HAR, speech commands, etc.\n\nI’m looking into adding an OpenCL backend to support more devices with a few more optimisations.\n\nI even managed to train a tiny GPT2-like transformer, just like minGPT.\n\nWhat’s missing? I definitely miss autograd, manual backpropagation can be tiresome. Maybe I’ll add it one day, since we have all backward functions implemented – I could add a “tape” to store all operations and some topology sorting to compute gradients in the right order. But for now, I can live without it.\n\nAlso a deliberate limitation of being a single-header, header-only tensor library made me sacrifice support of CUDA, which requires kernels to be stored in `.cu`\n\nfiles and compiled with `nvcc`\n\n(unless there is some other way, I’m not aware of?). Maybe WebGPU acceleration would be possible, I haven’t looked into it yet, but it’s asynchronous nature and rather immature state makes me doubt.\n\nI like the choice of C, and that the user has full control over all memory operations and optimisations. I’ve also experimented a bit with bindings to other languages. C++ was easy - simply wrap all C APIs into classes and the code suddenly becomes much shorter and more readable due to operator overloading. I tried Go bindings, but CGo overhead makes it a bit harder to justify.\n\nThis isn’t a competitor to PyTorch. It’s a half-serious anti-PyTorch, small enough to skim through and understand, yet practical enough to build and train various models.\n\nThe full library is available at [https://github.com/zserge/utensil](https://github.com/zserge/utensil) and if this long post made you curious - clone the repo, add a layer, train a model, break and debug it, any contributions and feedback are welcome!\n\nI hope you’ve enjoyed this article. You can follow – and contribute to – on [Github](https://github.com/zserge), [Mastodon](https://mastodon.social/@zserge), [Twitter](https://twitter.com/zsergo) or subscribe via [rss](/rss.xml).\n\n*Jul 14, 2026*\n\nSee also:\n[AI or ain't: Neural Networks](/posts/ai-nn/) and [more](/posts/).", "url": "https://wpnews.pro/news/tensor-is-the-might", "canonical_source": "https://zserge.com/posts/tensor/", "published_at": "2026-07-14 00:00:00+00:00", "updated_at": "2026-07-14 09:36:33.818108+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "neural-networks", "developer-tools"], "entities": ["Fabrice Bellard", "libnc"], "alternates": {"html": "https://wpnews.pro/news/tensor-is-the-might", "markdown": "https://wpnews.pro/news/tensor-is-the-might.md", "text": "https://wpnews.pro/news/tensor-is-the-might.txt", "jsonld": "https://wpnews.pro/news/tensor-is-the-might.jsonld"}}