{"slug": "deep-learning-from-scratch-in-1400-lines-neve-frost-framework", "title": "Deep Learning from Scratch in 1400 Lines - Neve & Frost Framework", "summary": "A developer has released Frost, a deep learning framework written in roughly 1,400 lines of code, built on top of the Neve programming language. The framework includes parallel dataloaders and GPU kernels, and ships with a ResNet-18 benchmark that users can run themselves. The developer says the project targets three pain points in existing tooling: optimizer complexity in PyTorch, difficult C++/CUDA interoperability, and limited parallelism.", "body_md": "The title literally means: \"I have parallel dataloaders, GPU kernels and a high-performance computing programming language all expressed in a framework with 1400 lines of code\".\n\nYou may run the ResNet-18 benchmark yourself\n\n([github.com/NoSavedDATA/Neve_benchmarks](https://github.com/NoSavedDATA/Neve_benchmarks))\n\nI released the Neve programming language a while ago. Now, this is the release of the Frost deep learning framework, alongside with the first benchmark.\n\nOther results for Neve:\n\nCurrently working in a better GPU programming interface, towards the implementation of flash-attention.\n\nNeve documentation ([neve-lang.dev](https://neve-lang.dev))\n\nNeve repo ([github.com/NoSavedDATA/Neve](https://github.com/NoSavedDATA/Neve)).\n\nYoutube for updates ([youtube.com/@nosaveddata3994](https://www.youtube.com/@nosaveddata3994)).\n\nDiscord for extensive talks/suggestions ([discord.gg/hP5feM7cV](https://discord.gg/hP5feM7cV))\n\n────────────────────────────────────────\n\nOnce day I was reading some papers, and a very interesting paper was published. It was the sophia optimizer. I took a glance in an unnoficial ([code](https://github.com/kyegomez/Sophia/blob/main/Sophia/main.py)) for it, and I questioned myself why did it have to be so difficult to add new optimizers in PyTorch. I experimented the optimizer, and the results were quite bad with a lot of NaNs. Turns out another paper published later claimed this and other optimizers had overstated claims.\n\nImagine wasting hours studying a 10 pages of a paper, then hardly trying to debug it and asses whether other person discoveries are true. All that code reading complexity makes this a challengeful task.\n\nProblem 1: even optmizers are hard to understand in PyTorch.\n\nFew weeks later, ([flash attention](https://arxiv.org/pdf/2205.14135)) was released, and the algorithm actually achieved a speed-up of 2x. The problem, it was C++ CUDA. Most high-level GPU kernel frameworks were imature to the point the flash attention author chose not to use them.\n\nNow take a look what is necessary for adding C++ code in PyTorch\n\n``` python\nfrom setuptools import setup, Extension\nfrom torch.utils import cpp_extension\n\nsetup(name=\"extension_cpp\",\n      ext_modules=[\n          cpp_extension.CppExtension(\n              \"extension_cpp\",\n              [\"muladd.cpp\"],\n              extra_compile_args={\n                  \"cxx\": [\n                      # define Py_LIMITED_API with min version 3.9 to expose only the stable\n                      # limited API subset from Python.h\n                      \"-DPy_LIMITED_API=0x03090000\",\n                      # define TORCH_TARGET_VERSION with min version 2.10 to expose only the\n                      # stable API subset from torch\n                      \"-DTORCH_TARGET_VERSION=0x020a000000000000\",\n                  ]\n              },\n              py_limited_api=True)],  # Build 1 wheel across multiple Python versions\n      cmdclass={'build_ext': cpp_extension.BuildExtension},\n      options={\"bdist_wheel\": {\"py_limited_api\": \"cp39\"}}  # 3.9 is minimum supported Python version\n)\n```\n\nThat comprehends problem 2: lack of high-level CUDA code and hard interoperability.\n\nFor my Bachelor's thesis, I implemented the ([BBF](https://github.com/NoSavedDATA/PyTorch-BBF-Bigger-Better-Faster-Atari-100k)) Reinforcement Learning for Atary. A bit before that, I took a glance code of the ([Efficient Zero](https://github.com/YeWR/EfficientZero/tree/main/core)) reinforcement learning model. It has a parallelism that PyTorch does not handle, and the implementation required using Cython packages for having threads (literaly coding in C, then just calling C functions from Python). Later, I realized PyTorch also needed to implement its data worker threads in C, another workaround over Python Global Interpreter Lock (GIL). Not only that, even preprocessing implementations like the BPE are made in C, C++, Rust, etc...\n\nThat leads us problem 3, lack of parallelism. That is when I decided to create a programming language, a few months before finishing my Bachelor's, which matured to my Master's project\n\nSumming up, currently, people must choose between languages like Python for high-level productivity, C and relatives for compute efficiency, Lua/Julia for advanced interoperability and other languages for concurrency. Thus, since in my job I had to wait hours for my neural networks to train, I decided to create a programming language in the time in between trainings. One language that had all these features, which are of high value for deep learning research. Nowadays, I believe it matured to such a point that it may be extended to other complex problem domains.\n\nJulia makes dynamic typying speed reach close to C++ speeds. It also has a mark sweep and channels for parallelism. The idea is very interesting. Let's take a look a in its cuda kernels.\n\n```\nfunction mma_kernel!(Z::CuDeviceMatrix{Float32}, \n                     X::CuDeviceMatrix{BFloat16}, \n                     Y::CuDeviceMatrix{BFloat16})\n\n    # Grid and block indices\n    bx = blockIdx().x\n    by = blockIdx().y\n\n    # Thread and warp indices\n    tid  = threadIdx().x\n    lane = (tid - 1) % 32\n    warp = (tid - 1) ÷ 32\n```\n\n**STOP!! Why am I seeing blockIdx().x in my code? Was this supposed to be a high-level scientific language or CUDA in C++28?**\n\nBesides, it does not expose intrisics like the cp_async, which is crucial for high-speed matrix multiplication. They must be explicitly added throgh interop intrisics. And it has the \"end\" keyword, which in my opinion incurs a lot of code pollution.\n\nMojo has a Python interop, so it did not have to build all libs and frameworks from scratch +1 point. It has ([Byte-Pair Encoding benchmarks](https://github.com/atsentia/mojo-tokenizer))!! +1. It is only the BPE inference, no traning -1 point. It has ([flash-attention gpu kernels](https://www.spheron.network/blog/modular-max-mojo-gpu-cloud-llm-inference/)) +1 point.\n\nIt runs MAX, which allows GPU code portability across different hardware, +2 points.\n\nIt lacks channels, so I would hardly try to make a parallel dataloader in it. -2 points. It uses Rust ownership +0 points.\n\nNow let's look at Mojo kernels for the flash attention.\n\n```\n@always_inline\ndef fused_attention_cpu[\n    BN: Int, BD: Int\n](\n    Q: LayoutTensor,\n    K: LayoutTensor,\n    V: LayoutTensor,\n    O: LayoutTensor[mut=True, ...],\n):\n    comptime N = K.shape[0]()\n    comptime D = K.shape[1]()\n\n    comptime for tile_n in range(N // BN):\n        var Q_tile = Q.tile[BN, D](tile_n, 0)\n\n        comptime for tile_d in range(D // BD):\n            var m_1 = (\n                LayoutTensor[Q_tile.dtype, Layout(BN, 1), MutAnyOrigin]\n                .stack_allocation()\n                .fill(Scalar[Q_tile.dtype].MIN)\n            )\n\n            var l_1 = (\n                LayoutTensor[Q_tile.dtype, Layout(BN, 1), MutAnyOrigin]\n                .stack_allocation()\n                .fill(0)\n            )\n\n            var O_i = (\n                LayoutTensor[\n                    Q_tile.dtype, Layout.row_major(BN, BD), MutAnyOrigin\n                ]\n                .stack_allocation()\n                .fill(0)\n            )\n\n            comptime for tile_n_idx in range(N // BN):\n                var K_tile = K.tile[BN, D](tile_n_idx, 0)\n                var V_tile = V.tile[BN, BD](tile_n_idx, tile_d)\n\n                var S = matmul_b_transpose(Q_tile, K_tile)\n                var m_2 = max(m_1, rebind[type_of(m_1)](max[axis=1](S)))\n```\n\nQuite interesting. It has layouts and tiling, inspired by CuTe and Cutlass. It actually inspired the way Neve layouts and tiles work. Nevertheless, it still has a heavy syntax. Note the keyword comptime appears frequently (a sort of metaprogramming). This adds some cognitive overhead. And the layouts are quite verbose. The layout fill() and stack_allocation() can be simplified.\n\nTriton has layouts/tiling similar to Mojo, but is dynamically typed and has no comptime headaches. The problem is that Triton does not make Python Dataloaders easier to implement from the systems programming language perspective. We actually need a complete new programming language for this.\n\n``` python\n@triton.jit\ndef _attn_fwd_inner(\n[...]\nK_block_ptr = tl.advance(K_block_ptr, (0, lo))\n    V_block_ptr = tl.advance(V_block_ptr, (lo, 0))\n\n    # loop over k, v and update accumulator\n    for start_kv in range(lo, hi, BLOCK_SIZE_KV):\n        # Just let the compiler know that start_n is a multiple of BLOCK_N, so the compiler can do optimizations\n        start_kv = tl.multiple_of(start_kv, BLOCK_SIZE_KV)\n\n        # -- compute qk ----\n        K_block = tl.load(K_block_ptr)\n        QK_block = tl.dot(Q_block, K_block)\n\n        if STAGE == 2:\n            mask = offs_q[:, None] >= (start_kv + offs_kv[None, :])\n[...]\n    # A LAYOUT\n    Q_block_ptr = tl.make_block_ptr(\n        base=Q + qvk_offset,\n        shape=(SEQ_LEN, HEAD_DIM),\n        strides=(stride_Q_seq, stride_Q_dim),\n        offsets=(block_index_q * BLOCK_SIZE_Q, 0),\n        block_shape=(BLOCK_SIZE_Q, HEAD_DIM),\n        order=(1, 0),\n    )\n[...]\n        # Algebra\n        # -- compute qk ----\n        K_block = tl.load(K_block_ptr)\n        QK_block = tl.dot(Q_block, K_block)\n```\n\nLet's see how Neve GPU matrix multiplication looks like.\n\n```\ngpu void @(\n            layout<bf16,m,n> x, layout<bf16,p,n> y,\n            layout<float,m,p,smem> z\n        )\n    int warp_rows = min((m+63)//64, 4)\n    int wx = warp%warp_rows, wy = warp//warp_rows\n\n    var smem_a = layout<bf16,256,2,16,smem>()\n    var smem_b = layout<bf16,128,2,16,smem>()\n\n    int warp_m = min(m//16*16, 64), warp_p = min(p//16*16, 64)\n\n    int m_tiles = warp_m//16\n    int p_tiles = warp_p//8\n\n    var c = layout<int,4,8,4>()\n\n    int m_cp_tiles = min(m//16,4)\n    [0..m_cp_tiles] i\n        cp_async16(smem_a{256,8}(wx*m_cp_tiles+i, lane),\n                        x{256,8}(wx*m_cp_tiles+i, lane))\n\n    int p_cp_tiles = min(p//16,4)\n    [0..p_cp_tiles] i\n        cp_async16(smem_b{256,8}(wy*p_cp_tiles+i,lane),\n                        y{256,8}(wy*p_cp_tiles+i,lane))\n\n    cp_commit_group()\n    cp_wait_group(0)\n\n    syncthreads()\n\n    var a = layout<int,4,4>()\n    var b = layout<int,8,2>()\n\n    [0..m_tiles] i\n        int row = (wx*m_tiles+i)*16+lane%16\n        int col = lane//16*8\n        ldmatrix_x4(\n                a{i,0},\n                smem_a{row, col})\n\n    [0..p_tiles] i\n        int row = (wy*p_tiles+i)*8+lane%8\n        int col = ((lane//8)%2)*8\n        ldmatrix_x2(\n                b{i,0},\n                smem_b{row, col})\n\n    [0..m_tiles, 0..p_tiles] i, j\n        mma_16x8x16(c{i,j,0},\n                    a{i,0},\n                    b{j,0})\n\n    syncthreads()\n\n    [0..m_tiles, 0..p_tiles, 0..4] i, j, k\n        int global_row = bx*256+ (wx * m_tiles + i) * 16 + (lane // 4) + (k // 2) * 8\n        int global_col = by*128+ (wy * p_tiles + j) * 8  + (lane % 4) * 2 + (k % 2)\n\n        if global_row<m and global_col<p\n            z[global_row * p + global_col] = c[i, j, k]\n\n    syncthreads()\n\nkernel void mma_kernel(bf16[] x, bf16[] y, float[] z, int M, int N, int P)\n    var v = layout<bf16, M, N>(x)\n    var u = layout<bf16, P, N>(y)\n\n    z += v[256,N](bx,0) @ u[128,N](by,0)\n```\n\nDid you like it? It took me one month to replicate the ([HGEMM](https://bruce-lee-ly.medium.com/nvidia-tensor-core-cuda-hgemm-advanced-optimization-5a17eb77dd85)) repo to the beta version of frost, which used C++ CUDA kernels instead of Neve. Currently, I still need to do the benchmarks, but I will wait until I finish flash attention.\n\nHere you see CUDA instructions like mma_16x8x16, cp_async and syncronizations. These are all mandatory when we pretend to implement SOTA matrix multiplications. Now look at the second function, it is almost z += v @ u, but with tiling. The whole previous function is successfully reduced into a new gpu operator.\n\nThere is also the new multi-loop expression\n\n```\n    [0..m_tiles, 0..p_tiles] i, j\n```\n\nWhen it comes to writing, Neve syntax is more pythonic than Julia (yes to \"class\" and no to \"struct\", floating methods and the \"end\" word) and Mojo (no comptime headache keywords, no ownership transfer expressions). It is not dynamically typed as Triton, but allows to define dataloaders from scratch. And strong typying is seem as a benefit in systems programming.\n\nNeve has high-level kernel algebra, while simultaneously allowing CUDA intrinsics in different hierarchies of complexity.\n\nNow let's look at the budget for each programming language.\n\nLet's talk about some other features. The optmizer, the backpropagation and parallelism.\n\nOptimizers declaration can hardly be more compressed than this Frost implementation.\n\n```\nkernel void AdamW_k(float[] param, float[] d, float[] m, float[] v,\n                    float lr, int n, float beta1, float beta2,\n                    float beta1_correction, float beta2_correction,\n                    float eps, float wd)\n    int i = bx*256 + tx\n    if i>=n\n        return\n\n    float m_i = (1-beta1)*d[i] + beta1*m[i]\n    float v_i = (1-beta2)*(d[i]*d[i]) + beta2*v[i]\n    m[i] = m_i\n    v[i] = v_i\n\n    m_i/=beta1_correction\n    v_i/=beta2_correction\n\n    param[i] -= lr*( m_i / (sqrt(v_i) + eps) + wd*param[i] )\n\nclass AdamW\n    float lr, beta1, beta2, wd, eps\n    array<gpu_tensor> m, v\n    int ts\n\n    ctor(float lr, float beta1, float beta2)\n        self.lr = lr\n        self.beta1 = beta1\n        self.beta2 = beta2\n        self.ts = 1\n        self.wd = 0.01\n        self.eps = 0.0000001\n\n    def float step()\n        int i=0\n        for param in $optim_info.params\n            if i+1>self.v.size()\n                self.m.append(new gpu_tensor(param.dims, \"zeros\"))\n                self.v.append(new gpu_tensor(param.dims, \"zeros\"))\n\n            float beta1_correction = 1.0-pow(self.beta1, self.ts)\n            float beta2_correction = 1.0-pow(self.beta2, self.ts)\n\n            launch [param.dims_prod/|256] [256] AdamW_k(param.ptr offby 0, param.d offby 0,\n                                     self.m[i].ptr offby 0, self.v[i].ptr offby 0,\n                                     self.lr, param.dims_prod, self.beta1, self.beta2,\n                                     beta1_correction, beta2_correction, self.eps,\n                                     self.wd\n                                 )\n            param.d=nil\n            i=i+1\n            self.ts = self.ts+1\n        $optim_info.params.clear()\n        tarena_reset()\n```\n\nPyTorch comparison ([AdamW](https://github.com/pytorch/pytorch/blob/main/torch/optim/adamw.py)). Torch does not even have the kernel in the same code.\n\nAll it took were 60 lines\n\n([Backprop youtube shorts](https://www.youtube.com/shorts/3qizuGUWIxY))\n\n```\n# count primes in 10 threads\ndef int is_prime(int n)\n    for i=2, i<(n//2+1)\n        if n%i==0\n            return 0\n    return 1\n\ndef int count_prime_vec(array<int> input_numbers, channel<int,10> ch)\n    int num_primes = 0\n    for i in input_numbers\n        num_primes = num_primes + is_prime(i)\n\n    print(\"Found num primes \", num_primes, \" on thread \", tid)\n    ch <- num_primes\n\nmain\n    array<int> v = arange_int(2,250001)\n    channel<int,10> ch\n\n    finish\n        # split v across 10 threads\n        asyncs 10 count_prime_vec(>v, ch)\n\n    var primes = ch.sum()\n    print(\"Total primes: \", primes)\n```\n\nHere we see two important parallel expressions. The first is the \">\" operator inside the \"asyncs\". It splits v across 10 segments and send each one to different threads in count_prime_vec. The second fathoms the channels. Here, it is used to store the results from the sums of primes for each thread. Then, the main thread uses .sum() to aggregate all results. Neve channels are implemented with lock-free structures. They may block threaded operations if their capacity is full (capacity 10 in the example). This adds room for a lot of possibilities.\n\n```\n    def float worker()\n        print(\"Start worker\")\n        int yield_ptr, bs=self.batch_size\n    print(\"worker \", tid)\n\n        while self.load_ch.alive()\n        yield_ptr = self.increment_yield_ptr()\n\n        for b=0, b<bs\n        self.getitem_w(yield_ptr+b, b)\n\n        self.load_ch <- tid\n        self.x.switch()\n        self.y.switch()\n\n    def tuple<gpu_tensor,gpu_tensor> batch()\n    int w <- self.load_ch\n\n    var x = self.x.load(w)\n    var y = self.y.load(w)\n    x = x.view([$cfg.bs, 3, 32, 32])\n\n        return x, y\n[...]\n    finish\n        asyncs $cfg.num_workers ds.worker()\n\n        [0..steps] i\n                    optimize_logic()\n```\n\nHere channels were used to signal when a worker finished processing its batch. Once the main thread starts processing the batch, it flips its pointer value inside a ping-pong memory buffer, so it may process new data in parallel with data consumption.\n\n3 seeds in a RTX 4090.\n\n| Language | Acc | Time | Backend | \n|---|---|---|---|\n| **Neve** | 65.46%+-0.37 | 592s+-6s | Naive Kernel | \n| **Neve** | 65.46%+-0.45 | 261s+-9s | Partial cuDNN | \n| **PyTorch** | 74.82%+-1.27 | 64s+-6s | cuDNN | \n\nOlder Neve results surpassed PyTorch ( ([old NSK paper](https://arxiv.org/pdf/2409.11600)) ). It won't take too long until the kernels get corrected and optimized.\n\nMore about ([Neve Syntax](https://dev.to/no_saved_data/neve-towards-a-unified-programming-model-for-the-complete-deep-learning-stack-1g86))", "url": "https://wpnews.pro/news/deep-learning-from-scratch-in-1400-lines-neve-frost-framework", "canonical_source": "https://dev.to/no_saved_data/deep-learning-from-scratch-in-1400-lines-neve-frost-framework-4jof", "published_at": "2026-09-11 03:22:32+00:00", "updated_at": "2026-09-11 03:56:39.022077+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "developer-tools", "ai-infrastructure", "neural-networks"], "entities": ["Frost", "Neve", "PyTorch", "ResNet-18", "Sophia optimizer", "Flash Attention", "NoSavedDATA"], "alternates": {"html": "https://wpnews.pro/news/deep-learning-from-scratch-in-1400-lines-neve-frost-framework", "markdown": "https://wpnews.pro/news/deep-learning-from-scratch-in-1400-lines-neve-frost-framework.md", "text": "https://wpnews.pro/news/deep-learning-from-scratch-in-1400-lines-neve-frost-framework.txt", "jsonld": "https://wpnews.pro/news/deep-learning-from-scratch-in-1400-lines-neve-frost-framework.jsonld"}}