Neve - Towards a Unified Programming Model for the Complete Deep Learning Stack

A developer known as No Saved DATA introduced Neve, a new programming language designed to unify high-level Python-like syntax with low-level efficiency for deep learning. The language features LLVM JIT compilation, GPU kernel interfaces, and built-in support for parallelism and channels, aiming to overcome limitations such as Python's GIL. The project is open source and available on GitHub.

Hi folks, this is No Saved DATA. I dedicate this post to describe some of the features I put in Neve to make it an expressive high-level language close to Python/PyTorch syntax , while also allowing efficient low-level code. I am sharing this now, because I believe the language has already strongs traits that allow it to be extended to other problem domains. Current results: Besides, I recently added GPU Kernels code interface. However, the complete framework will still take some more months. The current state is an evolution of a post I made in r/ProgrammingLanguages some months ago https://www.reddit.com/r/ProgrammingLanguages/comments/1ql585o/brand new nsk programming language python syntax/ https://www.reddit.com/r/ProgrammingLanguages/comments/1ql585o/brand new nsk programming language python syntax/ You can check the documentation https://neve-lang.dev https://neve-lang.dev Don't forget to star the repo https://github.com/NoSavedDATA/Neve https://github.com/NoSavedDATA/Neve . And sub to the channel xD https://www.youtube.com/@nosaveddata3994 https://www.youtube.com/@nosaveddata3994 . Discord for extensive talks/suggestions https://discord.gg/hP5feM7cV https://discord.gg/hP5feM7cV ──────────────────────────────────────── Intro I started creating Neve after seeing the code of the Efficient Zero reinforcement learning model. It has a parallelism that PyTorch does not handle, and the implementation required using Cython packages for having threads literaly coding in C, then just calling C functions from Python . Later, I realized PyTorch also needed to implement its data worker threads in C, another workaround over Python Global Interpreter Lock GIL . Not only that, even preprocessing implementations like the BPE are made in C, C++, Rust, etc... So, currently, people must choose between languages like Python for high-level productivity, C and relatives for compute efficiency, Lua for advanced interoperability and other languages for concurrency. Thus, since in my job I had to wait hours for my neural networks to train, I decided to create a programming language in the remaining time. One language that had all these features, which are of high value for deep learning research. Nowadays, I believe it matured to such a point that it may be extended to other complex problem domains. Since Python syntax is very simple and has most of the users, I chose it as the basis. But it run a LLVM JIT in its background. Now I will explain important expressions and features in Neve. ──────────────────────────────────────── Finish/Async and Data Split I experimented Jax deep learning framework for a while. During this period, I learned an expression that would take a tensor or a vector as inputs. It could vectorized the function over the first dimension. A threaded adaptation I made for Neve is: def int foo array<int v print "Thread ", tid, " has vector:" v.print main array<int u = arange int 2,20 finish asyncs 3 foo u This splits a vector across three threads, so it can be processed in parallel. This is useful when you have a list of files, and want a function to process the files across N threads. ──────────────────────────────────────── Channels I saw fireship videos a long time ago about Elixir and Erlang. These languages have actor-message passing, which were used in scaling applications to massive concurrency. Then, this year, my advisor suggested me to study Go and Rust, so I could see the tendencies about modern languages. I got surprised by Go channels expressions, which I thought to be an evolution of the actor-message model but in the end they solve different problems . Go also applies channels to green-threads concurrency within a single OS thread , but I was happy with using it for standard threads. Once I finally adapted Go channels to Neve, I was able to reduce some five lines of code in data loaders. Even if it was only five lines less, it got much cleaner. def float worker print "Start worker" int yield ptr, bs=self.batch size print "worker ", tid while self.load ch.alive yield ptr = self.increment yield ptr for b=0, b<bs self.getitem w yield ptr+b, b self.load ch <- tid self.x.switch self.y.switch def tuple<gpu tensor,gpu tensor batch int w <- self.load ch var x = self.x.load w var y = self.y.load w x = x.view $cfg.bs, 1, 28,28 return x, y These are functions from the dataloader class. The channel communicates which threads have data ready to be consumed. Then, the cpu tensors self.x and self.y can process and yield data using ping-pong buffers. It is much lower level than PyTorch, but without the need of implementing the underlying parallelism in C++. That gets rid of boilerplate mutexes and more than 100 lines of C++ code. Posteriorly, once Neve gets inheritance and interfaces, most of the parallel logic may be hidden, so it can be even closer to PyTorch. The training code is already similar to PyTorch ... gpu tensor a, b a, b = ds.batch var y hat = model.forward a ce loss y hat, b $backprop.backward ──────────────────────────────────────── Anonymous Functions This expression is crucial for mapping tensor operations to their respective backward ops. def int add int x, int y return x+y def int mult int x, int y return x y main map<str, Function<int, int, int m m "add" = add m "mult" = mult print m "mult" 3,4 ──────────────────────────────────────── Generics def T bar<T, U T x, U y print "bar x: ", x print "bar y: ", y return x main int z = bar 3,4 z = bar 5,"$% OU" print "z ", z Generics may yield complex code, but may also save hundreds of lines when the same matrix multiplication function should be implemented for different data types int4, int8, float16, bf16, etc... I still didn't test the generics in this scenario :p . ──────────────────────────────────────── Operation Overload Defining new operations for data types is simple. def gpu tensor @ gpu tensor a, gpu tensor b ... Which works thanks to generics. The operation is consumed as: js var z = x @ y For gpu tensor types. ──────────────────────────────────────── Globals Neve has no primary data type globals. Instead, global values can only be defined as unique instances of classes. class Backprop array<BackNode ops def float register gpu tensor l, gpu tensor r, gpu tensor out, str op self.ops.append new BackNode l, r, out, op This defines the global Backprop class that holds the backs backward function definitions . Then, any tensor operation may use the global instance of Backprop to keep track of the operations to execute later. def gpu tensor @ gpu tensor a, gpu tensor b ... $Backprop.register a, b, ret, "mma" Once Neve finds an "$", it automatically inserts in the main an instruction to create a new instance of that class, so it can be used everywhere. Althought standard global values are not supported, this expression forces global variables to belong to a common scope. It helps preventing pollution/confusion versus standard global vars. For example, you could put all your globals inside a class named Config, then use any of its values. $Config.ip It is straightforward to spot it belongs to a global scope. ──────────────────────────────────────── GPU Kernels python import nsk cuda gpu void @ layout<bf16, m, n x, layout<bf16, p, n y, float z ... kernel void mma kernel bf16 x, bf16 y, float z, int M, int N, int P var v = layout<bf16, M, N x var u = layout<bf16, P, N y z += v 256,N bx,0 @ u 128,N by,0 This one tiles z, v and u, storing the matrix multiplication result in the tiled z positions. The operator overload recovers a function that has shared memory async copies, which are overlapped with tensor core operations, all described in Neve itself. The layout expression is subject to change, but it won't be too much different from the current. ──────────────────────────────────────── Interoperability and Libraries Support In the early stage I was very inexperient with programming languages, so I tried to implement all my important functions and composite data types in C++, and call the functions from Neve. The negative side was that the quick sort was orders of magnitude slower than Python. The positive, I made a C++ tokenizer and parser to extract LLVM bindings. NSK had a heavy focus in using C++ bindings for functionalities. Now it is almost unnecessary, as basically everything can be designed in Neve itself. Use C++ interop when you: - Need system calls only found in C++ you may create a library that maps these calls to Neve ; - Want a custom memory allocator I used this one for GPU mallocs/memory arena . The way Neve adopts C++ functions: extern "C" int float cpu print Scope Struct scope struct, void tensor, DT array vec { After compiling and importing, the functions map naturally to Neve functions and data types. For example, the expression: x.print Will call any function named float cpu print, given that x is a float cpu. That implementation could either be defined in C++ or Neve. Functions that have composite data types require explicit prototypes in Neve, in order to extract the nested type. But if a function takes a composite data type as argument, it is better to define it in Neve when possible. It also allows adding LLVM extension functions in C++, which enable using LLVM for generating IR directly. Besides, it is possible to add new LLVM data-types. C++ and LLVM functions must be compiled to dynamic libraries, and their make require linking system packages. The documentation has a in-depth guide on how to make them work, and the youtube channel has some tutorials about it as well. Overall, I recommend building libraries in Neve itself. You can import libraries using imports in the current directory. python import my nv file import my lib/my nv file These import other .nv files. It is also possible to turn them into packages if you organize them under ~/.local/neve/lib/, then import as: python import my pkg name If you get into the my pkg name folder, you can commit it to github, then anyone can install it with nsm install <my git user /<my pkg name Nsm is automatically installed along with neve when executing the bash install. It works for both Neve and C++ compiled packages more testing is necessary . ──────────────────────────────────────── Other Features ──────────────────────────────────────── Limitations ──────────────────────────────────────── I hope you enjoyed the tour. Ready to test? wget -qO- https://github.com/NoSavedDATA/Neve/releases/download/neve-bin/install.sh | bash I have been building this entirely solo so far. Let me know what you think of the syntax choices, especially the approach to parallelism and GPU kernels Do you think Neve can help you in your domains?