Show HN: CNET – Complex-valued neural networks from scratch in C++/CUDA

A developer released CNet, a C++/CUDA framework for building and researching deep complex-valued neural networks and for optimizing complex-valued functions via gradient descent with Wirtinger derivatives, on Hacker News' Show HN. CNet ships a CPU reference and a CUDA kernel for every built-in layer, requires hand-written kernels wired into GPU dispatch through `GpuMapping` (`gpu/gpumapping.h`) and the `GPU_LAYER_TABLE` registration macro (`gpu/gpu_data.h`) with no automatic CPU-to-GPU codegen, and defines class probabilities with the Born rule p_k = |z_k|^2 / |z|^2 and loss L(z, y) = -sum_k y_k log(|z_k|^2 / |z|^2). The author positions CNet not as a competitor to PyTorch and JAX, which both support complex tensors and complex-aware autodiff, but as a complex-first, auditable substrate for research on complex-valued networks in signal communications, medical imaging and ECG, deepfake detection and acoustic analysis.

CNet is a C++/CUDA framework for building and researching deep complex valued neural networks, as well as for optimization of complex valued functions using gradient descent with Wirtinger derivatives. https://en.wikipedia.org/wiki/Wirtinger derivatives Every built-in layer ships both a CPU reference and a CUDA kernel. Adding a new operator means writing its CPU forward/backward class and a CUDA kernel, then wiring the kernel into the GPU dispatch via GpuMapping gpu/gpumapping.h and the GPU LAYER TABLE registration macro gpu/gpu data.h — there is no automatic CPU→GPU codegen; the kernel is written once, by hand. Why are the complex valued neural network CVNN interesting? For one, it is a research area that is currently neglected due to the huge commercial success of the real valued ones. This Medium article https://machine-learning-made-simple.medium.com/complex-valued-neural-networks-might-be-the-future-of-deep-learning-c51f71f4c835 summarizes well why CVNN might be the future of Deep Learning: "complex numbers work very well for phasic data... The fields where these can be implemented most readily are: Signal Communications; Healthcare both medical image and ECG ; Deep Fake Detections; and Acoustic Analysis for Industrial Maintenance and expansion." Despite the fact that mathematically the field of complex numbers "in spite of the well-known and obvious identification of with these two are entirely different as far as their vector space structure is concerned." PyTorch and JAX both support complex tensors and complex-aware autodiff, and for mainstream work they are the right tools — far more mature, faster at scale, multi-GPU, and backed by large ecosystems. CNet does not compete on that axis. It exists because, for research on complex-valued networks specifically , a complex-first, transparent substrate has concrete advantages: - Complex is the default, not a bolted-on dtype. In the mainstream frameworks complex coverage is uneven: many layers, normalizations, activations, initializers and optimizers assume real tensors, so a CVNN is assembled from real building blocks plus hand-written patches whose gradient conventions must be checked case by case. In CNet every variable, layer, loss and the optimizer are complex by construction . - Wirtinger gradients are explicit, first-class objects. Each layer manipulates the pair$ \partial L/\partial z,\ \partial L/\partial \bar z $ directly, and the non-holomorphic case e.g.$|z|^2$ is handled in the open rather than hidden behind a conjugation convention. Every layer ships a CPU reference and a GPU kernel checked against each other and against finite differences, so the complex gradient is auditable , not trusted. - Spectral structure is built in. The DFT, the Hadamard spectral product, the inverse DFT and the$\mathrm{conv} = \mathrm{IFFT} \mathrm{FFT} \cdot \mathrm{FFT} $ equivalence are first-class layers, so problems whose natural domain is the frequency plane map onto the framework directly. - A physics-native model and loss — the Born rule. This is the sharpest departure from the mainstream frameworks. CNet reads a network's output as a complex amplitude vector $z$ a state and defines the class probabilities by the quantum-measurement Born rule$p k = |z k|^2 / |z|^2$ , with loss$\mathcal{L} z, y = -\sum k y k \log\big |z k|^2 / |z|^2\big $ . Classification is a measurement on an amplitude produced by a cascade of complex — often unitary — operations, rather than real logits passed through a softmax. This inductive bias is natural for physical and wave/field data: an IQ baseband sample is the complex envelope of a real passband signal, an MRI$k$ -space sample is a complex Fourier coefficient, and a wavefunction is a complex amplitude. PyTorch and JAX provide no such head or loss out of the box. - A small, auditable substrate. The full forward/backward of every layer and its CUDA kernel is explicit and readable, and the GPU execution model — a depth-ordered graph cloned across the batch — is visible and controllable. In short: use PyTorch or JAX to ship; use CNet to study complex-valued learning itself — to model data in its native complex / amplitude form with a Born-rule measurement head, to prototype operators whose complex gradients you need to see and control, and to work natively in the spectral domain. For sequence models text, bytes, Go moves or any other token stream you write only the model graph and the data; impl/cnet app.h https://github.com/crasmarum/CNet/blob/main/impl/cnet app.h supplies training on one GPU or many NCCL , Adam with warm-up and cosine decay, gradient clipping, validation with best-checkpoint saving, and inference. include "../impl/cnet app.h" struct MyData : app::DataSource { // vocab , context , sample heldout, rng, in, tgt ... }; struct MyModel : app::ModelDef { // build net, vocab, context - app::Heads embedding, loss ... }; int main int argc, char argv { FLAGS::Parse argc, argv ; MyData data ... ; MyModel model ... ; app::Trainer trainer model, data ; // options from standard flags: -steps -lr -batch -save path ... return trainer.run ; } Inference on a saved model: app::Infer inf; inf.loadModel "model.mod" ; std::vector<int out = inf.generate prompt ids, 300, 0.8, / seed / 1 ; // or nextProbs , sample , score Build and run: make app SRC=examples/char lm.cpp multi-GPU build NCCL=0 for a single-GPU box ./char lm -data book.txt -steps 5000 -batch 8 -save path lm.mod -tw cublas true -tw real loss true ./char lm -data book.txt -generate true -model lm.mod -prompt "ROMEO:" -gen len 300 for r in 0 1 2 3; do ./char lm -data book.txt -batch 24 -world 4 -rank $r -save path lm.mod & done; wait 4 GPUs examples/char lm.cpp https://github.com/crasmarum/CNet/blob/main/examples/char lm.cpp is a complete byte-level language model in about 100 lines, and examples/README app.md https://github.com/crasmarum/CNet/blob/main/examples/README app.md lists every flag and API call. Fresh weights use a normal initialization -init std 0.02 , which trains much better than CNet's older init inputs start. The following code shows how to build a minimal neural net for the MNIST/Fashion MNIST datasets, using the Fourier Transform and Hadamard layers with a Gelu activation function: CNet cnet; auto inp = cnet.add new CInput OutSize 28 28 ; auto fft = cnet.add new FourierTrans InSize 28 28 , {inp} ; auto h data = cnet.add new CInput OutSize 28 28 ; auto hdm = cnet.add new Hadamard InSize 28 28 , InSize 28 28 , {fft, h data} ; auto gelu = cnet.add new CGelu InSize 28 28 , {hdm} ; auto l data = cnet.add new CInput OutSize 28 28 10 ; auto lin = cnet.add new Linear InSize 28 28 , InSize 28 28 10 , {gelu, l data} ; auto outp = cnet.add new CrossEntropy InSize 10 , {lin} ; Please see the file cnet.cpp https://github.com/crasmarum/CNet/blob/main/cnet.cpp for how to train on CPU/GPU the above complex valued neural net with the MNIST/Fashion MNIST datasets. In order to add a new complex valued function, say CFunc class and implement forward plus either backward or, more simply, the two Wirtinger derivatives dz and dz star , from which CNet builds the backward pass. A layer also needs a constructor that takes a Uid , a clone method and a getName : CNet clones every layer once per batch entry on the GPU and uses the name when saving and restoring models. class CSigmoid: public CFunc { public: CSigmoid Uid uid, InSize in size : CFunc uid, in size, OutSize in size.value {} CSigmoid InSize in size : CFunc in size, OutSize in size.value {} virtual CFunc clone Uid uid { return new CSigmoid uid, InSize input .length ; } virtual std::string getName { return "CSigmoid " + std::to string uid ; } }; While writing an implementation for the forward method is usually straightforward: virtual void forward { for int i = 0; i < no outputs ; ++i { // one copy of the output per consuming layer for int in indx = 0; in indx < input .length ; ++in indx { complex<float g = 1.0f / 1.0f + exp -input .z in indx ; mutable output i - real offset i + in indx = g.real ; mutable output i - imag offset i + in indx = g.imag ; } } } providing an implementation for the backward method is in general more difficult. The CNet framework makes things easier whenever you can compute the Wirtinger derivatives https://en.wikipedia.org/wiki/Wirtinger derivatives for the corresponding multivariable functions as you can write implementations for the dz and the dz star methods instead. For example for the Sigmoid function while the From the above observations we can easily implement the dz and the dz star methods: virtual complex<float dz int out indx, int in indx { if out indx = in indx { return 0; } auto g = 1.0f / 1.0f + exp -input .z in indx ; return g 1.0f - g ; } virtual complex<float dz star int out indx, int in indx { return 0; } A function that depends on dz star . This is the usual case in practice: the complex sigmoid above has poles at CGelu instead applies a real function A quick smoke test is to minimize the squared norm of the Sigmoid outputs with a small net: L2Out computes CNet net; auto inp = net.add new CInput OutSize 128 ; auto sigm = net.add new CSigmoid InSize 128 , {inp} ; auto l2 = net.add new L2Out InSize 128 , {sigm} ; net.init inputs ; net.init exec graph true ; for int var = 0; var < 1000; ++var { net.forward ; cout << var << "\tLoss: " << L2Out net l2 - loss << endl; net.backward 0 ; net.updateInputs 0.9 ; } Running the above code from the command line: ~/cnet -test sigmoid true one can observe that the loss is decreasing: Depth 0: Input 1 4096, Depth 1: CSigmoid 2 4096, Depth 2: L2Out 3 4128, 0 Loss: 32.004 1 Loss: 28.508 2 Loss: 25.422 3 Loss: 22.722 4 Loss: 20.374 5 Loss: 18.338 ... 791 Loss: 0.100 ... 999 Loss: 0.079 Please see the file examples/sigmoid.h https://github.com/crasmarum/CNet/blob/main/examples/sigmoid.h for additional details. A falling loss only shows that the gradient points downhill; to check dz and dz star themselves, compare the backward pass with central finite differences of the forward pass, as the finite-difference checks in examples/test bornattn mh.cpp https://github.com/crasmarum/CNet/blob/main/examples/test bornattn mh.cpp do. Two practical notes: - Cost of the generic backward. When you implement only dz and dz star , the inherited backward evaluates them for every output, input pair, which is$O n {in} \cdot n {out} $ calls. That is fine for the 128-element example above but far too slow for large activations. For element-wise or otherwise sparse layers, override backward and accumulate the gradient directly, as CGelu and TokenNorm do. - GPU. A new CFunc runs on the CPU. To run it inside a GPU graph it also needs a GPU mapping with its own kernels; see Computation Graph on CUDA computation-graph-on-cuda and the existing layers under gpu/ . In order to execute the forward and the backward passes on a CNet complex valued neural net, the functional graph is ordered by depth, as shown in the following diagram: On an NVIDIA GPU the CNet framework will try to clone the CPU net and execute the forward and the backward passes in parallel following the graph computation depth. In the following image you can see how a batch of size 32 is deployed on a GPU: Because the clones execute concurrently across the batch, GPU memory grows linearly with the batch size while the per-step wall-clock time stays essentially flat. The table below measures this cost for a 6-block complex FNet flattened token-major stream of T E = 16384 complex values on a single NVIDIA H100 eager : | batch | GPU mem MB | fwd ms/step | bwd ms/step | |---|---|---|---| | 16 | 167.8 | 0.33 | 0.32 | | 32 | 335.5 | 0.33 | 0.32 | | 64 | 671.1 | 0.33 | 0.32 | | 128 | 1342.2 | 0.33 | 0.32 | | 256 | 2684.4 | 0.33 | 0.32 | Memory is linear in the batch size — each clone holds its own activations — while the forward and backward times per step remain near-constant from batch 16 up to 256, since the clones are launched level by level and run in parallel. Throughput therefore scales with the batch at roughly fixed latency. A reduced-memory inference layout Vars::INFER DIMS drops the per-example footprint further by not allocating the gradient/optimizer buffers that only training needs. Because the clone arena is static across steps — device pointers for activations, weights, and Wirtinger gradients never move after allocateOnGpu — the entire level-by-level launch schedule can be captured once into a CUDA graph and replayed with a single cudaGraphLaunch CNet::GpuForwardGraphed / GpuBackwardGraphed . This removes the per-kernel launch/sync overhead: on an H100 the forward+backward step drops from 0.65 ms to 0.32 ms ~2×, flat across batch while staying bit-identical to eager execution, and the captured step runs faster than eager torch.complex64 . The standalone bench clone example reproduces this: make bench clone ./bench clone -fft fast true eager ./bench clone -fft fast true -bc graph 1 replay captured CUDA graphs ./bench clone -fft fast true -bc check 1 verify graph replay == eager bit-identical The Input Layer is used as the main input for the neural network as well as the input parameters for other layers, e.g., Linear or Hadamard layers. For example, in the following code snippet inp is the main input and the h data is the parameter for the Hadamard function. include "impl/cinput.h" CNet cnet; auto inp = cnet.add new CInput OutSize 28 28 ; auto h data = cnet.add new CInput OutSize 28 28 ; auto hdm = cnet.add new Hadamard InSize 28 28 , InSize 28 28 , {inp, h data} ; The main variable for this layer is the output size, e.g., CInput OutSize 1024 . As the main input of a neural net, on CPU you can set its complex values from a batch using this method: setInput InputBatch& batch, int b indx As the input parameter for other layers, its values are either randomly set at the beginning of training a network, e.g., net.init inputs for deep residual models prefer net.normalInit 0.02, seed , see the -init std flag below , or are restored from a model via net.restore std::string file . See the file cnet.cpp https://github.com/crasmarum/CNet/blob/main/cnet.cpp for more details. The Embedding Layer is used only as the main input for the neural network. The main variables of an embedding layer are the embedding dimension, the number of embeddings and the maximum number of input tokens. For example, in the following code snippet we create an embedding consisting of no embedings = 100 tokens of dimension emb dim = 300 that has an input of maximum max in tokens = 64 tokens and has an output of emb dim max no tokens = 300 64 complex numbers. Note that the output is padded with zeros if necessary. include "impl/embed.h" int emb dim = 300; int max in tokens = 64; int no embedings = 100; CNet cnet; auto emb = cnet.add new CEmbedding emb dim, max in tokens, no embedings ; auto fft = cnet.add new FourierTrans InSize emb dim max in tokens , {emb} ; On a CPU you can set the output embedding values using a vector<int data structure, e.g., embedding.setInput {2, 1, 1, 3} ; while on GPU you can do it via the net.batchToGpu InputFunc inp, OutputFunc outp, Batch batch ; method. The initial complex values for the embeddings are either randomly set at the beginning of training a network, e.g., net.init inputs , or are restored from a model via net.restore std::string file . This layer implements the Discrete Fourier Transform https://en.wikipedia.org/wiki/Discrete Fourier transform function The main variable of this function is its input/output size. Example of its usage: include "impl/ft.h" CNet cnet; auto inp = cnet.add new CInput OutSize 28 28 ; auto fft = cnet.add new FourierTrans InSize 28 28 , {inp} ; This layer implements the Inverse Discrete Fourier Transform https://en.wikipedia.org/wiki/Discrete Fourier transform , the exact unitary inverse of FourierTrans : It is typically used to bring a Hadamard product back to the time domain, i.e. to realise a circular convolution as include "impl/ft.h" auto fft = cnet.add new FourierTrans InSize 28 28 , {inp} ; auto hdm = cnet.add new Hadamard InSize 28 28 , InSize 28 28 , {fft, h data} ; auto ifft = cnet.add new InverseFourierTrans InSize 28 28 , {hdm} ; This layer implements a causal variant of the DFT: a lower-triangular-masked Fourier sum This makes it a parameter-free causal token mixer — the autoregressive analogue of self-attention used in FNet https://aclanthology.org/2022.naacl-main.319.pdf -style language models: it mixes information across tokens while never letting a position see its own future. With a token-major CEmbedding layout token -fft fast , which evaluates the causal transform via a Bluestein/chirp-z factorisation with batched cuFFT forward and Wirtinger backward . include "impl/ft.h" auto emb = cnet.add new CEmbedding emb dim, max in tokens, no embedings ; auto mix = cnet.add new TriangFourier InSize emb dim max in tokens , {emb} ; This is a content-based causal token mixer: the complex analogue of single-head self-attention, but with the attention scores given by the quantum-measurement Born rule instead of a softmaxed dot product. The input is the concatenation of three complex blocks The score TokenwiseLinear projections of the stream: include "impl/bornattn.h" int N = 64, E = 128, d = E / 2; // N tokens, width E, head dim d auto Q = cnet.add new TokenwiseLinear N, E, d , {pre, wQ} ; auto K = cnet.add new TokenwiseLinear N, E, d , {pre, wK} ; auto V = cnet.add new TokenwiseLinear N, E, d , {pre, wV} ; auto att = cnet.add new BornAttention N, d , {Q, K, V} ; // input Q;K;V - O BornAttention N, d, causal, eps, heads splits the heads heads of heads = 1 is the single-head operator above. RotaryEmbed N, d, base, heads restarts the frequency ladder every heads 1 or a non-zero eps use an extended record; old checkpoints restore unchanged. Smaller heads matter for the linear mode below: its state and per-token cost scale with Because the score is a squared inner product it has an exact finite feature map: lifting each vector to its outer product setLinear true, chunk switches the GPU kernels to this form: tokens are processed in chunks of chunk N % chunk == 0 , each chunk mixes with itself through the quadratic kernel and with everything before it through the carried state, and the state advances by one cuBLAS batched GEMM per chunk. The backward is exact too query gradients read the stored per-chunk forward state; key and value gradients a suffix state swept in reverse . The CPU operator stays quadratic and is the oracle: make test bornattn mh && ./test bornattn mh checks the linear forward and all input gradients against it on eight configurations, together with finite differences, a head-slicing oracle, serialization and GPU/CPU parity. auto att = cnet.add new BornAttention N, d, / causal= /true, / eps= /0.f, / heads= /4 , {Qr, Kr, V} ; BornAttention cnet att - setLinear true, / chunk= /256 ; // GPU: exact O N d h^2 form The same operator, so the same model: a run switched between the two forms reaches the same loss within the quadratic kernel's own run-to-run spread, which comes from its atomic backward . Attention layer alone, | N | quadratic | linear chunk 256 | speed-up | |---|---|---|---| | 1k | 108 ms, 0.25 GB | 20 ms | 5.5x | | 4k | 692 ms, 1.4 GB | 78 ms | 8.9x | | 16k | 11.3 s, 14.9 GB | 310 ms, 6.1 GB | 37x | | 64k | out of memory | 1.24 s, 12.6 GB | --- | Constraints: causal attention only, N % chunk == 0 , gpu/born linear cnet.md https://github.com/crasmarum/CNet/blob/main/gpu/born linear cnet.md . setDecay true trainer flag -born decay true multiplies the score by make test bornattn mh . In the controlled PyTorch study of the companion paper the decay was the single most effective fix for long contexts at 128k it recovered 0.66 of the 0.70 nats between plain Born and gated linear attention . The running state also gives constant-memory autoregressive decoding: one token in, one token out, the state updated in place, never touching past tokens. CNet graphs are static, so the decoder is a second graph with one token per step, built from a restored model by impl/stepnet.h https://github.com/crasmarum/CNet/blob/main/impl/stepnet.h : the same layers and wiring, every per-token size divided by RotaryEmbed given an absolute position setPos , and BornAttention in step mode setStepMode true , forwardStep . Fed the tokens one at a time it reproduces the fixed-window model's next-token distribution to TokenwiseLinear , TokenNorm , RotaryEmbed , BornAttention , GatedLinearAttention , Residual , CGelu , SequenceCrossEntropy ; Fourier mixers and the SoftMax block norm have no constant-state form and are rejected. See the generate example below. The degree-one counterpart of Born attention, added to compare the two inside CNet with everything else held fixed. Where Born attention routes with the probability amplitude BornAttention heads heads of a RotaryEmbed on TokenNorm normally follows the layer. The current token is never gated. The inner product uses the same convention as BornAttention , setStepMode true / forwardStep give the one-token recurrence for the streaming decoder. include "impl/gla.h" int N = 4096, E = 256, d = E / 2, heads = 4; auto Q = cnet.add new TokenwiseLinear N, E, d , {pre, wQ} ; // then RotaryEmbed N, d, 10000.0, heads on Q and K auto K = cnet.add new TokenwiseLinear N, E, d , {pre, wK} ; auto V = cnet.add new TokenwiseLinear N, E, d , {pre, wV} ; auto G = cnet.add new TokenwiseLinear N, E, heads , {pre, wG} ; // per-head gate logits auto gla = cnet.add new GatedLinearAttention N, d, heads , {Q, K, V, G} ; GatedLinearAttention cnet gla - setChunk 512 ; // GPU chunk length N % chunk == 0 auto att = cnet.add new TokenNorm N, d , {gla} ; // output norm: the mix has no normaliser On the GPU the layer runs chunkwise like the linear Born mode: the inter-chunk readout and the state update are complex GEMMs four real cuBLAS GEMMs on the re/im planes , one kernel forms the intra-chunk amplitudes, and the backward sweeps the chunks in reverse with a suffix state. Constraints: causal only, N % chunk == 0 , make test gla && ./test gla runs the finite-difference, recurrence, serialisation and GPU-parity checks. On the PG-19 character model 6 blocks, -born decay true ; the gated kernel is a first version 942 vs 815 and 835 ms per step . A controlled repeat of the same architecture in the PyTorch mirror puts the gated mixer level with Born plus a fixed per-head decay 1.230 vs 1.210 nats/char; plain Born 1.370 , so its gain is forgetting; CNet's plain Born trailing its mirror is under investigation. Born attention remains CNet's recommended mixer; the gated layer is an option. Design, derivation and measurements: gpu/gla cnet.md https://github.com/crasmarum/CNet/blob/main/gpu/gla cnet.md . RotaryEmbed applies a position-dependent complex phase rotation to each token's features — rotary position embeddings RoPE expressed as what they natively are in a complex network: a rotation. The input is Applied to the queries and keys of BornAttention , the per-feature inner product relative phase include "impl/rotary.h" int N = 64, dh = 64; auto Qr = cnet.add new RotaryEmbed N, dh , {Q} ; // rotate queries auto Kr = cnet.add new RotaryEmbed N, dh , {K} ; // rotate keys auto att = cnet.add new BornAttention N, dh , {Qr, Kr, V} ; This layer implements the Hadamard function which is simply the element-wise multiplication: In the complex valued neural network world, the Hadamard Layer is equivalent with the Convolution layer, because the Fourier Transform famously commutes with the convolution: On the Equivalence of Convolutional and Hadamard Networks using DFT https://arxiv.org/abs/1810.11650 . You can see an example of using the Hadamard layer hdm with some parameters provided by an Input Layer h data in the following code snippet: include "impl/hadamard.h" CNet cnet; auto inp = cnet.add new CInput OutSize 28 28 ; auto fft = cnet.add new FourierTrans InSize 28 28 , {inp} ; auto h data = cnet.add new CInput OutSize 28 28 ; auto hdm = cnet.add new Hadamard InSize 28 28 , InSize 28 28 , {fft, h data} ; This layer implements the Residual function which computes the element-wise addition: You can see an example of using the Residual layer res in the following code snippet: include "impl/residual.h" CNet cnet; //... auto fft = cnet.add new FourierTrans InSize 28 28 , {inp} ; //... auto hdm = cnet.add new Hadamard InSize 28 28 , InSize 28 28 , {fft, h data} ; auto res = cnet.add new Residual InSize 28 28 , InSize 28 28 , {fft, hdm} ; This layer is the equivalent of the fully connected / dense layer and is performing a matrix multiplication: You can see an example of using the Linear layer lin in the following code snippet: include "impl/linear.h" CNet cnet; // ... auto l data = cnet.add new CInput OutSize 512 10 ; auto lin = cnet.add new Linear InSize 512 , InSize 512 10 , {gelu, l data} ; auto outp = cnet.add new CrossEntropy InSize 10 , {lin} ; This is a position-wise linear layer: the input is seen as same Sharing the weight across positions costs Linear over the flattened sequence — this is what makes a per-token feed-forward block affordable in a sequence model. The constructor takes the number of tokens and the input/output embedding widths: include "impl/tokenwise.h" int N = 64, E = 128, H = 4 E; // N tokens, width E, hidden H auto w1 = cnet.add new CInput OutSize E H ; auto ff = cnet.add new TokenwiseLinear N, E, H , {hidden, w1} ; On the GPU, TokenwiseLinear has two kernel paths: a naive per-element complex multiply-accumulate, and a batched cuBLAS GEMM path the complex product is four real cublasSgemmBatched calls . The batched path is a large win once the projection is wide — most visibly a language-model head at vocabulary width — where it measured ≈3.4× faster 607 → 177 ms/step on a vocab-8192, N=256 , E=256 model on an H100; at small output widths e.g. a character vocabulary the two paths are indistinguishable. The dp shakespeare example selects it with -tw cublas true . This activation function is the equivalent of Relu: You can see an example of using the CRelu layer rel in the following code snippet: include "impl/relu.h" CNet cnet; // ... auto inp = cnet.add new CInput OutSize 512 ; auto rel = cnet.add new Crelu InSize 512 , {inp} ; This activation function is the equivalent of Gelu: You can read more about Gaussian Error Linear Units GELUs https://arxiv.org/abs/1606.08415 . You can see an example of using the CGelu layer rel in the following code snippet: include "impl/relu.h" CNet cnet; // ... auto inp = cnet.add new CInput OutSize 512 ; auto rel = cnet.add new CGelu InSize 512 , {inp} ; Despite its name, this layer is the complex Unlike CGelu , this activation is scale-covariant and commutes with the unitary Fourier Transform , so it can be used inside a spectral network without breaking the FFT/convolution equivalence — useful as the nonlinearity between spectral mixing layers. include "impl/softmax.h" auto inp = cnet.add new CInput OutSize 512 ; auto nrm = cnet.add new SoftMax InSize 512 , {inp} ; Where SoftMax normalizes a whole block by its global Euclidean norm, TokenNorm normalizes each token independently by its own root-mean-power — the complex analogue of gain-free RMSNorm/LayerNorm, and the per-token counterpart of SoftMax . The input is A learnable affine is intentionally omitted a following TokenwiseLinear supplies gain and bias , which keeps the layer parameter-free and its Wirtinger backward exact. With {dd},T/ 2 d r^3 $ and $\partial L/\partial \bar{x} {dd} = \partial L/\partial \bar{y} {dd} /r - x {dd},T/ 2 d r^3 $ verified: CPU finite-difference and GPU-vs-CPU to machine precision . Used as a pre-norm normalize the residual-stream input to each sub-block , it lets deep complex transformers train stably — on tiny-shakespeare it breaks the plateau that global block-normalization hits. The constructor takes the number of tokens, the per-token feature width, and an optional include "impl/tokennorm.h" int N = 64, E = 128; auto normed = cnet.add new TokenNorm N, E , {last} ; // per-token RMS pre-norm auto Q = cnet.add new TokenwiseLinear N, E, E / 2 , {normed, wQ} ; ComplexDropout is inverted dropout for complex activations: at training time each element is kept with probability inference it is the identity. Since the map is multiplication by a real per-element constant not stored — it is a deterministic hash of seed, step, element recomputed identically in the forward and backward kernels, and the GPU mapping bumps step each forward for a fresh mask the seed folds in the layer depth so stacked dropout layers are independent . The CPU path is the identity, so generation/evaluation are unaffected. The constructor takes the input size and the drop probability: include "impl/dropout.h" int N = 64, E = 128; auto sub = cnet.add new TokenwiseLinear N, E, E , {g, w} ; // a sub-layer output auto drop = cnet.add new ComplexDropout InSize N E , 0.1f , {sub} ; Caveat for the Born-rule loss. Dropout is a conventional regularizer, but the Born-rule SequenceCrossEntropy has a unigram saddle whose escape is numerically knife-edge it is tipped by the summation order of the TokenwiseLinear GEMM — see the language-model example . Empirically, any stochastic perturbation — dropout or gradient noise — suppresses that escape : on tiny-shakespeare / sub-word data a model that learns fine with dropout off gets pinned at the unigram plateau with -dropout 0.1 . So for Born-rule language-model training, leave dropout off; it is provided for conventional real-logit CrossEntropy networks and as a documented research knob. ConjLagProduct computes the instantaneous autocorrelation at a fixed lag tau : It is the nonlinear, non-holomorphic primitive behind cyclostationary and higher-order modulation features: the time-mean of the output estimates the lag- tau autocorrelation R tau bounded |R| <= 1 for power-normalised input, so well-scaled , while an FFT of the output over time gives the cyclic spectrum spectral lines at the symbol-rate / carrier cycle frequencies . It is the complex, two-input generalisation of CModulus2 |z|^2 = x conj x is the tau = 0 case . The Wirtinger backward verified to ~1e-4 against central finite differences on CPU, and exactly against the CUDA kernel differentiates y = a conj b through both inputs a = x t , b = x {t-tau} . The constructor takes the input length and the lag: include "impl/conjlag.h" int N = 128, tau = 8; auto r = cnet.add new ConjLagProduct InSize N , tau , {x} ; // x t conj x {t-tau} auto Rt = cnet.add new MeanPool InSize N , OutSize 4 , {r} ; // pooled autocorrelation R tau This is the a-priori signal structure the pooled-power features miss. On held-out RML2016.10a 11-way modulation , adding a small bank of ConjLagProduct lags R tau and |R tau |^2 to the arch-11 spectral-conv classifier raised accuracy from 0.340 to 0.363 under an identical recipe — a modest but real gain in the predicted direction separating the same-band QAM16/QAM64 and 8PSK/QPSK confusions that linear/power features cannot . A note on scaling: read out the time-mean R tau O 1 , not the raw |FFT r tau |^2 energy ~N , which otherwise saturates the Born-rule logits. The full cyclostationary front end adds a SoftMax-normalised cyclic spectrum on top of this: SoftMax FFT r tau , an O 1 read-out of the spectral-line pattern that keeps the Born logits well-scaled. On the artifact-corrected RML22 benchmark this cyclostationary classifier reaches ~0.50 overall — on par with the VT-CNN2 real-valued baseline held-out RML2016.10a ~0.49 , closing most of the pooled-power gap. SegmentedFourier is a short-time Fourier transform expressed as a block-diagonal DFT: the length- N input is split into N/window contiguous windows of length window , and each window is DFT'd independently unitary, 1/sqrt window scaling . Output index b w+p window b , frequency bin p : With a following CModulus2 and MeanPool it yields a spectrogram N/window time steps × window frequency bins — a time–frequency feature that keeps local temporal structure the global FourierTrans averages away. It is the exact unitary FourierTrans when window = N ; its Wirtinger backward mirrors FourierTrans per window and is verified to ~2e-4 against central finite differences on CPU, and exactly against the CUDA kernel roots are computed on the fly in the kernel . The constructor takes the input length and the window length which must divide it : include "impl/segfourier.h" int N = 128, window = 32; auto stft = cnet.add new SegmentedFourier InSize N , window , {x} ; // 4 windows x 32 bins auto mag = cnet.add new CModulus2 InSize N , {stft} ; // |.|^2 spectrogram Empirically, on RML a segmented-Fourier branch windows {16, 32} gave no gain over the cyclic spectrum above — the two features are redundant — which we record as an honest negative. It is provided as a general-purpose spectral/time–frequency primitive. This loss function is simply the square of the You can see an example of using the L2Out loss function l2 in the following code snippet: include "impl/l2out.h" CNet net; auto inp = net.add new CInput OutSize 128 ; auto sigm = net.add new CSigmoid InSize 128 , {inp} ; auto l2 = net.add new L2Out InSize 128 , {sigm} ; The CrossEntropy loss implements the multivariable function You can read more on it in the On the Equivalence of Convolutional and Hadamard Networks using DFT https://arxiv.org/abs/1810.11650 research paper. You can see an example of using the CrossEntropy loss function ce in the following code snippet, as well as in the cnet.cpp https://github.com/crasmarum/CNet/blob/main/cnet.cpp file: include "impl/crossent.h" CNet cnet; // ... auto l data = cnet.add new CInput OutSize 28 28 10 ; auto lin = cnet.add new Linear InSize 28 28 , InSize 28 28 10 , {gelu, l data} ; auto ce = cnet.add new CrossEntropy InSize 10 , {lin} ; This is the autoregressive analogue of CrossEntropy . The input is The per-position targets the input tokens shifted left by one are set with setTargets std::vector<int before each forward pass. Combined with a token-major CEmbedding , a causal TriangFourier mixer and TokenwiseLinear feed-forward blocks, this trains a character-level language model where every position predicts its next token in a single pass. include "impl/seqcrossent.h" int N = 64, vocab = 65; auto logits = cnet.add new TokenwiseLinear N, emb dim, vocab , {last, w out} ; auto seq = cnet.add new SequenceCrossEntropy InSize N vocab , vocab , {logits} ; // ... each step: seq- setTargets targets ; // per-position next-token targets The learnable CInput / CEmbedding parameters are trained with the Wirtinger gradients accumulated during the backward pass. Three update rules are available, each on both CPU and GPU: - Plain gradient descent — updateInputs l rate CPU / gpuUpdateInputs l rate GPU . - Momentum Adam first moment only — adamUpdate ... . - Full Adam first + second moment with bias correction, and optional gradient clip-by-value — trueAdamUpdate ... . On GPU a batch is trained as a set of net clones, so the update is called once per step with the learning rate scaled by the batch size the per-clone gradients are summed onto the ancestor parameters first : // plain SGD cnet.gpuUpdateInputs l rate / batch.size ; // momentum Adam cnet.adamUpdate l rate / batch.size , beta, step ; // full Adam with gradient clipping grad clip = 0 disables the clip cnet.trueAdamUpdate l rate / batch.size , beta1, beta2, eps, step, grad clip ; On CPU, where gradients are accumulated over several samples before an update, the Adam variants additionally take that accumulation batch size: net.adamUpdate l rate, batch size, beta, step ; net.trueAdamUpdate l rate, batch size, beta1, beta2, eps, step ; Because the Born-rule CrossEntropy / SequenceCrossEntropy losses are ill-conditioned, full Adam trueAdamUpdate with gradient clipping is recommended over plain SGD for classification and language-model training. The ill-conditioning is concrete: the exact gradient of correct token, so a single hard example can produce an explosive update that parameter-level gradient clipping cannot catch the bad gradient is already summed into the minibatch . SequenceCrossEntropy therefore offers an optional probability floor setEps eps the -ce eps flag in the language-model example : the target modulus is clamped from below, in the gradient only — a uniform measurement-noise floor, i.e. label-smoothing for the Born rule. The forward loss stays exact, so the reported metric is unchanged; only the optimization is regularised, and eps = 0 is bit-identical to the exact gradient. A value around CNet is currently supported only on Linux/MacOS and you can build it using make. The Makefile checks for the presence of the NVIDIA CUDA Compiler nvcc and if nvcc is found, it compiles the project with CUDA support. Otherwise, it falls back to using g++. git clone https://github.com/crasmarum/CNet.git cd CNet make With nvcc the build emits native code for Ampere sm 80 , A100 and Hopper sm 90 , H100/H200/GH200 plus compute 90 PTX that newer GPUs JIT-compile. Override the targets with GPU ARCH , for example make GPU ARCH="-gencode arch=compute 100,code=sm 100" for Blackwell. Example programs make dp shakespeare , make generate , make radioml , the tests are rebuilt when any header they include changes; if a binary misbehaves after a large edit, rm -rf build/obj and rebuild. If the NVIDIA CUDA Compiler is found, the make file will automatically compile CNet with CUDA support. For example, once you download locally the MNIST/Fashion MNIST datasets, you can train the example neural network from the command line like follows. ~/cnet -mnist images ~/train-images.idx3-ubyte -mnist labels ~/train-labels.idx1-ubyte \ -model path ~/test.mod -mnist gpu train true You can change the default learning rate and the GPU batch size via the l rate and batch size command line flags: ~/cnet -mnist images ~/train-images.idx3-ubyte -mnist labels ~/train-labels.idx1-ubyte \ -model path ~/test.mod -mnist gpu train true -l rate 0.001 -batch size 40 You can find the example training code in the cnet.cpp https://github.com/crasmarum/CNet/blob/main/cnet.cpp file: MnistDataReader reader; assert reader.Open mnist images, mnist labels, 60000 ; assert reader.readData ; CNet cnet; assert cnet.hasGPU ; int cinp, coutp; createMnistNet cnet, &cinp, &coutp ; std::cout << cnet.cpuNet .toString << std::endl; CInput inp = CInput cnet cinp ; CrossEntropy cen = CrossEntropy cnet coutp ; cnet.init inputs ; assert cnet.allocateOnGpu batch size ; std::cout << std::setprecision 4 << "L rate: " << l rate << "\t batch size: " << batch size << std::endl; float avg loss = 0; for int epoch = 0; epoch < no epochs; ++epoch { reader.shuffle ; for int time = 0; time < reader.size / batch size; ++time { auto batch = reader.nextBatch batch size ; cnet.gpuForward inp, cen, batch ; avg loss += cnet.getLoss cen, &batch ; if time % 100 == 99 { std::cout << epoch << "\t" << time << "\t loss: " << avg loss / 100 << std::endl; avg loss = 0; } if time % 1000 == 999 { if cnet.getInputsFromGpu { std::cerr << "Warning: could not get data from GPU." << std::endl; } else { cnet.save model path ; std::cout << "Model saved at: " << model path << std::endl; } } cnet.gpuBackward ; cnet.gpuUpdateInputs l rate / batch.size ; } } This loop is deliberately minimal: plain gradient descent gpuUpdateInputs and CNet's original init inputs start, which is enough for this shallow network. For deeper models use Adam trueAdamUpdate , as dp shakespeare does and the normal initialization cnet.cpuNet .normalInit 0.02, seed after init inputs ; the -init std flag in dp shakespeare , gollm and app::Trainer , which trains much better. For token-sequence models, app::Trainer your-own-model-in-one-file-apptrainer-and-appinfer packages the whole loop: batching, Adam with warm-up and cosine decay, clipping, validation, checkpoints and multi-GPU. CNet trains across multiple GPUs with data parallelism , using the same clone mechanism it uses to batch within one GPU — only one level up. Each GPU holds a full copy of the network; the global batch is split across GPUs; after each backward the per-GPU gradients are summed with an NCCL all-reduce ; and every replica applies the identical optimizer step, so the replicas stay in sync and each step is the same update as single-GPU training with the full batch, up to the order of floating-point summation. Two pieces make this work see gpu/gpu func.h : ancestorGradBuffers exposes each parameter's gradient buffer for an in-place all-reduce, and the replicas must start from the same weights. dp shakespeare and app::Trainer initialize every rank with the same seed; collectAncestorParams / loadAncestorParams can broadcast weights instead. The standard DDP architecture — one process per GPU — avoids serializing CNet's many kernel launches through a single CUDA context, and scales near-linearly. A complete, self-contained example trains a complex FNet character-level language model on tiny-shakespeare: make dp shakespeare 1 GPU ./dp shakespeare -world 1 -batch 48 -steps 3000 -data tiny shakespear2.txt N GPUs one process per GPU ; the helper script launches and pins them ./examples/run dp shakespeare.sh 4 -batch 48 -steps 3000 By default the example uses the parameter-free TriangFourier causal mixer. Flags let it assemble a fully complex transformer instead: -born true swaps in the BornAttention content-based mixer with TokenwiseLinear Q/K/V/O projections , -rope true adds RotaryEmbed rotary position embeddings on the queries/keys, -token norm true adds per-token TokenNorm pre-normalization pre-LN , and -ce eps 0.001 enables the Born-loss probability floor for stable deep training: ./examples/run dp shakespeare.sh 4 -born true -rope true -token norm true -ce eps 0.001 \ -emb 160 -tokens 64 -blocks 8 -batch 64 -steps 8000 -grad clip 0.5 -val data <file takes a separate held-out text file, and then all of -data is training data; without it the last 10% of -data is held out. The character corpus is memory-mapped one byte per character, 64-bit positions, one page-cache copy shared by all ranks on a node , so multi-gigabyte corpora such as the full PG-19 training set 11.7 GB load in seconds with a few GB of host RAM. -born decay true adds the fixed per-head decay described above recommended for contexts beyond a few thousand tokens ; -pos table false drops the learned positional table RoPE only , which is what lets the trained model stream past its context see below . Two more flags select the multi-head and linear forms of the mixer: -heads 4 splits the -born linear true -born chunk 256 runs the exact chunkwise linear kernels GPU only; -tokens must be a multiple of -born chunk . With them the same model trains at contexts the quadratic kernel cannot reach: at -tokens 4096 -batch 8 the quadratic kernel runs out of memory on a 40 GB A100 while the linear form steps in 0.78 s, and at -tokens 1024 the whole training step is 6.4x faster 0.21 s vs 1.35 s . Everything else — RoPE, TokenNorm, the Born read-out, checkpoints — is unchanged, and a checkpoint trained one way can be used the other way. ./examples/run dp shakespeare.sh 1 -born true -heads 4 -born linear true -born chunk 256 \ -rope true -token norm true -tw cublas true -ce eps 0.001 \ -emb 256 -tokens 4096 -blocks 4 -batch 8 -steps 3000 -save path born4k.mod -gla true replaces the Born mixer in every attention block by the gated linear attention over complex amplitudes GatedLinearAttention with a TokenwiseLinear gate projection and a TokenNorm after the mix; -heads , -rope , -token norm and -born chunk apply as before; GPU chunkwise kernels, so -tokens must be a multiple of -born chunk . Checkpoints trained either way serialise, restore and stream through generate unchanged: ./examples/run dp shakespeare.sh 1 -gla true -heads 4 -born chunk 512 \ -rope true -token norm true -tw cublas true -ce eps 0.001 \ -emb 256 -tokens 4096 -blocks 6 -batch 4 -steps 6000 -save path gla4k.mod The trained model can be checkpointed -save path model.mod and then sampled without retraining. Training also writes a model.mod.vocab sidecar holding the character⟷id table, so generation is self-contained and no longer needs the training corpus to reconstruct the insertion-order vocabulary; older checkpoints without a sidecar fall back to -data . Add -gen gpu true to sample on the device through the fast Bluestein/Born forward instead of the CPU reference — seconds rather than minutes for a hybrid/FNet model: ./dp shakespeare -generate true -gen gpu true -model model.mod -prompt "ROMEO:" -gen len 600 -temp 0.8 A pretrained fully-complex Born-rule model Born attention + RoPE + TokenNorm, ~2.1M complex parameters, val 1.52 nats/char on tiny-shakespeare is available as a GitHub release asset https://github.com/crasmarum/CNet/releases/download/v0.1/rope20k.mod and on the Hugging Face Hub https://huggingface.co/crasmarum/born-attention-lm — download it and generate directly: curl -L -o rope20k.mod https://github.com/crasmarum/CNet/releases/download/v0.1/rope20k.mod ./dp shakespeare -generate true -model rope20k.mod -prompt "ROMEO:" -gen len 600 -temp 0.8 The standalone generate example make generate , CPU only, no NCCL samples from any all-Born or all-GLA checkpoint through the streaming decoder of impl/stepnet.h : the prompt is prefilled one token at a time and every generated character costs one state update per layer, independent of how much has been generated. Characters are flushed as they are sampled, so the text appears live. The vocabulary is rebuilt from the training text in insertion order, so -data must be the file the model was trained on for the published checkpoint, the standard 1,115,394-byte tiny-shakespeare with 65 distinct characters : make generate ./generate -model rope20k.mod -data tiny shakespear2.txt -prompt "ROMEO:" -gen len 400 -temp 0.8 ./generate -model rope20k.mod -data tiny shakespear2.txt -prompt "ROMEO:" -gen len 400 -stream false recompute the window per token ./generate -model rope20k.mod -data tiny shakespear2.txt -check 64 \ -prompt "First Citizen: Before we proceed any further, hear me speak." streaming vs window: max |dp| ~ 5e-6 -check K feeds the first K prompt tokens K at most the model's context through both paths and reports the largest difference between their next-token distributions. On a 4-block, 4-head model with a 512-token context, 300 characters take 2.6 s streaming against 101 s through the full-window path. Keep -threads default 8 at or below the physical cores: the per-token layers are tiny and a parallel region per layer per token over a hundred threads is pure overhead. Models with a learned positional table are limited to their training context; when it runs out the generator re-prefills from the last Three trainer flags matter for speed: | flag | effect | |---|---| | -tw cublas true | token-wise layers through batched cuBLAS GEMMs instead of the reference kernel; always use it | | -tw real loss true | with -tw cublas : the token-wise backward computes only the dz GEMMs and writes dz as their conjugate 8 instead of 16 GEMMs, about 38% less backward time in those layers ; exact for a real loss, so use it for language models but not for Jacobian or parity tests | | -init std 0.02 default | every weight, embedding included, gets independent N 0, std² real and imaginary parts. The old init inputs start -init std 0 sets all weights near i/n with n the layer's weight count: layers start near rank one and the CGelu residual stream shrinks fourfold per block. On the 12M PG-19 model it trailed this start by 0.22 nats/byte after 2,000 steps | | -tf32 true | TF32 tensor cores for the cuBLAS GEMMs default strict FP32 . About 1.6x faster per step, but on the PG-19 byte model it trails FP32 by 0.04 validation nats at equal steps over 1,500 steps it is ahead at equal wall-clock time ; opt in deliberately | | -profile steps N | after 20 warm-up steps, time N steps per layer type forward, backward, share and per phase batch, forward, loss, backward, gradient reduce, Adam , print the table and exit | -ksync false skips the host synchronization after every kernel; results are identical, and on the models measured it does not change the step time the GPU is the bottleneck, not launches . ./dp shakespeare -data pg19 train.txt -emb 256 -heads 4 -tokens 4096 -blocks 6 -batch 8 -steps 100 \ -rope true -pos table false -token norm true -tw cublas true -fft fast true -born chunk 512 -gla true -profile steps 10 PHASE / PROFILE / SUBSECTION lines The attention layers compute their within-chunk part with cuBLAS GEMMs scores, decay and causal mask applied elementwise, then weights times values , so their cost no longer depends on per-thread register arrays. Measured on one H100 SXM, PG-19 byte model E=256, 6 blocks, 4 heads, context 4096, batch 8 , ms per training step: | mixer | FP32 | TF32 | |---|---|---| | gated complex mixer -gla true | 329 | 203 | | linear Born + decay -born true -born linear true -born decay true | 383 | 246 | In FP32 the token-wise layers are about 40% of the step: their backward runs 16 real GEMMs because CNet keeps both Wirtinger gradients. Streaming is bounded only by what the model itself remembers of position. A model trained with the default learned positional table -pos table true , one row per context position cannot be fed past its context: the twin has no row to read. With -pos table false the model is RoPE-only the embedding feeds the first block through a TokenNorm , RoPE takes the absolute position in the streaming twin, and the state runs for as long as the input lasts. generate -eval scores a text file through the twin in byte-counted windows: ./generate -model pg19.mod -eval pg19 test.txt -eval offset 0 -eval chars 5242880 -eval window 5242880 -threads 8 EVAL ... scored=5242879 skipped=0 nats/char=... bits/char=... bpb all=... -eval window is the number of bytes between state resets 0 = the training context, the old behaviour . Bytes outside the training vocabulary are not scored and are charged 8 bits each in bpb all . A long file is sharded by whole windows -eval offset and -eval chars as multiples of the window ; the CPU twin runs at a few hundred bytes per second per process. The same recurrence runs on the GPU, N tokens at a time. dp shakespeare -eval stream restores a model, allocates -eval batch windows in parallel and feeds each window as consecutive N -token forwards with the mixer state carried between them GatedLinearAttentionGpu::setCarry ; BornAttentionGpu::setCarry on the density state and the normalizer, restored Born layers being switched to the linear form with chunk -born chunk and RoPE at the absolute byte position; per-position losses are read back, and the state is reset at window boundaries only. ./dp shakespeare -tw cublas true -fft fast true -born chunk 512 -eval stream pg19 test.txt \ -model pg19.mod -eval window 5242880 -eval batch 8 WINDOW k ... nats/char ... bits/char ... then EVAL STREAM ... bits/char=... bpb all=... Pass -tw cublas true without it the token-wise layers run through the slow reference kernel , and -eval batch must be at least 2. On a GH200 a 41 MB test set in eight 5 MB windows takes about three minutes for a 6-block model with the gated mixer and six with linear Born. The GPU pass reproduces the CPU twin exactly for the gated mixer and to within 0.1% of the summed loss for linear Born fp32 chunkwise accumulation order . PG-19 byte models. scripts/pg19 5m/ https://github.com/crasmarum/CNet/blob/main/scripts/pg19 5m fetches PG-19 fetch pg19.sh 1200 : the test and validation partitions and the first 1,200 training books from the public bucket , trains two RoPE-only twins train 5m.sh 8 15000 : E=256, 6 blocks, 4 heads, context 4096, batch 8, 15,000 steps on one GH200, about two hours each; linear Born with the fixed decay, then the gated mixer and scores them score 5m gpu.sh on the GPU, eval5m.sh on the CPU . Scored on the official test partition in 5,242,880-byte windows, the protocol of the MBLM hierarchy Egli et al., 2025 : | model 4.03M complex parameters, 0.49 GB of training bytes | best val nats/char | test, 5M-byte windows bits/byte | validation, 5M windows / 4k resets | |---|---|---|---| | linear Born + fixed decay -born true -born linear true -born decay true -pos table false | 1.121 | 1.656 | 1.690 / 1.696 | | gated complex mixer -gla true -pos table false | 1.148 | 1.700 | 1.732 / 1.736 | | MBLM Mamba-2 + two Transformer stages, 350M parameters, 100 GB of bytes, 5M-byte context | — | 2.448 | — | The carried state neither drifts over five million bytes nor adds much beyond the training context the 5M-window and 4k-reset scores are within 0.006 of each other , and without the positional table the decayed Born twin leads the gated one at every checkpoint. Samples from both twins are in the papers' results. The per-step work is the forward/backward plus a single coalesced NCCL all-reduce of the gradients; the validation loss is identical across GPU counts, confirming correctness. Measured on RTX A6000 CUDA 12.8, NCCL 2.26 , same global batch: | GPUs | ms / step | speed-up | |---|---|---| | 1 | 157 | 1.00x | | 2 | 81 | 1.94x | | 3 | 56 | 2.79x | The current version of the CNet framework has some limitations: - multi-GPU training is data-parallel only no model/tensor parallelism yet ; per-layer kernel-launch overhead on the single-GPU path is eliminated by optional CUDA Graph capture see above ; - adding a layer means hand-writing both its CPU reference and its CUDA kernel wired through GpuMapping / GPU LAYER TABLE ; there is no higher-level CPU→GPU codegen yet. - supported only on Linux/MacOS.