CNet is a C++/CUDA framework for building and researching deep complex valued neural networks, as well as for optimization of complex valued functions using gradient descent with Wirtinger derivatives.
Every built-in layer ships both a CPU reference and a CUDA kernel. Adding a new operator means writing its CPU forward/backward class and a CUDA kernel, then wiring the kernel into the GPU dispatch via GpuMapping (gpu/gpumapping.h) and the GPU_LAYER_TABLE registration macro (gpu/gpu_data.h) — there is no automatic CPU→GPU codegen; the kernel is written once, by hand.
Why are the complex valued neural network (CVNN) interesting? For one, it is a research area that is currently neglected due to the huge commercial success of the real valued ones. This Medium article summarizes well why CVNN might be the future of Deep Learning: "complex numbers work very well for phasic data... The fields where these can be implemented most readily are: Signal Communications; Healthcare (both medical image and ECG); Deep Fake Detections; and Acoustic Analysis for Industrial Maintenance and expansion."
Despite the fact that mathematically the field of complex numbers "in spite of the well-known and obvious identification of with these two are entirely different as far as their vector space structure is concerned."
PyTorch and JAX both support complex tensors and complex-aware autodiff, and for mainstream work they are the right tools — far more mature, faster at scale, multi-GPU, and backed by large ecosystems. CNet does not compete on that axis. It exists because, for research on complex-valued networks specifically, a complex-first, transparent substrate has concrete advantages:
Complex is the default, not a bolted-on dtype. In the mainstream frameworks complex coverage is uneven: many layers, normalizations, activations, initializers and optimizers assume real tensors, so a CVNN is assembled from real building blocks plus hand-written patches whose gradient conventions must be checked case by case. In CNet every variable, layer, loss and the optimizer are complexby construction . #
Wirtinger gradients are explicit, first-class objects. Each layer manipulates the pair$(\partial L/\partial z,\ \partial L/\partial \bar z)$ directly, and the non-holomorphic case (e.g.$|z|^2$ ) is handled in the open rather than hidden behind a conjugation convention. Every layer ships a CPU reference and a GPU kernel checked against each other and against finite differences, so the complex gradient isauditable , not trusted. #
Spectral structure is built in. The DFT, the Hadamard (spectral) product, the inverse DFT and the$\mathrm{conv} = \mathrm{IFFT}(\mathrm{FFT} \cdot \mathrm{FFT})$ equivalence are first-class layers, so problems whose natural domain is the frequency plane map onto the framework directly. #
A physics-native model and loss — the Born rule. This is the sharpest departure from the mainstream frameworks. CNet reads a network's output as a complexamplitude vector$z$ (astate ) and defines the class probabilities by the quantum-measurement (Born) rule$p_k = |z_k|^2 / |z|^2$ , with loss$\mathcal{L}(z, y) = -\sum_k y_k \log\big(|z_k|^2 / |z|^2\big)$ . Classification is ameasurement on an amplitude produced by a cascade of complex — often unitary — operations, rather than real logits passed through a softmax. This inductive bias is natural for physical and wave/field data: an IQ baseband sampleis the complex envelope of a real passband signal, an MRI$k$ -space sampleis a complex Fourier coefficient, and a wavefunctionis a complex amplitude. PyTorch and JAX provide no such head or loss out of the box. #
A small, auditable substrate. The full forward/backward of every layer and its CUDA kernel is explicit and readable, and the GPU execution model — a depth-ordered graph cloned across the batch — is visible and controllable.
In short: use PyTorch or JAX to ship; use CNet to study complex-valued learning itself — to model data in its native complex / amplitude form with a Born-rule measurement head, to prototype operators whose complex gradients you need to see and control, and to work natively in the spectral domain.
For sequence models (text, bytes, Go moves or any other token stream) you write only the model graph and the data; impl/cnet_app.h supplies training on one GPU or many (NCCL), Adam with warm-up and cosine decay, gradient clipping, validation with best-checkpoint saving, and inference.
#include "../impl/cnet_app.h"
struct MyData : app::DataSource { // vocab(), context(), sample(heldout, rng, in, tgt)
...
};
struct MyModel : app::ModelDef { // build(net, vocab, context) -> app::Heads(embedding, loss)
...
};
int main(int argc, char **argv) {
FLAGS::Parse(argc, argv);
MyData data(...); MyModel model(...);
app::Trainer trainer(model, data); // options from standard flags: -steps -lr -batch -save_path ...
return trainer.run();
}
Inference on a saved model:
app::Infer inf;
inf.loadModel("model.mod");
std::vector<int> out = inf.generate(prompt_ids, 300, 0.8, /*seed*/ 1); // or nextProbs(), sample(), score()
Build and run:
make app SRC=examples/char_lm.cpp # multi-GPU build (NCCL=0 for a single-GPU box)
./char_lm -data book.txt -steps 5000 -batch 8 -save_path lm.mod -tw_cublas true -tw_real_loss true
./char_lm -data book.txt -generate true -model lm.mod -prompt "ROMEO:" -gen_len 300
for r in 0 1 2 3; do ./char_lm -data book.txt -batch 24 -world 4 -rank $r -save_path lm.mod & done; wait # 4 GPUs
examples/char_lm.cpp is a complete byte-level language model in about 100 lines, and
examples/README_app.md lists every flag and API call. Fresh weights use a normal
initialization (-init_std 0.02), which trains much better than CNet's older init_inputs() start.
The following code shows how to build a minimal neural net for the MNIST/Fashion MNIST datasets, using the Fourier Transform and Hadamard layers with a Gelu activation function:
CNet cnet;
auto inp = cnet.add(new CInput(OutSize(28 * 28)));
auto fft = cnet.add(new FourierTrans(InSize(28 * 28)), {inp});
auto h_data = cnet.add(new CInput(OutSize(28 * 28)));
auto hdm = cnet.add(new Hadamard(InSize(28 * 28), InSize(28 * 28)), {fft, h_data});
auto gelu = cnet.add(new CGelu(InSize(28 * 28)), {hdm});
auto l_data = cnet.add(new CInput(OutSize(28 * 28 * 10)));
auto lin = cnet.add(new Linear(InSize(28 * 28), InSize(28 * 28 * 10)), {gelu, l_data});
auto outp = cnet.add(new CrossEntropy(InSize(10)), {lin});
Please see the file cnet.cpp for how to train on CPU/GPU the above complex valued neural net with the MNIST/Fashion MNIST datasets.
In order to add a new complex valued function, say CFunc class and implement forward() plus either backward() or, more simply, the two Wirtinger
derivatives dz() and dz_star(), from which CNet builds the backward pass. A layer also
needs a constructor that takes a Uid, a clone() method and a getName(): CNet clones every layer once per batch
entry on the GPU and uses the name when saving and restoring models.
class CSigmoid: public CFunc {
public:
CSigmoid(Uid uid, InSize in_size) : CFunc(uid, in_size, OutSize(in_size.value())) {}
CSigmoid(InSize in_size) : CFunc(in_size, OutSize(in_size.value())) {}
virtual CFunc* clone(Uid uid) { return new CSigmoid(uid, InSize(input().length_)); }
virtual std::string getName() { return "CSigmoid_" + std::to_string(uid_); }
};
While writing an implementation for the forward() method is usually straightforward:
virtual void forward() {
for (int i = 0; i < no_outputs(); ++i) { // one copy of the output per consuming layer
for (int in_indx = 0; in_indx < input().length_; ++in_indx) {
complex<float> g = 1.0f / (1.0f + exp(-input().z(in_indx)));
mutable_output(i)->real_[offset(i) + in_indx] = g.real();
mutable_output(i)->imag_[offset(i) + in_indx] = g.imag();
}
}
}
providing an implementation for the backward() method is in general more difficult. The CNet framework makes things easier whenever you can compute
the Wirtinger derivatives for the corresponding multivariable functions as you can
write implementations for the dz() and the dz_star() methods instead.
For example for the Sigmoid function
while the
From the above observations we can easily implement the dz() and the dz_star() methods:
virtual complex<float> dz(int out_indx, int in_indx) {
if (out_indx != in_indx) {
return 0;
}
auto g = 1.0f / (1.0f + exp(-input().z(in_indx)));
return g * (1.0f - g);
}
virtual complex<float> dz_star(int out_indx, int in_indx) {
return 0;
}
A function that depends on dz_star(). This is
the usual case in practice: the complex sigmoid above has poles at CGelu
instead applies a real function
A quick smoke test is to minimize the squared norm of the Sigmoid outputs with a small net: L2Out computes
CNet net;
auto inp = net.add(new CInput(OutSize(128)));
auto sigm = net.add(new CSigmoid(InSize(128)), {inp});
auto l2 = net.add(new L2Out(InSize(128)), {sigm});
net.init_inputs();
net.init_exec_graph(true);
for (int var = 0; var < 1000; ++var) {
net.forward();
cout << var << "\tLoss: " << ((L2Out*)net[l2])->loss() << endl;
net.backward(0);
net.updateInputs(0.9);
}
Running the above code from the command line:
~/cnet -test_sigmoid true
one can observe that the loss is decreasing:
Depth 0: Input_1 4096,
Depth 1: CSigmoid_2 4096,
Depth 2: L2Out_3 4128,
0 Loss: 32.004
1 Loss: 28.508
2 Loss: 25.422
3 Loss: 22.722
4 Loss: 20.374
5 Loss: 18.338
...
791 Loss: 0.100
...
999 Loss: 0.079
Please see the file examples/sigmoid.h for additional details. A falling loss only shows that
the gradient points downhill; to check dz() and dz_star() themselves, compare the backward pass with central
finite differences of the forward pass, as the finite-difference checks in examples/test_bornattn_mh.cpp do.
Two practical notes:
Cost of the generic backward. When you implement onlydz() anddz_star() , the inheritedbackward() evaluates
them for every (output, input) pair, which is$O(n_{in} \cdot n_{out})$ calls. That is fine for the 128-element example above but far too slow for large activations. For element-wise or otherwise sparse layers, overridebackward() and accumulate the gradient directly, asCGelu andTokenNorm do. #
GPU. A newCFunc runs on the CPU. To run it inside a GPU graph it also needs a GPU mapping with its own
kernels; seeComputation Graph on CUDA and the existing layers undergpu/ .
In order to execute the forward and the backward passes on a CNet complex valued neural net, the functional graph is ordered by depth, as shown in the following diagram:
On an NVIDIA GPU the CNet framework will try to clone the CPU net and execute the forward and the backward passes in parallel following the graph computation depth. In the following image you can see how a batch of size 32 is deployed on a GPU:
Because the clones execute concurrently across the batch, GPU memory grows linearly with the
batch size while the per-step wall-clock time stays essentially flat. The table below measures
this cost for a 6-block complex FNet (flattened token-major stream of T * E = 16384 complex
values) on a single NVIDIA H100 (eager):
| batch | GPU mem (MB) | fwd (ms/step) | bwd (ms/step) |
|---|---|---|---|
| 16 | 167.8 | 0.33 | 0.32 |
| 32 | 335.5 | 0.33 | 0.32 |
| 64 | 671.1 | 0.33 | 0.32 |
| 128 | 1342.2 | 0.33 | 0.32 |
| 256 | 2684.4 | 0.33 | 0.32 |
Memory is linear in the batch size — each clone holds its own activations — while the forward and
backward times per step remain near-constant from batch 16 up to 256, since the clones are launched
level by level and run in parallel. Throughput therefore scales with the batch at roughly fixed
latency. A reduced-memory inference layout (Vars::INFER_DIMS) drops the per-example footprint
further by not allocating the gradient/optimizer buffers that only training needs.
Because the clone arena is static across steps — device pointers for activations, weights, and
Wirtinger gradients never move after allocateOnGpu — the entire level-by-level launch schedule
can be captured once into a CUDA graph and replayed with a single cudaGraphLaunch
(CNet::GpuForwardGraphed / GpuBackwardGraphed). This removes the per-kernel launch/sync
overhead: on an H100 the forward+backward step drops from 0.65 ms to 0.32 ms (~2×, flat across
batch) while staying bit-identical to eager execution, and the captured step runs faster than
eager torch.complex64. The standalone bench_clone example reproduces this:
make bench_clone
./bench_clone -fft_fast true # eager
./bench_clone -fft_fast true -bc_graph 1 # replay captured CUDA graphs
./bench_clone -fft_fast true -bc_check 1 # verify graph replay == eager (bit-identical)
The Input Layer is used as the main input for the neural network as well as the input parameters for other layers, e.g., Linear or Hadamard layers.
For example, in the following code snippet inp is the main input and the h_data is the parameter for the Hadamard function.
#include "impl/cinput.h"
CNet cnet;
auto inp = cnet.add(new CInput(OutSize(28 * 28)));
auto h_data = cnet.add(new CInput(OutSize(28 * 28)));
auto hdm = cnet.add(new Hadamard(InSize(28 * 28), InSize(28 * 28)), {inp, h_data});
The main variable for this layer is the output size, e.g., CInput(OutSize(1024)). As the main input of a neural net, on CPU you can set its complex values
from a batch using this method:
setInput(InputBatch& batch, int b_indx)
As the input parameter for other layers, its values are either randomly set at the beginning of training a network, e.g., net.init_inputs() (for deep residual models prefer net.normalInit(0.02, seed), see the -init_std flag below),
or are restored from a model via net.restore(std::string file).
See the file cnet.cpp for more details.
The Embedding Layer is used only as the main input for the neural network. The main variables of an embedding layer are
the embedding dimension, the number of embeddings and the maximum number of input tokens. For example, in the following
code snippet we create an embedding consisting of no_embedings = 100 tokens of dimension emb_dim = 300 that has an input
of maximum max_in_tokens = 64 tokens and has an output of emb_dim * max_no_tokens = 300 * 64 complex numbers. Note that the output is
padded with zeros if necessary.
#include "impl/embed.h"
int emb_dim = 300;
int max_in_tokens = 64;
int no_embedings = 100;
CNet cnet;
auto emb = cnet.add(new CEmbedding(emb_dim, max_in_tokens, no_embedings));
auto fft = cnet.add(new FourierTrans(InSize(emb_dim * max_in_tokens)), {emb});
On a CPU you can set the output embedding values using a vector<int> data structure, e.g.,
embedding.setInput({2, 1, 1, 3});
while on GPU you can do it via the
net.batchToGpu(InputFunc *inp, OutputFunc *outp, Batch *batch);
method. The initial complex values for the embeddings are either randomly set at the beginning of training a network, e.g., net.init_inputs(),
or are restored from a model via net.restore(std::string file).
This layer implements the Discrete Fourier Transform function
The main variable of this function is its input/output size. Example of its usage:
#include "impl/ft.h"
CNet cnet;
auto inp = cnet.add(new CInput(OutSize(28 * 28)));
auto fft = cnet.add(new FourierTrans(InSize(28 * 28)), {inp});
This layer implements the Inverse Discrete Fourier Transform, the exact (unitary) inverse of FourierTrans:
It is typically used to bring a Hadamard product back to the time domain, i.e. to realise a circular convolution as
#include "impl/ft.h"
auto fft = cnet.add(new FourierTrans(InSize(28 * 28)), {inp});
auto hdm = cnet.add(new Hadamard(InSize(28 * 28), InSize(28 * 28)), {fft, h_data});
auto ifft = cnet.add(new InverseFourierTrans(InSize(28 * 28)), {hdm});
This layer implements a causal variant of the DFT: a lower-triangular-masked Fourier sum
This makes it a parameter-free causal token mixer — the autoregressive analogue of self-attention used in FNet-style language models: it mixes information across tokens while never letting a position see its own future. With a token-major CEmbedding layout (token -fft_fast), which evaluates the causal transform via a Bluestein/chirp-z factorisation with batched cuFFT (forward and Wirtinger backward).
#include "impl/ft.h"
auto emb = cnet.add(new CEmbedding(emb_dim, max_in_tokens, no_embedings));
auto mix = cnet.add(new TriangFourier(InSize(emb_dim * max_in_tokens)), {emb});
This is a content-based causal token mixer: the complex analogue of single-head self-attention, but with the attention scores given by the quantum-measurement (Born) rule instead of a softmaxed dot product. The input is the concatenation of three complex blocks
The score TokenwiseLinear projections of the stream:
#include "impl/bornattn.h"
int N = 64, E = 128, d = E / 2; // N tokens, width E, head dim d
auto Q = cnet.add(new TokenwiseLinear(N, E, d), {pre, wQ});
auto K = cnet.add(new TokenwiseLinear(N, E, d), {pre, wK});
auto V = cnet.add(new TokenwiseLinear(N, E, d), {pre, wV});
auto att = cnet.add(new BornAttention(N, d), {Q, K, V}); // input [Q;K;V] -> O
BornAttention(N, d, causal, eps, heads) splits the heads heads of heads = 1 is the single-head operator above. RotaryEmbed(N, d, base, heads) restarts the frequency ladder every heads > 1 (or a non-zero eps) use an extended record; old checkpoints restore unchanged. Smaller heads matter for the linear mode below: its state and per-token cost scale with
Because the score is a squared inner product it has an exact finite feature map: lifting each vector to its outer product setLinear(true, chunk) switches the GPU kernels to this form: tokens are processed in chunks of chunk (N % chunk == 0), each chunk mixes with itself through the quadratic kernel and with everything before it through the carried state, and the state advances by one cuBLAS batched GEMM per chunk. The backward is exact too (query gradients read the stored per-chunk forward state; key and value gradients a suffix state swept in reverse). The CPU operator stays quadratic and is the oracle: make test_bornattn_mh && ./test_bornattn_mh checks the linear forward and all input gradients against it on eight configurations, together with finite differences, a head-slicing oracle, serialization and GPU/CPU parity.
auto att = cnet.add(new BornAttention(N, d, /*causal=*/true, /*eps=*/0.f, /*heads=*/4), {Qr, Kr, V});
((BornAttention*) cnet[att])->setLinear(true, /*chunk=*/256); // GPU: exact O(N d_h^2) form
The same operator, so the same model: a run switched between the two forms reaches the same loss (within the quadratic kernel's own run-to-run spread, which comes from its atomic backward). Attention layer alone,
| N | quadratic | linear (chunk 256) | speed-up |
|---|---|---|---|
| 1k | 108 ms, 0.25 GB | 20 ms | 5.5x |
| 4k | 692 ms, 1.4 GB | 78 ms | 8.9x |
| 16k | 11.3 s, 14.9 GB | 310 ms, 6.1 GB | 37x |
| 64k | out of memory | 1.24 s, 12.6 GB | --- |
Constraints: causal attention only, N % chunk == 0, gpu/born_linear_cnet.md.
setDecay(true) (trainer flag -born_decay true) multiplies the score by make test_bornattn_mh). In the controlled PyTorch study of the companion paper the decay was the single most effective fix for long contexts (at 128k it recovered 0.66 of the 0.70 nats between plain Born and gated linear attention).
The running state also gives constant-memory autoregressive decoding: one token in, one token out, the state updated in place, never touching past tokens. CNet graphs are static, so the decoder is a second graph with one token per step, built from a restored model by impl/stepnet.h: the same layers and wiring, every per-token size divided by RotaryEmbed given an absolute position (setPos), and BornAttention in step mode (setStepMode(true), forwardStep). Fed the tokens one at a time it reproduces the fixed-window model's next-token distribution to TokenwiseLinear, TokenNorm, RotaryEmbed, BornAttention, GatedLinearAttention, Residual, CGelu, SequenceCrossEntropy; Fourier mixers and the SoftMax block norm have no constant-state form and are rejected. See the generate example below.
The degree-one counterpart of Born attention, added to compare the two inside CNet with everything else held fixed. Where Born attention routes with the probability amplitude BornAttention (heads heads of
a RotaryEmbed on TokenNorm normally follows the layer. The current token is never gated. The inner product uses the same convention as BornAttention, setStepMode(true) / forwardStep() give the one-token recurrence for the streaming decoder.
#include "impl/gla.h"
int N = 4096, E = 256, d = E / 2, heads = 4;
auto Q = cnet.add(new TokenwiseLinear(N, E, d), {pre, wQ}); // then RotaryEmbed(N, d, 10000.0, heads) on Q and K
auto K = cnet.add(new TokenwiseLinear(N, E, d), {pre, wK});
auto V = cnet.add(new TokenwiseLinear(N, E, d), {pre, wV});
auto G = cnet.add(new TokenwiseLinear(N, E, heads), {pre, wG}); // per-head gate logits
auto gla = cnet.add(new GatedLinearAttention(N, d, heads), {Q, K, V, G});
((GatedLinearAttention*) cnet[gla])->setChunk(512); // GPU chunk length (N % chunk == 0)
auto att = cnet.add(new TokenNorm(N, d), {gla}); // output norm: the mix has no normaliser
On the GPU the layer runs chunkwise like the linear Born mode: the inter-chunk readout and the state update are complex GEMMs (four real cuBLAS GEMMs on the re/im planes), one kernel forms the intra-chunk amplitudes, and the backward sweeps the chunks in reverse with a suffix state. Constraints: causal only, N % chunk == 0, make test_gla && ./test_gla runs the finite-difference, recurrence, serialisation and GPU-parity checks. On the PG-19 character model (6 blocks, -born_decay true); the gated kernel is a first version (942 vs 815 and 835 ms per step). A controlled repeat of the same architecture in the PyTorch mirror puts the gated mixer level with Born plus a fixed per-head decay (1.230 vs 1.210 nats/char; plain Born 1.370), so its gain is forgetting; CNet's plain Born trailing its mirror is under investigation. Born attention remains CNet's recommended mixer; the gated layer is an option. Design, derivation and measurements: gpu/gla_cnet.md.
RotaryEmbed applies a position-dependent complex phase rotation to each token's features — rotary position embeddings (RoPE) expressed as what they natively are in a complex network: a rotation. The input is
Applied to the queries and keys of BornAttention, the per-feature inner product relative phase
#include "impl/rotary.h"
int N = 64, dh = 64;
auto Qr = cnet.add(new RotaryEmbed(N, dh), {Q}); // rotate queries
auto Kr = cnet.add(new RotaryEmbed(N, dh), {K}); // rotate keys
auto att = cnet.add(new BornAttention(N, dh), {Qr, Kr, V});
This layer implements the Hadamard function which is simply the element-wise multiplication:
In the complex valued neural network world, the Hadamard Layer is equivalent with the Convolution layer, because the Fourier Transform famously commutes with the convolution: On the Equivalence of Convolutional and Hadamard Networks using DFT.
You can see an example of using the Hadamard layer hdm with some parameters provided by an Input Layer h_data in the following code snippet:
#include "impl/hadamard.h"
CNet cnet;
auto inp = cnet.add(new CInput(OutSize(28 * 28)));
auto fft = cnet.add(new FourierTrans(InSize(28 * 28)), {inp});
auto h_data = cnet.add(new CInput(OutSize(28 * 28)));
auto hdm = cnet.add(new Hadamard(InSize(28 * 28), InSize(28 * 28)), {fft, h_data});
This layer implements the Residual function which computes the element-wise addition:
You can see an example of using the Residual layer res in the following code snippet:
#include "impl/residual.h"
CNet cnet;
//...
auto fft = cnet.add(new FourierTrans(InSize(28 * 28)), {inp});
//...
auto hdm = cnet.add(new Hadamard(InSize(28 * 28), InSize(28 * 28)), {fft, h_data});
auto res = cnet.add(new Residual(InSize(28 * 28), InSize(28 * 28)), {fft, hdm});
This layer is the equivalent of the fully connected / dense layer and is performing a matrix multiplication:
You can see an example of using the Linear layer lin in the following code snippet:
#include "impl/linear.h"
CNet cnet;
// ...
auto l_data = cnet.add(new CInput(OutSize(512 * 10)));
auto lin = cnet.add(new Linear(InSize(512), InSize(512 * 10)), {gelu, l_data});
auto outp = cnet.add(new CrossEntropy(InSize(10)), {lin});
This is a position-wise linear layer: the input is seen as same
Sharing the weight across positions costs Linear over the flattened sequence — this is what makes a per-token feed-forward block affordable in a sequence model. The constructor takes the number of tokens and the input/output embedding widths:
#include "impl/tokenwise.h"
int N = 64, E = 128, H = 4 * E; // N tokens, width E, hidden H
auto w1 = cnet.add(new CInput(OutSize(E * H)));
auto ff = cnet.add(new TokenwiseLinear(N, E, H), {hidden, w1});
On the GPU, TokenwiseLinear has two kernel paths: a naive per-element complex multiply-accumulate, and a batched cuBLAS GEMM path (the complex product is four real cublasSgemmBatched calls). The batched path is a large win once the projection is wide — most visibly a language-model head at vocabulary width — where it measured ≈3.4× faster (607 → 177 ms/step) on a vocab-8192, N=256, E=256 model on an H100; at small output widths (e.g. a character vocabulary) the two paths are indistinguishable. The dp_shakespeare example selects it with -tw_cublas true.
This activation function is the equivalent of Relu:
You can see an example of using the CRelu layer rel in the following code snippet:
#include "impl/relu.h"
CNet cnet;
// ...
auto inp = cnet.add(new CInput(OutSize(512)));
auto rel = cnet.add(new Crelu(InSize(512)), {inp});
This activation function is the equivalent of Gelu:
You can read more about Gaussian Error Linear Units (GELUs).
You can see an example of using the CGelu layer rel in the following code snippet:
#include "impl/relu.h"
CNet cnet;
// ...
auto inp = cnet.add(new CInput(OutSize(512)));
auto rel = cnet.add(new CGelu(InSize(512)), {inp});
Despite its name, this layer is the complex
Unlike CGelu, this activation is scale-covariant and commutes with the (unitary) Fourier Transform, so it can be used inside a spectral network without breaking the FFT/convolution equivalence — useful as the nonlinearity between spectral mixing layers.
#include "impl/softmax.h"
auto inp = cnet.add(new CInput(OutSize(512)));
auto nrm = cnet.add(new SoftMax(InSize(512)), {inp});
Where SoftMax normalizes a whole block by its global Euclidean norm, TokenNorm normalizes each token independently by its own root-mean-power — the complex analogue of (gain-free) RMSNorm/LayerNorm, and the per-token counterpart of SoftMax. The input is
A learnable affine is intentionally omitted (a following TokenwiseLinear supplies gain and bias), which keeps the layer parameter-free and its Wirtinger backward exact. With {dd},T/(2 d r^3)$ and $\partial L/\partial \bar{x}{dd} = (\partial L/\partial \bar{y}{dd})/r - x{dd},T/(2 d r^3)$ (verified: CPU finite-difference and GPU-vs-CPU to machine precision). Used as a pre-norm (normalize the residual-stream input to each sub-block), it lets deep complex transformers train stably — on tiny-shakespeare it breaks the plateau that global block-normalization hits. The constructor takes the number of tokens, the per-token feature width, and an optional
#include "impl/tokennorm.h"
int N = 64, E = 128;
auto normed = cnet.add(new TokenNorm(N, E), {last}); // per-token RMS pre-norm
auto Q = cnet.add(new TokenwiseLinear(N, E, E / 2), {normed, wQ});
ComplexDropout is inverted dropout for complex activations: at training time each element is kept with probability inference it is the identity.
Since the map is multiplication by a real per-element constant not stored — it is a deterministic hash of (seed, step, element) recomputed identically in the forward and backward kernels, and the GPU mapping bumps step each forward for a fresh mask (the seed folds in the layer depth so stacked dropout layers are independent). The CPU path is the identity, so generation/evaluation are unaffected. The constructor takes the input size and the drop probability:
#include "impl/dropout.h"
int N = 64, E = 128;
auto sub = cnet.add(new TokenwiseLinear(N, E, E), {g, w}); // a sub-layer output
auto drop = cnet.add(new ComplexDropout(InSize(N * E), 0.1f), {sub});
Caveat for the Born-rule loss. Dropout is a conventional regularizer, but the Born-rule SequenceCrossEntropy has a unigram saddle whose escape is numerically knife-edge (it is tipped by the summation order of the TokenwiseLinear GEMM — see the language-model example). Empirically, any stochastic perturbation — dropout or gradient noise — suppresses that escape: on tiny-shakespeare / sub-word data a model that learns fine with dropout off gets pinned at the unigram plateau with -dropout 0.1. So for Born-rule language-model training, leave dropout off; it is provided for conventional (real-logit CrossEntropy) networks and as a documented research knob.
ConjLagProduct computes the instantaneous autocorrelation at a fixed lag tau:
It is the nonlinear, non-holomorphic primitive behind cyclostationary and higher-order modulation features: the time-mean of the output estimates the lag-tau autocorrelation R(tau) (bounded |R| <= 1 for power-normalised input, so well-scaled), while an FFT of the output over time gives the cyclic spectrum (spectral lines at the symbol-rate / carrier cycle frequencies). It is the complex, two-input generalisation of CModulus2 (|z|^2 = x*conj(x) is the tau = 0 case). The Wirtinger backward (verified to ~1e-4 against central finite differences on CPU, and exactly against the CUDA kernel) differentiates y = a*conj(b) through both inputs a = x_t, b = x_{t-tau}. The constructor takes the input length and the lag:
#include "impl/conjlag.h"
int N = 128, tau = 8;
auto r = cnet.add(new ConjLagProduct(InSize(N), tau), {x}); // x_t * conj(x_{t-tau})
auto Rt = cnet.add(new MeanPool(InSize(N), OutSize(4)), {r}); // pooled autocorrelation R(tau)
This is the a-priori signal structure the pooled-power features miss. On held-out RML2016.10a (11-way modulation), adding a small bank of ConjLagProduct lags (R(tau) and |R(tau)|^2) to the arch-11 spectral-conv classifier raised accuracy from 0.340 to 0.363 under an identical recipe — a modest but real gain in the predicted direction (separating the same-band QAM16/QAM64 and 8PSK/QPSK confusions that linear/power features cannot). A note on scaling: read out the time-mean R(tau) (O(1)), not the raw |FFT(r_tau)|^2 (energy ~N), which otherwise saturates the Born-rule logits.
The full cyclostationary front end adds a SoftMax-normalised cyclic spectrum on top of this: SoftMax(FFT(r_tau)), an O(1) read-out of the spectral-line pattern that keeps the Born logits well-scaled. On the artifact-corrected RML22 benchmark this cyclostationary classifier reaches ~0.50 overall — on par with the VT-CNN2 real-valued baseline (held-out RML2016.10a ~0.49), closing most of the pooled-power gap.
SegmentedFourier is a short-time Fourier transform expressed as a block-diagonal DFT: the length-N input is split into N/window contiguous windows of length window, and each window is DFT'd independently (unitary, 1/sqrt(window) scaling). Output index b*w+p (window b, frequency bin p):
With a following CModulus2 and MeanPool it yields a spectrogram (N/window time steps × window frequency bins) — a time–frequency feature that keeps local temporal structure the global FourierTrans averages away. It is the exact (unitary) FourierTrans when window = N; its Wirtinger backward mirrors FourierTrans per window and is verified to ~2e-4 against central finite differences on CPU, and exactly against the CUDA kernel (roots are computed on the fly in the kernel). The constructor takes the input length and the window length (which must divide it):
#include "impl/segfourier.h"
int N = 128, window = 32;
auto stft = cnet.add(new SegmentedFourier(InSize(N), window), {x}); // 4 windows x 32 bins
auto mag = cnet.add(new CModulus2(InSize(N)), {stft}); // |.|^2 spectrogram
Empirically, on RML a segmented-Fourier branch (windows {16, 32}) gave no gain over the cyclic spectrum above — the two features are redundant — which we record as an honest negative. It is provided as a general-purpose spectral/time–frequency primitive.
This loss function is simply the square of the
You can see an example of using the L2Out loss function l2 in the following code snippet:
#include "impl/l2out.h"
CNet net;
auto inp = net.add(new CInput(OutSize(128)));
auto sigm = net.add(new CSigmoid(InSize(128)), {inp});
auto l2 = net.add(new L2Out(InSize(128)), {sigm});
The CrossEntropy loss implements the multivariable function
You can read more on it in the On the Equivalence of Convolutional and Hadamard Networks using DFT research paper.
You can see an example of using the CrossEntropy loss function ce in the following code snippet, as well as in the
cnet.cpp file:
#include "impl/crossent.h"
CNet cnet;
// ...
auto l_data = cnet.add(new CInput(OutSize(28 * 28 * 10)));
auto lin = cnet.add(new Linear(InSize(28 * 28), InSize(28 * 28 * 10)), {gelu, l_data});
auto ce = cnet.add(new CrossEntropy(InSize(10)), {lin});
This is the autoregressive analogue of CrossEntropy. The input is
The per-position targets (the input tokens shifted left by one) are set with setTargets(std::vector<int>) before each forward pass. Combined with a token-major CEmbedding, a causal TriangFourier mixer and TokenwiseLinear feed-forward blocks, this trains a character-level language model where every position predicts its next token in a single pass.
#include "impl/seqcrossent.h"
int N = 64, vocab = 65;
auto logits = cnet.add(new TokenwiseLinear(N, emb_dim, vocab), {last, w_out});
auto seq = cnet.add(new SequenceCrossEntropy(InSize(N * vocab), vocab), {logits});
// ... each step:
seq->setTargets(targets); // per-position next-token targets
The learnable CInput / CEmbedding parameters are trained with the Wirtinger gradients accumulated during the backward pass. Three update rules are available, each on both CPU and GPU:
- Plain gradient descent —
updateInputs(l_rate)(CPU) /gpuUpdateInputs(l_rate)(GPU). - Momentum Adam (first moment only) —
adamUpdate(...). - Full Adam (first + second moment with bias correction, and optional gradient clip-by-value) —
trueAdamUpdate(...).
On GPU a batch is trained as a set of net clones, so the update is called once per step with the learning rate scaled by the batch size (the per-clone gradients are summed onto the ancestor parameters first):
// plain SGD
cnet.gpuUpdateInputs(l_rate / batch.size());
// momentum Adam
cnet.adamUpdate(l_rate / batch.size(), beta, step);
// full Adam with gradient clipping (grad_clip = 0 disables the clip)
cnet.trueAdamUpdate(l_rate / batch.size(), beta1, beta2, eps, step, grad_clip);
On CPU, where gradients are accumulated over several samples before an update, the Adam variants additionally take that accumulation batch size:
net.adamUpdate(l_rate, batch_size, beta, step);
net.trueAdamUpdate(l_rate, batch_size, beta1, beta2, eps, step);
Because the Born-rule CrossEntropy / SequenceCrossEntropy losses are ill-conditioned, full Adam (trueAdamUpdate) with gradient clipping is recommended over plain SGD for classification and language-model training.
The ill-conditioning is concrete: the exact gradient of correct token, so a single hard example can produce an explosive update that parameter-level gradient clipping cannot catch (the bad gradient is already summed into the minibatch). SequenceCrossEntropy therefore offers an optional probability floor setEps(eps) (the -ce_eps flag in the language-model example): the target modulus is clamped from below, in the gradient only — a uniform measurement-noise floor, i.e. label-smoothing for the Born rule. The forward loss stays exact, so the reported metric is unchanged; only the optimization is regularised, and eps = 0 is bit-identical to the exact gradient. A value around
CNet is currently supported only on Linux/MacOS and you can build it using make. The Makefile checks for the presence of the NVIDIA CUDA Compiler (nvcc) and if nvcc is found, it compiles the project with CUDA support. Otherwise, it falls back to using g++.
git clone https://github.com/crasmarum/CNet.git
cd CNet
make
With nvcc the build emits native code for Ampere (sm_80, A100) and Hopper ( sm_90, H100/H200/GH200) plus compute_90 PTX that newer GPUs JIT-compile. Override the targets with GPU_ARCH, for example make GPU_ARCH="-gencode arch=compute_100,code=sm_100" for Blackwell. Example programs (make dp_shakespeare, make generate, make radioml, the tests) are rebuilt when any header they include changes; if a binary misbehaves after a large edit, rm -rf build/obj and rebuild.
If the NVIDIA CUDA Compiler is found, the make file will automatically compile CNet with CUDA support. For example, once you download locally the MNIST/Fashion MNIST datasets, you can train the example neural network from the command line like follows.
~/cnet -mnist_images ~/train-images.idx3-ubyte -mnist_labels ~/train-labels.idx1-ubyte \
-model_path ~/test.mod -mnist_gpu_train true
You can change the default learning rate and the GPU batch size via the l_rate and batch_size command line flags:
~/cnet -mnist_images ~/train-images.idx3-ubyte -mnist_labels ~/train-labels.idx1-ubyte \
-model_path ~/test.mod -mnist_gpu_train true -l_rate 0.001 -batch_size 40
You can find the example training code in the cnet.cpp file:
MnistDataReader reader;
assert(reader.Open(mnist_images, mnist_labels, 60000));
assert(reader.readData());
CNet cnet;
assert(cnet.hasGPU());
int cinp, coutp;
createMnistNet(cnet, &cinp, &coutp);
std::cout << cnet.cpuNet().toString() << std::endl;
CInput *inp = (CInput*)cnet[cinp];
CrossEntropy *cen = (CrossEntropy*)cnet[coutp];
cnet.init_inputs();
assert(cnet.allocateOnGpu(batch_size));
std::cout << std::setprecision(4) << "L_rate: " << l_rate
<< "\t batch_size: " << batch_size << std::endl;
float avg_loss = 0;
for (int epoch = 0; epoch < no_epochs; ++epoch) {
reader.shuffle();
for (int time = 0; time < reader.size() / batch_size; ++time) {
auto batch = reader.nextBatch(batch_size);
cnet.gpuForward(inp, cen, batch);
avg_loss += cnet.getLoss(cen, &batch);
if (time % 100 == 99) {
std::cout << epoch << "\t" << time << "\t loss: " << (avg_loss / 100) << std::endl;
avg_loss = 0;
}
if (time % 1000 == 999) {
if (!cnet.getInputsFromGpu()) {
std::cerr << "Warning: could not get data from GPU." << std::endl;
} else {
cnet.save(model_path);
std::cout << "Model saved at: " << model_path << std::endl;
}
}
cnet.gpuBackward();
cnet.gpuUpdateInputs(l_rate / batch.size());
}
}
This loop is deliberately minimal: plain gradient descent (gpuUpdateInputs) and CNet's original init_inputs()
start, which is enough for this shallow network. For deeper models use Adam (trueAdamUpdate, as dp_shakespeare
does) and the normal initialization (cnet.cpuNet().normalInit(0.02, seed) after init_inputs(); the -init_std
flag in dp_shakespeare, gollm and app::Trainer), which trains much better. For
token-sequence models, app::Trainer packages the whole loop:
batching, Adam with warm-up and cosine decay, clipping, validation, checkpoints and multi-GPU.
CNet trains across multiple GPUs with data parallelism, using the same clone mechanism it uses to batch within one GPU — only one level up. Each GPU holds a full copy of the network; the global batch is split across GPUs; after each backward the per-GPU gradients are summed with an NCCL all-reduce; and every replica applies the identical optimizer step, so the replicas stay in sync and each step is the same update as single-GPU training with the full batch, up to the order of floating-point summation.
Two pieces make this work (see gpu/gpu_func.h): ancestorGradBuffers() exposes
each parameter's gradient buffer for an in-place all-reduce, and the replicas must
start from the same weights. dp_shakespeare and app::Trainer initialize every rank
with the same seed; collectAncestorParams() / loadAncestorParams() can broadcast
weights instead.
The standard DDP architecture — one process per GPU — avoids serializing CNet's many kernel launches through a single CUDA context, and scales near-linearly. A complete, self-contained example trains a complex FNet character-level language model on tiny-shakespeare:
make dp_shakespeare
./dp_shakespeare -world 1 -batch 48 -steps 3000 -data tiny_shakespear2.txt
./examples/run_dp_shakespeare.sh 4 -batch 48 -steps 3000
By default the example uses the parameter-free TriangFourier causal mixer. Flags let it assemble a fully complex transformer instead: -born true swaps in the BornAttention content-based mixer (with TokenwiseLinear Q/K/V/O projections), -rope true adds RotaryEmbed rotary position embeddings on the queries/keys, -token_norm true adds per-token TokenNorm pre-normalization (pre-LN), and -ce_eps 0.001 enables the Born-loss probability floor for stable deep training:
./examples/run_dp_shakespeare.sh 4 -born true -rope true -token_norm true -ce_eps 0.001 \
-emb 160 -tokens 64 -blocks 8 -batch 64 -steps 8000 -grad_clip 0.5
-val_data <file> takes a separate held-out text file, and then all of -data is training data; without it the last 10% of -data is held out. The character corpus is memory-mapped (one byte per character, 64-bit positions, one page-cache copy shared by all ranks on a node), so multi-gigabyte corpora such as the full PG-19 training set (11.7 GB) load in seconds with a few GB of host RAM. -born_decay true adds the fixed per-head decay described above (recommended for contexts beyond a few thousand tokens); -pos_table false drops the learned positional table (RoPE only), which is what lets the trained model stream past its context (see below). Two more flags select the multi-head and linear forms of the mixer: -heads 4 splits the -born_linear true -born_chunk 256 runs the exact chunkwise linear kernels (GPU only; -tokens must be a multiple of -born_chunk). With them the same model trains at contexts the quadratic kernel cannot reach: at -tokens 4096 -batch 8 the quadratic kernel runs out of memory on a 40 GB A100 while the linear form steps in 0.78 s, and at -tokens 1024 the whole training step is 6.4x faster (0.21 s vs 1.35 s). Everything else — RoPE, TokenNorm, the Born read-out, checkpoints — is unchanged, and a checkpoint trained one way can be used the other way.
./examples/run_dp_shakespeare.sh 1 -born true -heads 4 -born_linear true -born_chunk 256 \
-rope true -token_norm true -tw_cublas true -ce_eps 0.001 \
-emb 256 -tokens 4096 -blocks 4 -batch 8 -steps 3000 -save_path born4k.mod
-gla true replaces the Born mixer in every attention block by the gated linear attention over complex amplitudes (GatedLinearAttention with a TokenwiseLinear gate projection and a TokenNorm after the mix; -heads, -rope, -token_norm and -born_chunk apply as before; GPU chunkwise kernels, so -tokens must be a multiple of -born_chunk). Checkpoints trained either way serialise, restore and stream through generate unchanged:
./examples/run_dp_shakespeare.sh 1 -gla true -heads 4 -born_chunk 512 \
-rope true -token_norm true -tw_cublas true -ce_eps 0.001 \
-emb 256 -tokens 4096 -blocks 6 -batch 4 -steps 6000 -save_path gla4k.mod
The trained model can be checkpointed (-save_path model.mod) and then sampled without retraining. Training also writes a model.mod.vocab sidecar holding the character⟷id table, so generation is self-contained and no longer needs the training corpus to reconstruct the (insertion-order) vocabulary; older checkpoints without a sidecar fall back to -data. Add -gen_gpu true to sample on the device through the fast Bluestein/Born forward instead of the CPU reference — seconds rather than minutes for a hybrid/FNet model:
./dp_shakespeare -generate true -gen_gpu true -model model.mod -prompt "ROMEO:" -gen_len 600 -temp 0.8
A pretrained fully-complex Born-rule model (Born attention + RoPE + TokenNorm, ~2.1M complex parameters, val 1.52 nats/char on tiny-shakespeare) is available as a GitHub release asset and on the Hugging Face Hub — download it and generate directly:
curl -L -o rope20k.mod https://github.com/crasmarum/CNet/releases/download/v0.1/rope20k.mod
./dp_shakespeare -generate true -model rope20k.mod -prompt "ROMEO:" -gen_len 600 -temp 0.8
The standalone generate example (make generate, CPU only, no NCCL) samples from any all-Born or all-GLA checkpoint through the streaming decoder of impl/stepnet.h: the prompt is prefilled one token at a time and every generated character costs one state update per layer, independent of how much has been generated. Characters are flushed as they are sampled, so the text appears live. The vocabulary is rebuilt from the training text in insertion order, so -data must be the file the model was trained on (for the published checkpoint, the standard 1,115,394-byte tiny-shakespeare with 65 distinct characters):
make generate
./generate -model rope20k.mod -data tiny_shakespear2.txt -prompt "ROMEO:" -gen_len 400 -temp 0.8
./generate -model rope20k.mod -data tiny_shakespear2.txt -prompt "ROMEO:" -gen_len 400 -stream false # recompute the window per token
./generate -model rope20k.mod -data tiny_shakespear2.txt -check 64 \
-prompt "First Citizen: Before we proceed any further, hear me speak." # streaming vs window: max |dp| ~ 5e-6
-check K feeds the first K prompt tokens (K at most the model's context) through both paths and reports the largest difference between their next-token distributions. On a 4-block, 4-head model with a 512-token context, 300 characters take 2.6 s streaming against 101 s through the full-window path. Keep -threads (default 8) at or below the physical cores: the per-token layers are tiny and a parallel region per layer per token over a hundred threads is pure overhead. Models with a learned positional table are limited to their training context; when it runs out the generator re-prefills from the last
Three trainer flags matter for speed:
| flag | effect |
|---|---|
-tw_cublas true |
token-wise layers through batched cuBLAS GEMMs instead of the reference kernel; always use it |
-tw_real_loss true |
with -tw_cublas : the token-wise backward computes only the dz* GEMMs and writes dz as their conjugate (8 instead of 16 GEMMs, about 38% less backward time in those layers); exact for a real loss, so use it for language models but not for Jacobian or parity tests |
-init_std 0.02 (default) |
every weight, embedding included, gets independent N(0, std²) real and imaginary parts. The old init_inputs() start (-init_std 0 ) sets all weights near i/n with n the layer's weight count: layers start near rank one and the CGelu residual stream shrinks fourfold per block. On the 12M PG-19 model it trailed this start by 0.22 nats/byte after 2,000 steps |
-tf32 true |
TF32 tensor cores for the cuBLAS GEMMs (default strict FP32). About 1.6x faster per step, but on the PG-19 byte model it trails FP32 by 0.04 validation nats at equal steps over 1,500 steps (it is ahead at equal wall-clock time); opt in deliberately |
-profile_steps N |
after 20 warm-up steps, time N steps per layer type (forward, backward, share) and per phase (batch, forward, loss, backward, gradient reduce, Adam), print the table and exit |
-ksync false skips the host synchronization after every kernel; results are identical, and on the models measured it does not change the step time (the GPU is the bottleneck, not launches).
./dp_shakespeare -data pg19_train.txt -emb 256 -heads 4 -tokens 4096 -blocks 6 -batch 8 -steps 100 \
-rope true -pos_table false -token_norm true -tw_cublas true -fft_fast true -born_chunk 512 -gla true -profile_steps 10
The attention layers compute their within-chunk part with cuBLAS GEMMs (scores, decay and causal mask applied elementwise, then weights times values), so their cost no longer depends on per-thread register arrays. Measured on one H100 SXM, PG-19 byte model (E=256, 6 blocks, 4 heads, context 4096, batch 8), ms per training step:
| mixer | FP32 | TF32 |
|---|---|---|
gated complex mixer ( -gla true ) |
329 | 203 |
linear Born + decay ( -born true -born_linear true -born_decay true ) |
383 | 246 |
In FP32 the token-wise layers are about 40% of the step: their backward runs 16 real GEMMs because CNet keeps both Wirtinger gradients.
Streaming is bounded only by what the model itself remembers of position. A model trained with the default learned positional table (-pos_table true, one row per context position) cannot be fed past its context: the twin has no row to read. With -pos_table false the model is RoPE-only (the embedding feeds the first block through a TokenNorm), RoPE takes the absolute position in the streaming twin, and the state runs for as long as the input lasts. generate -eval scores a text file through the twin in byte-counted windows:
./generate -model pg19.mod -eval pg19_test.txt -eval_offset 0 -eval_chars 5242880 -eval_window 5242880 -threads 8
-eval_window is the number of bytes between state resets (0 = the training context, the old behaviour). Bytes outside the training vocabulary are not scored and are charged 8 bits each in bpb_all. A long file is sharded by whole windows (-eval_offset and -eval_chars as multiples of the window); the CPU twin runs at a few hundred bytes per second per process.
The same recurrence runs on the GPU, N tokens at a time. dp_shakespeare -eval_stream restores a model, allocates -eval_batch windows in parallel and feeds each window as consecutive N-token forwards with the mixer state carried between them ( GatedLinearAttentionGpu::setCarry; BornAttentionGpu::setCarry on the density state and the normalizer, restored Born layers being switched to the linear form with chunk -born_chunk) and RoPE at the absolute byte position; per-position losses are read back, and the state is reset at window boundaries only.
./dp_shakespeare -tw_cublas true -fft_fast true -born_chunk 512 -eval_stream pg19_test.txt \
-model pg19.mod -eval_window 5242880 -eval_batch 8
Pass -tw_cublas true (without it the token-wise layers run through the slow reference kernel), and -eval_batch must be at least 2. On a GH200 a 41 MB test set in eight 5 MB windows takes about three minutes for a 6-block model with the gated mixer and six with linear Born. The GPU pass reproduces the CPU twin exactly for the gated mixer and to within 0.1% of the summed loss for linear Born (fp32 chunkwise accumulation order).
PG-19 byte models. scripts/pg19_5m/ fetches PG-19 (fetch_pg19.sh 1200: the test and validation partitions and the first 1,200 training books from the public bucket), trains two RoPE-only twins (train_5m.sh 8 15000: E=256, 6 blocks, 4 heads, context 4096, batch 8, 15,000 steps on one GH200, about two hours each; linear Born with the fixed decay, then the gated mixer) and scores them (score_5m_gpu.sh on the GPU, eval5m.sh on the CPU). Scored on the official test partition in 5,242,880-byte windows, the protocol of the MBLM hierarchy (Egli et al., 2025):
| model (4.03M complex parameters, 0.49 GB of training bytes) | best val (nats/char) | test, 5M-byte windows (bits/byte) | validation, 5M windows / 4k resets |
|---|---|---|---|
linear Born + fixed decay ( -born true -born_linear true -born_decay true -pos_table false ) |
1.121 | 1.656 | 1.690 / 1.696 |
gated complex mixer ( -gla true -pos_table false ) |
1.148 | 1.700 | 1.732 / 1.736 |
| MBLM (Mamba-2 + two Transformer stages, 350M parameters, 100 GB of bytes, 5M-byte context) | — | 2.448 | — |
The carried state neither drifts over five million bytes nor adds much beyond the training context (the 5M-window and 4k-reset scores are within 0.006 of each other), and without the positional table the decayed Born twin leads the gated one at every checkpoint. Samples from both twins are in the papers' results.
The per-step work is the forward/backward plus a single coalesced NCCL all-reduce of the gradients; the validation loss is identical across GPU counts, confirming correctness. Measured on RTX A6000 (CUDA 12.8, NCCL 2.26), same global batch:
| GPUs | ms / step | speed-up |
|---|---|---|
| 1 | 157 | 1.00x |
| 2 | 81 | 1.94x |
| 3 | 56 | 2.79x |
The current version of the CNet framework has some limitations:
- multi-GPU training is data-parallel only (no model/tensor parallelism yet); per-layer kernel-launch overhead on the single-GPU path is eliminated by optional CUDA Graph capture (see above);
- adding a layer means hand-writing both its CPU reference and its CUDA kernel (wired through
GpuMapping/GPU_LAYER_TABLE); there is no higher-level CPU→GPU codegen yet. - supported only on Linux/MacOS.