{"slug": "the-model-is-a-variable-compiling-a-neural-network-into-your-c-binary", "title": "The Model Is a Variable: Compiling a Neural Network Into Your C++ Binary", "summary": "A developer has built UchenML, a C++20 framework that compiles trained neural networks directly into C++ binaries, enabling models to run in edge applications without inference servers or API calls. The framework, demonstrated in a browser-based dots game with a 418 KB WebAssembly binary, uses templated layers and constexpr models to integrate machine learning into standard software development workflows. The developer plans to publish the source code but has not committed to a timeline.", "body_md": "I strongly believe software frameworks cannot be designed in the abstract.\n\nUchenML came out of my experiments integrating machine learning into edge\n\napplications, and out of what I want to build next — distributed, heterogeneous\n\nintelligence.\n\nThe first demo, [dots](https://uchenml.tech/demos/dots), puts a trained neural network in your\n\nbrowser tab as your opponent. No inference server, no API call — a 418 KB\n\nWebAssembly binary and a weights file, served as static assets. (It still\n\ncarries introspection hooks for a future demo; stripped, it gets smaller.)\n\nBehind it: 2,760,322 parameters packed to fp16, and a 128-channel residual trunk\n\nrunning eleven 3×3 convolutions across the 24×24 board for every position it\n\nevaluates — up to 128 positions per move, all in the browser tab. That model was\n\ntrained with UchenML and deployed with UchenML. It is almost certainly bigger\n\nthan it needs to be — I spent the training budget on making it stronger, not on\n\nmaking it smaller.\n\nI am not currently able to publish the source code. There is a\n\n[publicly available snapshot](https://github.com/eugeneo/dots), about a year\n\nold. I plan to publish the current codebase, but I cannot commit to a timeline\n\nyet.\n\nI chose C++ for ease of integration and portability. With Python, the model\n\nlives in a separate world, and reaching it from the application that needs it\n\nmeans writing and maintaining glue. I want the model inside one software\n\ndevelopment workflow: same repository, same build, same review, same tests.\n\nThe longer-term bet is heterogeneous, distributed intelligence: not one large\n\nmodel behind an API, but many small specialized ones running where the data\n\nalready is — in the browser tab, inside the app, or in a cloud service. That\n\nonly works if a model is cheap to embed anywhere, which makes the deployment\n\ntarget the design constraint rather than an afterthought.\n\nUchenML is C++20, built with Bazel, and tested on Visual C++, GCC, and Clang. CI\n\nruns the suite on Linux, macOS, Windows, and under Emscripten for WebAssembly.\n\nThe compute backend is hand-written portable SIMD on Google's Highway, so a\n\nsingle kernel source covers AVX-512, NEON, and WASM SIMD with no\n\narchitecture-specific branch in the library. Small, fast, and easy to drop into\n\nan existing C++ project is the whole design brief.\n\nThe model is a variable — I usually make it `constexpr`\n\n— declared in a header.\n\nInference, training, and everything else are templated on that variable, so the\n\nsame information is never stated twice. You never describe a layer's input type\n\nwhen the previous layer already declared its output; the compiler works that\n\nout, along with parameter counts and scratch buffer sizes. There are extra\n\nfacilities for introspection and for advanced use cases, but the core is a\n\nsimple composition of layers.\n\nThe framework is split into clearly defined modules, and you only pay for what\n\nyou use — convolution, RNN, and attention are separate targets, and depending on\n\none does not drag in the others. Training mirrors that structure in a parallel\n\ntree, and deliberately stays out of the runtime one.\n\nModels compose with `|`\n\n:\n\n```\n#include \"uchen/layers.h\"\n#include \"uchen/linear.h\"\n\nconstexpr auto model = uchen::layers::FloatModel<1>\n                     | uchen::layers::Linear<2>\n                     | uchen::layers::Relu\n                     | uchen::layers::Linear<1>;\n\n// (1×2 weights + 2 biases) + (2×1 weights + 1 bias)\nstatic_assert(model.all_parameters_count() == 7);\n```\n\n`Linear<2>`\n\ndeclares only its output width; the input width is deduced from\n\nwhatever it is piped onto. That is why the same descriptor composes anywhere in\n\na stack without restating shapes — and why getting it wrong is a compile error.\n\nBecause layers are values, a block of architecture is a named constant. This is\n\nthe residual block the dots network is built from — two padded 3×3 convolutions\n\non one branch, identity on the other, summed and activated — with `C`\n\nchannels\n\nover an `S`\n\n×`S`\n\ngrid:\n\n```\ntemplate <size_t C, size_t S>\ninline constexpr auto kResBlock =\n    uchen::layers::Fork<2>\n  | uchen::layers::Parallel(uchen::layers::Conv2d<C, 3, 3, 1, 1>\n                              | uchen::layers::Relu\n                              | uchen::layers::Conv2d<C, 3, 3, 1, 1>,\n                            uchen::Layer<>())\n  | uchen::layers::Join\n  | uchen::layers::Reduce<uchen::PlusOp, 2>\n  | uchen::layers::Relu\n  | uchen::layers::Reshape<uchen::convolution::ConvolutionInput<C, S, S>>;\n```\n\n`kResBlock<64, 16>`\n\nand `kResBlock<96, 24>`\n\nare different types with different\n\nparameter counts, both resolved at compile time. A trunk is a `constexpr`\n\npipeline of these; multi-head outputs come from `Fork`\n\nand `Parallel`\n\nthe same\n\nway, with each head's loss supplied separately at training time.\n\nThe next post will dig into real-world model definitions and the more advanced\n\ncomposition they need.\n\nThe model variable is a functor, so evaluating it is a call, and a\n\n`ModelParameters`\n\nobject supplies the weights:\n\n``` js\nuchen::ModelParameters parameters(&model, weights);  // std::span<const float>\nauto output = model(input, parameters);\n```\n\nAn optional third argument preallocates the scratch space. Its size is another\n\ncompile-time property of the model, and passing it is what makes the forward\n\npass allocation-free:\n\n```\n// The context is usually too large for the stack, so it goes on the heap. It\n// can equally come from a memory-mapped region, an arena, or anywhere else.\nauto context = std::make_unique<uchen::ModelContext<Model>>(parameters);\nauto output = model(input, parameters, *context);\n```\n\nOne context can be reused across calls. Reusing it is not thread-safe, but the\n\nmodel is a pure function of its input and parameters, so a context per thread\n\nruns in parallel — and the parameters, read-only throughout, are shared between\n\nthem.\n\nThe parameter count is a property of the type too, so the weight buffer is a\n\nfixed-size array sized by the compiler — no allocation, no length field to get\n\nwrong:\n\n```\nstatic constexpr size_t kParamCount = uchen::ModelParameters<Model>::P;\nstd::array<float, kParamCount> weights{};\n```\n\nThe on-disk format is that same flat array of floats, so weights can be memory\n\nmapped rather than parsed. In the browser build the array is filled straight\n\nfrom the `ArrayBuffer`\n\nthe page fetched: the demo ships its weights fp16-packed\n\n— 5.5 MB on the wire, widened to floats on load, since WASM offers no fp16 SIMD\n\n— and binds them with one `span`\n\n. There is no model loader, no operator\n\nregistry, no interpreter warm-up. Once the bytes have arrived, the network is\n\nready.\n\nTraining uses the same model definition, with `uchen/training/`\n\nadded. Fitting a\n\npiecewise-linear function, end to end:\n\n```\n#include \"uchen/training/training.h\"\n\nuchen::training::TrainingData<uchen::Vector<float, 1>,\n                              uchen::Vector<float, 1>> data(samples.begin(),\n                                                            samples.end());\n\nuchen::training::Training training(&model,\n                                   {&model, kInitialWeights},\n                                   uchen::training::SgdOptimizer<Model>{});\n\nwhile (training.Loss(data) > 1e-4f) {\n  training = training.Generation(data, 0.001f).next;\n}\n\nuchen::ModelParameters parameters = training.parameters();\n```\n\n`TrainingData`\n\nis a collection of input-output pairs. It also handles splitting\n\na dataset into batches and reserving validation records.\n\n`Generation`\n\ndoes not mutate anything — it returns the next training state,\n\nwhich makes a step cheap to inspect, discard, or checkpoint, and makes the loop\n\nread as a fold over generations. Optimizers are pluggable: `SgdOptimizer`\n\n(stochastic gradient descent), momentum SGD, and `AdamOptimizer`\n\ncome out of the\n\nbox, alongside Kaiming-He initialization, per-layer gradient statistics, and\n\nmulti-threaded batch gradients.\n\nThe loss function is deduced from the model's output type, and the framework\n\nprovides a few common ones. Pass your own to the `Training`\n\nconstructor instead\n\nand its gradient is computed automatically through the chain.\n\nWhat comes out is a flat float array — and that is the entire deployment\n\nformat, the same bytes the browser build maps into its `std::array`\n\n. Training\n\nand serving cannot disagree, because there is nothing for them to disagree\n\nabout.\n\n`Training`\n\nis multi-threaded, but it does not by itself scale to real-world\n\nruns. The next post will discuss the primitives it is built from, and how I used\n\nthem to run a larger, more reliable training pipeline with Bazel as the executor.\n\nAdding a layer means writing a type the composition machinery recognizes. The\n\nruntime contract is small:\n\n```\nclass MyLayer {\n public:\n  using input_t  = uchen::Vector<float, 8>;\n  using output_t = uchen::Vector<float, 4>;\n  static constexpr size_t parameter_count = 36;\n  using scratch_area_t = std::array<float, 4>;\n\n  output_t operator()(const input_t& input,\n                      const uchen::Parameters& parameters,\n                      memory::LayerContext<scratch_area_t>* context) const;\n};\n```\n\n`scratch_area_t`\n\ncomes from the model context, and the framework controls its\n\nlifetime. The layer uses it for its output and for any intermediate buffers it\n\nneeds.\n\n`operator()`\n\nis reached through a dispatcher that asks the compiler which of\n\nthose signatures your layer actually defines, so a layer takes only what it\n\nneeds — the input alone, the input and its parameters, or those plus a scratch\n\ncontext. Composite layers own no storage; they slice the model's single flat\n\nparameter blob at compile-time offsets, which is why a `Fork`\n\nof four heads\n\nstill costs exactly one allocation-free array.\n\nA pipeable descriptor turns the type into something that composes:\n\n```\ninline constexpr uchen::Layer<MyLayerDesc> MyLayer;\n```\n\nThe descriptor's job is to map \"the model so far\" onto a concrete layer type,\n\nbinding the input shape the caller never had to write down. It hooks in through\n\nan ADL-found `StackLayer(model, desc)`\n\n. It is also where optimizations hook in —\n\nkernel fusion, or eliding a layer that is a no-op at the given input shape.\n\nGradients are a separate, optional surface. A layer becomes trainable when an\n\noverload of `ComputeGradients`\n\nfor it is visible:\n\n``` js\nauto ComputeGradients(const MyLayer& layer, const MyLayer::input_t& input,\n                      const Grad& output_gradient,\n                      const uchen::Parameters& parameters,\n                      std::span<float, MyLayer::parameter_count> gradients,\n                      const void* scratch);\n```\n\nEvery layer in the framework — convolution, RMSNorm, pooling, RNN, softmax —\n\narrived through exactly this path. There is no privileged built-in set.\n\nUchenML is still very much a work in progress, but it has reached the point\n\nwhere I can hand agentic tools a model and let them build and train it — which\n\nfrees me to spend more time on the application side. The feature list is long.\n\nAmong the things I want to add:\n\nI plan to post every few weeks. Next up is a deep dive into the dots project and\n\nhow UchenML held up in the real world, followed by articles on the optimization\n\ntechniques inside the framework, kernel fusion among them. More demos are in the\n\nworks too.", "url": "https://wpnews.pro/news/the-model-is-a-variable-compiling-a-neural-network-into-your-c-binary", "canonical_source": "https://dev.to/eugeneo_17/the-model-is-a-variable-compiling-a-neural-network-into-your-c-binary-oom", "published_at": "2026-08-12 03:26:46+00:00", "updated_at": "2026-08-12 03:46:01.164234+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["UchenML", "dots", "Google Highway", "Bazel", "Emscripten", "WebAssembly"], "alternates": {"html": "https://wpnews.pro/news/the-model-is-a-variable-compiling-a-neural-network-into-your-c-binary", "markdown": "https://wpnews.pro/news/the-model-is-a-variable-compiling-a-neural-network-into-your-c-binary.md", "text": "https://wpnews.pro/news/the-model-is-a-variable-compiling-a-neural-network-into-your-c-binary.txt", "jsonld": "https://wpnews.pro/news/the-model-is-a-variable-compiling-a-neural-network-into-your-c-binary.jsonld"}}