Mirror of neural texture compression trained with Evolution Strategies

A developer released a self-contained C++ experiment that encodes an RGB image as a low-resolution latent texture plus a tiny MLP decoder, training both entirely with Evolution Strategies rather than backpropagation. The project, published as a prior-art disclosure on GitHub, uses a per-texel credit assignment scheme that measures each latent value's loss change only over the pixels in its bilinear footprint, making ES practical on a 131k-value latent with just four perturbation pairs per step. On a 512×512 crop of kodim23, a 128×128×8 latent reached 32.2 dB PSNR at 3.27 bpp entropy-coded, training in about 150 seconds on a 32-thread CPU.

This is a Prior Art disclosure mirrored from the GitHub repo here, originally created September 3, 2026: https://github.com/richgel999/neural texture es https://github.com/richgel999/neural texture es A small, self-contained C++ experiment: an RGB image is encoded as a low-resolution latent texture plus a tiny MLP decoder, and both are trained entirely with Evolution Strategies — no backprop, no derivatives, no training framework. Dependencies are stb image , stb image write , and OpenMP. Write-up: Fitting a neural texture decoder with ES https://richg42.blogspot.com/2026/09/fitting-neural-texture-decoder-with-es.html I u,v ≈ MLP bilinear Z, u, v , phi u,v Z is the latent texture, phi a small positional encoding. At decode time each pixel bilinearly samples Z at its UV, appends phi , and runs the MLP. 512×512 crop of kodim23, 3000 iterations, latent quantized to 8 bits after training, MLP weights counted as fp16: | Latent | PSNR | bpp raw | bpp entropy coded | |---|---|---|---| | 64×64×4 | 26.9 dB | 0.56 | 0.47 | | 64×64×8 | 28.2 dB | 1.07 | 0.87 | | 128×128×4 | 30.3 dB | 2.06 | 1.65 | | 128×128×8 | 32.2 dB | 4.07 | 3.27 | The 128×128×8 run uses a 14 → 24 → 24 → 3 MLP 1035 weights, leaky ReLU, sigmoid output and trains in about 150 s on a 32-thread CPU. Quantizing the latent to 8 bits costs 0.04 dB. Output of that run out 128c8/ : target crop, reconstruction after 3000 iterations, and the eight latent channels side by side. | Target | Reconstruction 32.2 dB | |---|---| out 128c8/model.bin is the trained model; evaluate it with ntc kodim23.png --load out 128c8/model.bin --latent 128 128 8 --nfreq 1 --iters 0 . MLP: antithetic ES, 32 perturbation pairs per step, each pair evaluated on the same random 4096-pixel minibatch. The estimated gradient is fed to Adam. Latent: all latent values are perturbed at once and the full image is decoded twice per pair, 4 pairs per step. Each pixel's loss change is credited only to the 4 texels its bilinear tap reads. Ordinary ES already updates every parameter from one antithetic pair, but its variance grows with parameter count: the single scalar loss difference is the sum of thousands of separate local effects, and each texel's share is buried under everyone else's. Here each texel's loss difference is measured only over the pixels in its own bilinear footprint, so noise from the other ~16k texels is discarded instead of averaged. That credit assignment, not the per-evaluation cost, is what makes ES practical on a latent with 131k values using only 4 pairs per step. The latent stays fp32 during training; quantization is applied afterwards and reported at a configurable bit depth. CMake generating a Visual Studio solution MSVC , or any C++17 compiler with OpenMP on Linux/WSL. Tested with MSVC 2022 and MSVC 2026 on Windows, and gcc 13 under WSL2. Note that std::normal distribution differs between standard libraries, so the same seed gives slightly different results on each platform. cmake -S . -B build -G "Visual Studio 17 2022" -A x64 cmake --build build --config Release For Visual Studio 2026 use -G "Visual Studio 18 2026" , which needs CMake 4.2 or newer the CMake bundled with VS 2026 works . ntc image.png --out out --latent 128 128 8 --iters 3000 Run with no arguments, ntc trains on the checked-in kodim23.png using the default 64×64×4 latent. The image is located relative to the executable, so this works from the build directory as well as the repo root. Progress is printed to stdout MSE, PSNR, quantized PSNR and bitrate, latent stats, throughput . Reconstructions, a latent visualization, and model.bin are written to the output directory periodically. Useful options ntc --help lists them all : | Flag | Meaning | |---|---| | --latent W H C | latent texture size default 64 64 4 | | --mlp W1,W2,... | hidden layer widths default 24,24 | | --act leaky\|relu\|tanh\|sine | hidden activation | | --pos SPEC | positional features: uv , fourier:N , dct:N , local , lfourier:N , lquad , ldct:N , none | | --qbits N | latent bit depth for the reported quantized PSNR / bitrate | | --load model.bin --iters 0 | evaluate a saved model | | --mlp-pairs , --mlp-sigma , --mlp-lr , --lat-pairs , --lat-sigma , --lat-lr | ES hyperparameters | Images larger than 512×512 are center-cropped by default --crop . The Kodak test images are widely available; kodim23 is the parrots. Published September 3, 2026 blog post above and this repository . The following are disclosed here as public prior art. Neural texture representations using learned latent grids with small neural decoders, and Evolution Strategies / simultaneous-perturbation methods for derivative-free optimization, are established ideas. The technically distinctive part explored here is their combination with the decoder's known spatial dependency structure: all latent values are perturbed simultaneously, antithetic full-image evaluations produce per-pixel loss differences, and each pixel's loss difference is attributed only to the latent texels actually read by that pixel's filtering footprint. This yields simultaneous, support-restricted ES estimates for every latent texel while discarding loss variation from pixels a given texel cannot affect. Estimates for neighboring texels still share pixels and the same perturbation draw, so they are correlated rather than independent. For a given latent value, the omitted per-pixel loss terms do not depend on that value's perturbation, so their products with it have zero expectation in the ordinary Gaussian ES estimator. Footprint attribution therefore removes them without bias, as a variance-reduction mechanism that follows directly from the decoder's dependency graph. Implemented in this repository: ldct:N , found to improve quality at fine latent resolution. Described, not yet implemented: This is a deliberately simple research toy for learning and experimentation, not a codec. Nothing is tuned. Obvious next steps: quantization-aware training, block-compressed BC/ASTC latents inside the training loop, multiresolution latents, alternative losses.