The GitHub release is
here .
The existing neural texture work has made the entire space overly complicated.
The first step beyond GPU textures is to essentially keep the current GPU texture-style latent, but replace the fixed hardware generative decoder with a configurable MLP, and adopt PVRTC1-like block (which is at 1/4 resolution in PVRTC1) endpoint bilinear sampling:
Replace the classical GPU texture decoder (ASTC/BC1-7 etc.) with a neural net (MLP). Now you get as many output channels as needed. Then bilinear filter the "color" endpoints (like PVRTC1), and use a flexible (say 2-4) # of channels.
I've been testing the "colors" latent (which is typically 2-4 channels) at 1/4, 1/6, 1/8 and 1/16 sizes relative to the texture's resolution.
Per-texel "selectors"/"weights" can be unchanged: use 1 channel for single textures, 2-3 for materials. I've tested 1-5 bits per channel, with potentially a different number of bits on each channel.
The "encoder" is the training step which can use backprop or ES (Evolution Strategies) etc. It's amazingly flexible and looks remarkable. ES on the per-texel or lower res latents is
easily optimized , and at this tiny MLP size it's not a big deal.
Even on a single texture (not a material), the bitrate and quality is roughly competitive vs. transform methods on GPU texture latents. Except this method scales easily to materials and inference (decoding) cost is amortized across the material. The more channels the MLP outputs, the lower the effective bitrate.
Around 250-1200 MLP weights seems like a good range for this method. Tiny networks. I don't see why this can't be implemented directly in GPU hardware.
For inference on load (on the CPU) this approach seems entirely reasonable, using threading and SIMD to accelerate inference. No sidebands (like NTC uses) are needed for fast full-format BC7 encoding - the math is simple, and is already solved by
[bc7f](https://github.com/BinomialLLC/basis_universal/wiki/Transcoder-Internals-Analytical-Real-Time-Encoders-and-Transcoders#bc7f)
.
None of this is theoretical - I have this working now using CUDA for training (with a CPU SIMD fallback if needed). It's exciting.
Some hard numbers, comparing this neural texture design vs. BC7, are from
A single complex 1080x1080 photo, neural block texture vs. two BC7 encoders at matched 8.0 bpp bitrates (neural was 8.02, including quantized latents+all MLP weights as FP16):
Neural block texture: 43.15 dB
bc7f analytical: 43.78 dB
bc7e_scalar level 0: 41.03 dB
bc7e_scalar level 1: 44.85 dB
bc7e_scalar level 2: 44.67 dB
bc7e_scalar level 3: 45.18 dB
bc7e_scalar level 6 (slowest): 45.59 dB
Neural config: Per-texel "selectors" latent: 2 channels, 4+2 bits (6 bits total)
1/4 resolution "colors" latent: 4 channels, 8-bit channels (32 bits total)
MLP: 1,083 weights
Training: Quant aware, ES+CD, 8k iterations, 106 secs on a 5090
This is a sanity check: at equal very high bitrates, can this format and my trainer match a highly tuned widely deployed BC7 encoder? Yes it can, or very close.
Unlike any GPU texture format, this design is highly configurable/flexible. It easily scales down to low bitrates, or scales wider to handle multi-channel PBR textures with the same latents.