Notes on NNTC vs. neural texturing

A developer behind NNTC, an open-source texture compression project on GitHub, reports that a bilinear/degree-2 polynomial decoder fitted to each PBR material can match or beat MLP-based neural texturing on raw PSNR after CUDA encoder optimizations and BC4/BC5 awareness. The developer argues MLP training is more expensive and harder to solve, and that NNTC's use of high-resolution weight planes with GPU texture filtering hardware makes it far faster, encoding in seconds. NNTC currently supports up to 4 channels per latent plane, with possible expansion to 6-8 channels.

NNTC here on GitHub https://github.com/richgel999/nntc/ uses a bilinear/degree-2 polynomial decoder fitted to each PBR material i.e. a degree-2 polynomial whose only quadratic terms are the products between the two latents . Other solutions use full non-linear neural networks MLP's . In my testing, MLP's are usually but not always stronger, but not by much low PSNR difference, like ~1-3 dB . MLP's also make it harder to get filtered samples - NNTC easily leverages existing GPU texture filtering hardware. However, eventually as I optimized the NNTC-specific CUDA encoder, and made it BC4/BC5 aware, it started to beat my neural network based solution on raw PSNR. Training MLP's is a lot more expensive, and a harder problem to solve - and IMHO unnecessary for PBR textures if you configure the latent textures correctly . Encoding high frequency details into low-res feature channels and training MLP's to decode them is going to be quite expensive. The alternative is to do what GPU texture formats like BC1-7/PVRTC1/ASTC/etc. have been using for ages: use high resolution weight planes, and optionally for more efficient distribution supercompress the data in some way using RDO+LZ, DCT etc. One of NNTC's current limits: it only supports up to 4 channels on each of the two latent planes. I may expand this to up to ~6-8 channels in the future. If I hadn't implemented a neural net prototype first, I couldn't have found a way to NNTC. NNTC also encodes in seconds with CUDA. Even with backprop, I'm skeptical a neural solution can be competitive there.