{"slug": "taming-optimization-variance-in-compact-neural-shading-networks", "title": "Taming optimization variance in compact neural shading networks", "summary": "Researchers present a training algorithm that reduces optimization variance in small neural networks used for real-time neural shading, improving convergence reliability. The method uses an ensemble of network instances, pruning underperformers and dynamically resizing batches to maintain or improve wall-clock training times. Tested on learning reflectance functions, the approach yields better-performing models with higher likelihood than single-instance training.", "body_md": "We present a training algorithm to mitigate optimization instabilities in small neural networks, like those used in real-time neural shading applications. While large, overparameterized models exhibit predictable convergence, smaller architectures often suffer from high optimization variance: differently initialized models converge to disparate local minima. To reduce these training instabilities, we introduce an optimization approach that utilizes an ensemble of network instances during training. We prune underperforming instances and dynamically resize training batches to maintain wall-clock timings comparable to—or faster than—single-instance training. This strategy allows efficiently exploring the weight space yielding a well-performing model with significantly higher likelihood than optimizing a single instance only. We develop and analyze the algorithm in the context of learning reflectance functions for neural shading. This is a challenging task for small neural models due to the high dynamic range of the target function. In addition to our multi-instance training method, we also revisit the choices of loss functions, activation functions, and input parameterization to further improve quality and training robustness.", "url": "https://wpnews.pro/news/taming-optimization-variance-in-compact-neural-shading-networks", "canonical_source": "https://research.nvidia.com/labs/rtr/publication/bitterli2026taming/", "published_at": "2026-08-03 20:05:11+00:00", "updated_at": "2026-08-03 20:22:56.847704+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "computer-vision"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/taming-optimization-variance-in-compact-neural-shading-networks", "markdown": "https://wpnews.pro/news/taming-optimization-variance-in-compact-neural-shading-networks.md", "text": "https://wpnews.pro/news/taming-optimization-variance-in-compact-neural-shading-networks.txt", "jsonld": "https://wpnews.pro/news/taming-optimization-variance-in-compact-neural-shading-networks.jsonld"}}