GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats

A transformer model called GaussianRenderFormer renders 3D Gaussian splats directly into images without a rasterizer, reaching 38.08 dB PSNR on 300 unseen Objaverse objects against full, unpruned splats, according to the arXiv paper 2611.05593v1. The model was trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, while rasterizing the same pruned input reaches 44.02 dB. The authors attribute the gap to a bottleneck in the view decoder, since supplying each Gaussian's projected footprint adds 1.4 dB, and they do not rule out under-training.

arXiv:2611.05593v1 Announce Type: new Abstract: Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, GaussianRenderFormer renders 300 unseen objects at 38.08 dB PSNR against the full, unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The gap is not one of generalization, and our diagnostics are consistent with a bottleneck in the view decoder: telling it the projected footprint of each Gaussian, which it could in principle infer, adds 1.4 dB. Under-training is not ruled out. Code, trained models and videos: https://svlwoof.github.io/gaussianrenderformer