Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction

Researchers have developed Pixel Cube, a diffusion-based method for relighting dynamic portrait videos that achieves photorealism and temporal consistency. The system uses a hybrid dataset of real-captured and rendered videos, combined with an LED-based lighting system and per-frame HDR environment maps, to train a generative model that preserves subject identity, expression, and fine facial features under new lighting conditions. The method demonstrates state-of-the-art performance in photorealism, lighting harmony, and temporal consistency, with applications in portrait photography and generalization to unseen data.

arXiv:2606.02919v1 Announce Type: new Abstract: We present a diffusion-based method for relighting dynamic portrait videos with photorealism and temporal consistency. Our method is fueled by a hybrid training dataset that consists of real-captured and rendered dynamic portrait videos with diverse subject appearances, facial motions, head poses, and known lighting conditions. Specifically, we construct an LED-based lighting system for realistic lighting emulation and high-speed video relighting data acquisition. By leveraging the image priors embedded in pre-trained video diffusion models, and using per-frame high dynamic range HDR environment map as lighting control, we train a high-performance generative model for realistic and identity-preserving dynamic portrait video relighting. In addition to the environment map control, our model uses a synthesized background image to enable control on the camera's exposure level and color tone. Our model can produce temporally consistent relit portrait video that looks realistic and harmonious under a provided new environment and faithfully preserve the subject's expression and fine facial features, including skin tone, wrinkles, and facial hair. Our model generalizes well to unseen data, in terms of the subject appearance, motion, and lighting condition. We perform extensive experiments on relighting in-the-wild videos with various environment maps and demonstrate practical applications on portrait photography. Results show that our method achieves state-of-the-art performance in photorealism, lighting harmony, and temporal consistency.