On the Diffusibility of High-Dimensional Latents

Representation Autoencoders (RAEs) let diffusion models operate in the feature spaces of pretrained visual encoders, but many off-the-shelf encoders are not optimized for faithful reconstruction and discard fine-grained visual details, according to the paper "On the Diffusibility of High-Dimensional Latents." The work reports that finetuning these encoders for image reconstruction is the expected remedy for the lost detail.

Representation Autoencoders RAEs enable diffusion models to operate in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not optimized for faithful reconstruction, discarding fine-grained visual details. As expected, finetuning these encoders for image reco