{"slug": "one-layer-is-enough-adapting-pretrained-visual-encoders-for-image-generation", "title": "One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation", "summary": "Researchers at Apple and Johns Hopkins University introduced FAE (Feature Auto-Encoder), a framework that adapts pretrained visual encoders like DINO and SigLIP for image generation using as little as a single attention layer. FAE achieves near-state-of-the-art FID scores of 1.29 on ImageNet 256×256 with classifier-free guidance, demonstrating high-quality generation with minimal architectural changes.", "body_md": "[content type paper](/research/)published July 2026\n\nOne Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation\n\nAuthorsYuan Gao, Chen Chen, Tianrong Chen, Jiatao Gu\n\nOne Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation\n\nAuthorsYuan Gao, Chen Chen, Tianrong Chen, Jiatao Gu\n\nVisual generative models (e.g., diffusion models) typically operate in compressed latent spaces to balance training efficiency and sample quality. In parallel, there has been growing interest in leveraging high-quality pre-trained visual representations—either by aligning them inside VAEs or directly within the generative model. However, adapting such representations remains challenging due to fundamental mismatches between understanding-oriented features and generation-friendly latent spaces. Representation encoders benefit from high-dimensional latents that capture diverse hypotheses for masked regions, whereas generative models favor low-dimensional latents that must faithfully preserve injected noise. This discrepancy has led prior work to rely on complex objectives and architectures. In this work, we propose FAE (Feature Auto-Encoder), a simple-yet-effective framework that adapts pre-trained visual representations into low-dimensional latents suitable for generation using as little as a single attention layer, while retaining sufficient information for both reconstruction and understanding. The key is to couple two separate deep decoders: one trained to reconstruct the original feature space, and a second that takes the reconstructed features as input for image generation. FAE is generic—it can be instantiated with a variety of self-supervised encoders (e.g., DINO, SigLIP) and plugged into two distinct generative families–diffusion models and normalizing flows. Across class-conditional and text-to-image benchmarks, FAE achieves strong performance. For example, on ImageNet 256×256, our diffusion model with CFG attains an near–state-of-the-art FID of 1.29 (800 epochs) and 1.70 (80 epochs). Without CFG, FAE reaches the state-of-the-art FID of 1.48 (800 epochs) and 2.08 (80 epochs), demonstrating both high quality and fast learning.\n\nAdapting Self-Supervised Representations as a Latent Space for Efficient Generation\n\nNovember 4, 2025[research area Computer Vision](/research/?domain=Computer%20Vision), [research area Methods and Algorithms](/research/?domain=Methods%20and%20Algorithms)[conference ICLR](/research/?event=ICLR)\n\nWe introduce Representation Tokenizer (RepTok), a generative modeling framework that represents an image using a single continuous latent token obtained from self-supervised vision transformers. Building on a pre-trained SSL encoder, we fine-tune only the semantic token embedding and pair it with a generative decoder trained jointly using a standard flow matching objective. This adaptation enriches the token with low-level,…\n\nKaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling\n\nDecember 2, 2024[research area Computer Vision](/research/?domain=Computer%20Vision), [research area Methods and Algorithms](/research/?domain=Methods%20and%20Algorithms)[conference NeurIPS](/research/?event=NeurIPS)\n\nDiffusion models have emerged as a powerful tool for generating high-quality images from textual descriptions. Despite their successes, these models often exhibit limited diversity in the sampled images, particularly when sampling with a high classifier-free guidance weight. To address this issue, we present Kaleido, a novel approach that enhances the diversity of samples by incorporating autoregressive latent priors. Kaleido integrates an…", "url": "https://wpnews.pro/news/one-layer-is-enough-adapting-pretrained-visual-encoders-for-image-generation", "canonical_source": "https://machinelearning.apple.com/research/adapting-pretrained-visual-encoders", "published_at": "2026-07-15 00:00:00+00:00", "updated_at": "2026-07-15 17:46:42.627090+00:00", "lang": "en", "topics": ["machine-learning", "computer-vision", "generative-ai", "artificial-intelligence", "ai-research"], "entities": ["Apple", "Johns Hopkins University", "Yuan Gao", "Chen Chen", "Tianrong Chen", "Jiatao Gu", "DINO", "SigLIP"], "alternates": {"html": "https://wpnews.pro/news/one-layer-is-enough-adapting-pretrained-visual-encoders-for-image-generation", "markdown": "https://wpnews.pro/news/one-layer-is-enough-adapting-pretrained-visual-encoders-for-image-generation.md", "text": "https://wpnews.pro/news/one-layer-is-enough-adapting-pretrained-visual-encoders-for-image-generation.txt", "jsonld": "https://wpnews.pro/news/one-layer-is-enough-adapting-pretrained-visual-encoders-for-image-generation.jsonld"}}