DiffusionBench: Towards Holistic Evaluation of Generative Diffusion Transformers

Researchers released DiffusionBench, a unified codebase for holistic evaluation of generative diffusion transformers across tasks like ImageNet and text-to-image generation. The benchmark supports training and evaluation of tokenizers and diffusion models with a single interface, aiming to provide more comprehensive assessment beyond traditional ImageNet metrics.

.-----------. | \ / |/ | | | | | | | | | | | | | | / | |/ \| ' \ | ░▒▓█▓▒░▒▓ | | | | | | | | | | \ \ | | | | | | ▒▓█████▓▒ | | /| | | | | \ , | / |\ /| | | | | ▓███████▓ | | ↓ | | █████████ | | | | | ▓███████▓ | | \ / \ ' \ / | ' \ | ▒▓█████▓▒ | | | | / | | | | | | | | | | / \ | | | |\ | | | | '-----------' Because ImageNet evaluation alone is no longer enough 📣 Announcement post: Call for DiffusionBench: A Holistic Benchmark for Diffusion Transformers . Help us grow the benchmark with new evaluation axes, new metrics, and faithful reproductions of published methods. This repo contains the unified codebase for DiffusionBench. It supports training and evaluation across different generation tasks ImageNet, T2I, ... through a single interface. Please see the sections below for the detailed structure. Come join us /End2End-Diffusion/diffusion-bench/blob/main/assets/qualitative.webp Text-to-image samples at 256×256 from models trained for 200K iterations using DiffusionBench. install uv project manager if you don't already have it curl -LsSf https://astral.sh/uv/install.sh | sh install dependencies uv sync prepare data uv run python scripts/prepare.py --data {all,imagenet,t2i,eval} download pretrained models uv run hf download diffusion-bench/diffusion-bench --local-dir pretrained models --exclude .gitattributes Reproduction flow: Stage 1 → Stage 2 . Set these environment variables first used for the output directory and W&B logging : export EXPERIMENT NAME=<run-name export ENTITY=<wandb-entity export PROJECT=<wandb-project export WANDB KEY=<key Stage 1. Train the RAE tokenizer: uv run torchrun --standalone --nproc per node=8 \ src/train stage1.py \ --config STAGE1 CONFIG PATH \ --results-dir results/stage1 --precision bf16 --compile --wandb Stage 2. Train the diffusion model on VAE/RAE/Pixel space: uv run torchrun --standalone --nproc per node=8 \ src/train.py \ --config STAGE2 CONFIG PATH \ --results-dir results/stage2 --precision bf16 --compile --wandb Stage 2 training configs run online evaluation during training the eval: block . For standalone evaluation of a released checkpoint, use the sampling/ configs — each embeds stage 2.ckpt pointing into pretrained models/ and the eval-time guidance, so the weights load automatically: export EXPERIMENT NAME=<run-name stage 1 reconstruction rFID/PSNR/SSIM/LPIPS uv run torchrun --nproc per node=8 src/offline eval stage1.py --config STAGE1 CONFIG PATH stage 2 generation FID/IS, GenEval/DPGBench/... uv run torchrun --nproc per node=8 src/offline eval.py --config STAGE2 CONFIG PATH configs/ ├── stage1/ └── stage2/ ├── training/ │ ├── imagenet/ │ └── t2i/ └── sampling/ ├── imagenet/ └── t2i/ Stage 2 spans VAE 11 , RAE 6 , REG 4 , and Pixel 3 families, identical across ImageNet and T2I. Swap any config between tasks with a single path change. The sampling/ set mirrors training/ but adds the trained checkpoint and eval-time guidance, so it runs offline eval directly. For ImageNet, pick the CFG-off baseline STAGE2 CONFIG PATH .yaml or the per-model best-CFG variant STAGE2 CONFIG PATH -cfg<scale -t0.0-0.9.yaml . | Category | Methods | |---|---| Latent Space | Pixel Space RAE 30+ representation encoders : DINOv2 SigLIP2 WebSSL PE LangPE and more RAEv2 30+ representation encoders : DINOv2 SigLIP2 WebSSL PE LangPE etc VAE 10+ VAEs : FLUX.2 FLUX.1 SD3.5 VA-VAE E2E-VAE and more | Output Prediction | x-prediction v-prediction | Transport | Rectified-Flow MeanFlow Improved-MeanFlow Pixel-MeanFlow Drifting | Loss | Flow Matching REPA iREPA | Architecture | LightningDiT JiT DDT | Tasks | ImageNet : class-conditional generation T2I : text-to-image generation | Evaluation | ImageNet: FID IS T2I: GenEval DPGBench GenAIBench VQAScore | Training Backend | DDP FSDP TODO | | Status | Details | | |---|---|---| Coding Agents | Yes | Agent-compatible. See skills/ | AutoResearch We welcome contributions Please refer to docs/contributors.md /End2End-Diffusion/diffusion-bench/blob/main/docs/contributors.md and for further details. /End2End-Diffusion/diffusion-bench/blob/main/docs/contributing.md docs/contributing.md The codebase is built upon some amazing projects: We thank the authors for making their work publicly available.