Variance Reduction for Expectations with Diffusion Teachers

Researchers introduced CARV, a compute-aware variance-accounting framework that improves Monte Carlo estimation in diffusion-model pipelines, achieving 2-3x effective compute multipliers in text-to-3D distillation and attribution experiments without changing the objective. The method, which combines amortized reuse, timestep importance sampling, and stratified-inverse-CDF construction, also cut gradient variance by an order of magnitude in single-step distillation but did not improve downstream FID, indicating variance was not the bottleneck there.

Pretrained diffusion models serve as frozen teachers feeding downstream pipelines such as text-to-3D, single-step distillation, and data attribution. The teacher gradients these pipelines consume are Monte Carlo MC expectations over noise levels and Gaussian noise samples; their estimator variance dominates compute cost because each draw requires expensive upstream work rendering, simulation, encoding . We introduce CARV, a compute-aware variance-accounting framework that motivates a hierarchical MC estimator: amortize the expensive upstream computation over cheap diffusion-noise resamples, sharpened by timestep importance sampling and a stratified-inverse-CDF construction. In our text-to-3D distillation and attribution experiments, CARV delivers 2-3x effective compute multipliers most from amortized reuse; ~25% additional from IS+stratification without changing the objective; in single-step distillation, the same techniques cut gradient variance by an order of magnitude but do not improve downstream FID, marking the regime where MC variance is no longer the bottleneck.