Reward-Guided Generation in Diffusion Models: A Tutorial

This survey paper focuses on inference-time techniques for controlled generation with diffusion models, contrasting them with post-training methods. It introduces the concept of optimizing downstream reward functions during the sampling process, often referred to as alignment, which can involve conditioning on target properties or maximizing regressor outputs. The document outlines various derivative-free and derivative-based guidance methods, including SMC-based guidance and classifier guidance, along with their theoretical foundations using Doob's transform. It also discusses search-based algorithms and how these techniques can be applied to editing and refining generated designs, particularly in the context of protein design, while also highlighting similarities and differences with inference-time techniques in language models. Finally, the tutorial explores combining fine-tuning with inference-time techniques through policy distillation to improve computational efficiency.