Concepts
Diffusion in latent space
Intuition
Imagine an artist who needs to paint a huge canvas: instead of working on every little dot of paint from the start, they first make a smaller sketch that keeps only what matters in the scene, and only at the end scale that sketch up to the full painting. Generating images can become much cheaper this way, as long as the sketch does not discard what matters: spending effort on tiny details the eye barely notices is expensive and not very useful. So the heavy work of creating the image happens on the compressed sketch, and a final step reconstructs the full-size image.
How it works
Diffusion models generate an image by applying many denoising steps in sequence, and in pixel space the network and the gradients must be evaluated over all pixels at every step, which makes training and sampling expensive. The paper's solution is to separate compression from generation: a separately trained autoencoder, with a perceptual loss and a patch-based adversarial objective (to avoid the blurriness of using only L1 or L2), has an encoder E that produces the latent z = E(x), with height and width reduced by a downsampling factor f, and a decoder D that reconstructs the image. To keep latents from having arbitrarily high variance, a slight KL penalty (KL-reg.) or a vector quantization layer in the decoder (VQ-reg.) is applied. The diffusion model then operates on zt, the noisy version of z: a UNet conditioned on the timestep is trained to predict the noise, with a squared-error loss between the true noise and the predicted noise. Because the latent keeps the 2D spatial structure, the convolutional UNet exploits its image inductive biases and the compression can be mild; at generation time, a latent sample is decoded with a single pass through D. To condition generation on an input y (text, layout, class), a domain-specific encoder τθ projects y into an intermediate representation that supplies the keys and values of cross-attention layers in the UNet, while the queries come from the UNet's own intermediate representation; τθ and the UNet are optimized jointly. There is a trade-off in the factor f: with small f (LDM-1, which is pixel-space diffusion, and LDM-2) training is slow, because the perceptual compression is left over for the diffusion model, and with very large f fidelity stagnates because the autoencoder loses too much information.
In practice
The paper reports that downsampling factors from LDM-4 to LDM-16 balance efficiency and fidelity, and that LDM-4 and LDM-8 offered the best conditions for high-quality synthesis, with much lower FID and higher sample throughput than pixel-space diffusion in the DDIM tests (complex datasets such as ImageNet require smaller compression). Because the autoencoder is trained only once, it can be reused across multiple diffusion trainings, and changing the conditioning modality means swapping the encoder τθ; the paper trains a 1.45-billion-parameter text-to-image model on LAION-400M this way, as well as class- and layout-conditioned models. The authors themselves acknowledge the limits: sequential sampling is still slower than GANs, and the autoencoder's reconstruction capacity can become a bottleneck when fine pixel-level precision is required.
Use it in a design
Practice the cases that rely on this concept in the System Design school.