High-resolution image synthesis

01 · CONTEXT

Why high resolution is a cost problem

Generating high-resolution images is the subject of the book Generative AI System Design Interview: chapter 8 has exactly this title. This case treats it as a problem of machine learning system design.

Two open research papers show where the cost appears. Transformers are expressive but computationally infeasible for long sequences, such as high-resolution images. Diffusion models that operate directly in pixel space, meanwhile, often consume hundreds of GPU days in optimization, and inference is expensive because it requires sequential evaluations.

02 · LATENT COMPRESSION

Three ways to generate without touching the pixels

VQ-VAE is the basis of discrete compression: the encoder emits discrete codes, not continuous ones, and the prior is learned instead of fixed. With an autoregressive prior over these representations, the model generates high-quality images.

The Taming Transformers paper combines CNNs, which learn a vocabulary of image constituents, with transformers, which model the composition of those constituents in high-resolution images.

Latent diffusion models apply diffusion in the latent space of pretrained autoencoders, which significantly reduces computational requirements compared to pixel-space diffusion. That is the argument of cost at scale for working in latent space.

03 · SERVING

From request to final image

The diagram below shows an asynchronous service that generates in latent space, decodes, and refines the image in a cascade.

1. In the diagram, the client calls the API gateway, which puts the request on the job queue.

2. The generation worker operates in the latent space of a pretrained autoencoder, instead of operating directly in pixel space.

3. In the diagram, the autoencoder's decoder receives the worker's output and passes it to the refiner.

4. Generative models can be chained with super-resolution models; this cascade is what the refiner represents in this diagram.

5. A diffusion-based refiner, such as SR3, starts from pure Gaussian noise and refines the output iteratively, with a U-Net trained to remove noise at several levels.

6. In the diagram, the final image goes to storage and is delivered to the client through the CDN.

04 · TRAINING

Training in two stages

The diagram below separates the autoencoder's training, which defines the latent space, from the generator's training, which operates in that space.

1. In the diagram, the image dataset feeds the autoencoder training.

2. In a discrete version, as in VQ-VAE, the encoder emits discrete codes, not continuous ones, and the prior is learned.

3. In the diagram, the latent representations are kept in storage and serve as input for the generator training.

4. In the Taming Transformers paper the generator is a transformer that models the composition of the image's constituents; in latent diffusion, the generator is a diffusion model applied in latent space.

5. Operating in latent space significantly reduces computational requirements, compared to pixel-space diffusion.

6. In the diagram, the checkpoints feed sample evaluation, whose measurements go on to the training metrics.

05 · DECISIONS

Pixels or latent, and how to condition

To control the result, cross-attention layers make diffusion a flexible generator for general conditioning inputs, such as text or bounding boxes.