Text-to-image

01 · CONTEXT AND REQUIREMENTS

What the system needs to do

A text-to-image system receives a natural-language prompt and returns an image. Stable Diffusion, a public example, is described as a latent text-to-image diffusion model capable of generating photo-realistic images from any text input. In an interview, the topic comes down to ML system design.

The functional requirements are to accept a prompt, return an image, and allow retries. Among the non-functional ones, latency per image, GPU cost per image, quality as perceived by the user, and safety of the generated content weigh most on the design.

02 · LATENT DIFFUSION

Why diffuse in latent space

A diffusion model generates the image over many sequential steps. According to the Stable Diffusion authors, training and evaluating such a model directly in pixel space requires repeated evaluations in the high-dimensional space of RGB images; in addition, evaluating an already trained model is expensive in time and memory, because the same architecture runs in sequence for many steps.

Learning goes through stages with different roles. The first is perceptual compression, which removes high-frequency detail but learns little semantic variation; in the second stage, the generative model learns the semantic and conceptual composition of the data.

The proposal is to first train an autoencoder that provides a lower-dimensional space, perceptually equivalent to the data space, and to train the diffusion in that latent space. The autoencoding stage needs to be trained only once and can be reused across several diffusion trainings.

1. The diagram starts with the text prompt, which goes to the text encoder.

2. In the model documented in the model card, the unpooled output of the text encoder enters the denoising network via cross-attention.

3. The network operates in a lower-dimensional latent space, perceptually equivalent to the data space, provided by an autoencoder, and the diffusion is trained in that latent space.

4. The inference cost comes from running the same architecture in sequence for many steps, which makes evaluation expensive in time and memory.

5. The final latent is decoded into image space with a single pass through the decoder.

The compression rate is also a design decision: weak compression leaves perceptual compression to the diffusion model, and compression that is too strong loses information and limits the achievable quality. The authors conclude that latent diffusion improves training and sampling efficiency without degrading quality.

03 · TEXT AND GUIDANCE

From prompt to image: conditioning and CFG

Text conditions the diffusion through cross-attention: the LDM authors designed a general conditioning mechanism based on it, and in the model documented in the model card the unpooled output of the text encoder feeds the UNet through this path.

Classifier-free guidance (CFG) controls the balance between quality and diversity without a separate classifier. Classifier guidance complicates the training pipeline because it requires training an extra classifier; in CFG, a conditional and an unconditional model are trained jointly and their estimates are combined. In the LDM text-to-image experiment, applying this guidance greatly improves sample quality.

1. The diagram shows two branches that reach the same combination point: one with the prompt and one without.

2. A single neural network parameterizes both models; in the unconditional branch a null token is fed in.

3. Each sampling step evaluates the denoising model twice, once conditional and once unconditional.

4. The two estimates are combined to obtain a balance between sample quality and diversity.

04 · STEPS AND LATENCY

Steps, quality, and latency

The CFG authors note that the number of sampling steps has a large impact on sample quality and report that quality improves as the number of steps increases.

The LDM paper compares sampling speed for different numbers of steps with the DDIM sampler against FID. Relative to pixel-space diffusion, latent diffusion with moderate compression reaches lower FID and higher throughput.

The number of steps is a natural lever between quality and response time. When discussing custo, ask how much GPU each image consumes and how that changes with the number of steps and with the use of CFG. Offer generation profiles, such as a fast draft and a final version, and make clear what is lost in each.

05 · EVALUATION

How to measure whether the image is good

The LDM paper uses FID to compare model variants. CLIPScore, on the other hand, was proposed to evaluate image captions without the need for references, focusing on image-text compatibility, and complements reference-based metrics.

06 · SAFETY AND SERVICE

Safety and service design

The LDM authors list societal risks: it becomes easier to create and spread manipulated data or to spread disinformation and spam; generative models can also reveal their training data; and deep learning models tend to reproduce or worsen biases present in the data.

The Stable Diffusion repository does not recommend using the provided weights in services or products without additional safety mechanisms. Its reference sampling script incorporates a safety checker module to reduce the likelihood of explicit outputs.

In the service design, a fila separates request intake from execution on the GPU: the client receives an identifier, the workers consume the queue, and the request state is kept in a dedicated store.

1. The diagram starts with the client, which sends the prompt to the generation API.

2. Before entering the queue, the request goes through a prompt filter.

3. The queue hands requests to the GPU workers, which run the latent diffusion pipeline shown earlier.

4. In the Diffusers documentation, the Safety Checker compares, in embedding space, the class probability of fixed harmful concepts against the image after it is generated.

5. The approved image goes to object storage and reaches the client through the CDN.

6. The workers and the image filter send signals to the metrics and evaluation component.