Realistic face generation

01 · CONTEXT

The problem: generating faces that look real

Chapter 7 of the book Generative AI System Design Interview is titled Realistic Face Generation.

A GAN trains two models at the same time: a generator, which captures the data distribution, and a discriminator, which estimates whether a sample came from the training data or from the generator.

The generator's training aims to maximize the probability that the discriminator makes a mistake.

02 · TRAINING

The adversarial training loop

The diagram below shows the training loop and the periodic evaluation of the generator.

1. The diagram starts with the face dataset and the batch loader, which deliver real images to the discriminator.

2. The generator turns noise into images, and the discriminator estimates whether a sample came from the training data or from the generator.

3. The generator is trained to maximize the chance that the discriminator makes a mistake.

4. The two sides need to move in sync: the generator must not be trained too much without the discriminator being updated.

5. One training option is to give the discriminator and the generator individual learning rates, as in TTUR.

6. The evaluator generates images from a checkpoint, passes them through Inception-v3, and computes the mean and covariance for the FID.

03 · STABILITY

When the generator loses diversity

The original GAN paper lists as a disadvantage the need to synchronize D and G well: G must not be trained too much without updating D, to avoid the scenario in which G maps too many values of z to the same value of x.

The TTUR authors propose one individual learning rate for the discriminator and another for the generator.

04 · CONTROL

StyleGAN: more control over the generated face

StyleGAN starts from a learned constant input and adjusts the image style at each convolution layer based on the latent code.

According to the paper, the result is an automatic, unsupervised separation between high-level attributes, such as pose and identity in faces, and stochastic variation, such as freckles and hair.

Before that, the latent code passes through an intermediate latent space; the mapping network that produces it has 8 layers.

The same work presents FFHQ, a dataset of human faces with much higher quality and much wider variation than existing high-resolution datasets.

In the illustrative examples, the authors avoided sampling extreme regions of W with the truncation trick, but all the FIDs in the paper are computed without it.

05 · EVALUATION AND SERVING

Measuring quality and serving the images

FID was proposed to evaluate GANs and, according to the authors, is more consistent than the Inception Score. The latter does not compare the statistics of real samples with those of synthetic ones.

For the computation, the authors generate 50,000 images, pass them through Inception-v3, and compute the mean and covariance.

The diagram below shows the path of a synthetic face request in production.

1. The diagram starts with the client, who requests a synthetic face through the API gateway.

2. Generator sampling uses only a forward pass.

3. When showing examples, StyleGAN avoided sampling extreme regions of W with the truncation trick.

4. The diagram continues with the image written to storage and delivered to the client through the CDN.

5. The diagram ends with the service metrics going to monitoring.