Image captioning

01 · CONTEXT

What it means to caption an image

Captioning an image means generating a sentence that describes what it shows. It is a problem of ML system design that combines vision and language. According to the Show and Tell paper, a good description needs to capture the objects in the image, the relationships between them, their attributes, and the activities they are involved in, and also express this in natural language; that is why a language model is needed in addition to visual understanding.

02 · BASE ARCHITECTURE

Visual encoder and text decoder

The approach in the Show and Tell paper, called NIC, is a single model: it takes the image and is trained to maximize the likelihood of the sequence of words that describes it. The vision and language blocks can be pretrained on larger corpora.

1. The diagram starts with the input image, which goes to the visual encoder.

2. The encoder is a CNN pretrained on image classification, and its last hidden layer feeds the decoder.

3. The image is fed in only once, to inform the decoder about the content; in the authors' tests, feeding it at every step gave worse results.

4. The decoder generates the sentence one word at a time, until it emits the stop word, which signals that the sentence is complete.

5. The diagram ends at the application that consumes the finished caption.

At inference, the paper compares sampling with beam search: the latter keeps the k best partial sentences as candidates at each step, and using a beam of size one (greedy search) worsened the results on average.

03 · VISUAL ATTENTION

Looking at the right region for each word

The Show, Attend and Tell paper points out a limit of using only the convnet's top-layer representation: it may lose information useful for richer captions. The way out is to extract features from a lower convolutional layer, which lets the decoder focus on specific parts of the image.

1. The diagram starts with the image, which goes to the spatial feature extractor.

2. The extractor uses a lower convolutional layer, not a fully connected layer, so that the decoder can focus on parts of the image.

3. The attention module gives a weight to each region, read as the probability that it is the right place to focus for the next word, and summarizes the result in a context vector, a dynamic representation of the relevant part of the image.

4. The decoder generates one word per step, conditioned on the context vector, the previous hidden state, and the words already generated.

5. The edge going back from the decoder to the attention module represents the attention that changes as each word is generated.

04 · EVALUATION

How to measure whether the caption is good

Evaluation by human raters, who give each description a usefulness score, is described in the Show and Tell paper as the most reliable but time-consuming. Among automatic metrics, BLEU is a form of word n-gram precision between the generated sentence and the reference ones.

The sources disagree about BLEU: the NIC authors report that it correlates well with human evaluations, while the CIDEr authors state that BLEU and ROUGE correlate weakly with human judgment. Hence the CIDEr proposal, which measures a sentence's similarity to the consensus of how most people describe the image.

1. The diagram starts with the evaluation images, which feed the captioning model.

2. The evaluator receives the generated caption and the human reference captions, because CIDEr measures a sentence's similarity to the consensus of how most people describe the image.

3. Five reference sentences per image are not enough, according to the CIDEr authors, to measure how most people would describe the image; that is why they collected evaluation sets with many more descriptions per image.

4. CIDEr weights each n-gram by TF-IDF: the IDF discounts popular words, which are visually uninformative.

5. CIDEr-D is available on the MS COCO evaluation server and is more robust to gaming than the original version.

05 · PRODUCTION

Taking the model to a service

This section is a study proposal, not a company's architecture. Three decisions deserve attention when taking the model to production: whether the caption comes back immediately or later, where the images and captions are stored, and how quality is tracked over time. See also message queues e monitoramento.

1. The diagram starts with the client, which reaches the captioning service through the API gateway.

2. In the diagram, the captioning service connects to the image storage and the caption cache.

3. In the diagram, the service also connects to the inference queue, from which the inference worker consumes.

4. In the diagram, the worker connects to the storage, the cache, and the quality monitoring.