Google Photos: racist labeling

01 · CONTEXT

Labeling photos automatically

Google Photos is a personal photo service that uses machine learning to group photos with similar content and to enable search across users' collections. This case deals with a labeling incident in that service and the mitigation Google adopted.

The design question is: what do you do when an image classifier produces an offensive label and there is no immediate fix for the model? The case shows a practical answer and its cost.

1. In this illustrative design, the app sends the photo to storage.

2. The classifier consumes the new photo and writes the proposed labels to the index.

3. The user's search queries the label index and returns the matching photos.

02 · THE INCIDENT

The wrong label and the first response

The 2015 incident involved Photos labeling pictures of a user and a Black friend as “gorillas.”

An engineer who became the public face of the cleanup said the “gorilla” label would no longer be applied to groups of images, and Google said it was working on long-term fixes.

More than two years later, one of those fixes was erasing gorillas and some other primates from the service's vocabulary, according to Wired.

03 · THE MITIGATION

Blocking terms instead of fixing the model

Wired tested Photos with a collection of 40,000 images featuring many animals. Searches for “gorilla,” “chimp,” “chimpanzee,” and “monkey” returned “no results.”

A Google spokesperson confirmed that “gorilla” was censored from searches and image tags after the 2015 incident, and that “chimp,” “chimpanzee,” and “monkey” are also blocked.

1. In this design, the classifier proposes candidate labels and hands them to the filter.

2. The filter discards the labels that are on the blocklist and lets the rest through to the index.

3. In search, the typed term also goes through the filter; a blocked term finds no photos.

04 · WHY IT'S HARD

Training limits and scale

According to Wired, with enough data and computing power, software can be trained to categorize images with high accuracy, but it does not easily go beyond the experience of that training.

Engineers who put models into production need to worry about “corner cases” that are absent from the training data. A professor quoted in the report says it is very hard to model everything the system will see in production.

With a huge image base, a tiny chance of confusing one type of great ape with another can become near certainty.

1. In Photos, the search term goes through a filter of blocked terms.

2. The enterprise API is a different product; in the Wired test, it identified photos of gorillas and chimpanzees.

3. Two products from the same company gave different answers for the same type of image.

05 · LESSONS

What to take into the design

The spokesperson highlighted a Photos feature that lets users report errors and said the image labeling technology is still immature and far from perfect.

For the design of a labeling system, complementary layers help: evaluating the modelo by subgroup before launch, blocking terms as containment when the model errs severely, and keeping monitoramento with a reporting channel to find what testing did not find. One layer does not replace the others; the impact on people is also within the scope of compliance and accountability.