Blurring in Google Street View

01 · CONTEXT

Privacy at scale

Street imagery products gather huge, systematic collections of photos and, in the process, also capture license plates, faces and other information considered sensitive from a privacy standpoint. The system in this case automatically detects and blurs faces and license plates, to protect privacy in Google Street View.

The article points out what makes the problem hard. The scale is large, which demands fully automatic, optimized algorithms and a lot of computing power. There is little control over the capture conditions, and the appearance of objects varies widely. And the recall for faces and license plates would ideally be 100%.

But that is beyond the reach of state-of-the-art automatic methods: one of the best detectors in the world scored less than 78% on the authors' in-the-wild face set, and state-of-the-art face detectors alone are not enough for the desired recall. There is also a quality constraint: image quality must be preserved while keeping recall high, which forces control of the false positive rate.

02 · CAPTURE

Where the images come from and what they go through

The images are collected by a moving vehicle with its own camera system. The rig has nine 5-megapixel cameras in a rosette: eight pointed roughly parallel to the ground, in a radial arrangement, and one fisheye lens pointed up. The upward-facing camera is ignored, because any faces it captured would have too low a resolution to be identifiable.

The work uses the images from the individual cameras, before stitching into a panorama. The same car and camera configuration is assumed across different regions and times, and the post-processor receives the camera's index in the rosette in its feature vector.

The article describes the algorithms, not the infrastructure around them. The diagram that follows is a design proposal for this exercise, a design choice that does not describe Google's infrastructure. Its components are object storage for raw and redacted images, a queue, detection and blur workers, a panorama assembly service and a CDN.

1. In the reference design, the images from each individual camera arrive in object storage as raw images.

2. Each raw image becomes a task in the queue.

3. The queue distributes the tasks among many detection workers, which is how the design keeps up with the volume of images.

4. The detection workers decide which boxes will be blurred and hand them to the blur stage.

5. The blur stage writes the redacted images to object storage.

6. With the redacted images, the assembly service stitches them into a panorama and the CDN delivers the result to the public.

03 · DETECTION

Two detectors and a post-processor

The core idea combines a sliding-window detector tuned to a high-recall, low-precision operating point with a fast post-processing stage. That stage removes false positives using domain-specific information that the detector lacks.

The detector scans the image with windows of various sizes. At the end, the remaining windows are filtered and merged according to their score and the overlap between scales.

The system brings together a high-recall primary detector, a high-precision, low-sensitivity secondary detector and a fast post-processing stage, which generates a feature vector from the detectors' outputs and scores the box.

1. An individual camera image enters the detection stage.

2. The primary and secondary detectors run in parallel on the same image and their outputs are stored.

3. Each box from the primary detector is passed to the post-processor.

4. When the secondary returns a box, the post-processor's neural network is skipped and the corresponding primary box gets a perfect score.

5. If the network's output reaches the threshold, the box is blurred in the final image.

04 · POST-PROCESSOR

Recovering precision without losing recall

Post-processing uses a neural network with features derived from the box's pixels and from the detectors' outputs. The features were designed to capture information the detectors did not have. One of them is the ratio between the box's height and base, which tries to capture whether the real size of the object indicated by the box is reasonable.

Other signals enter the vector: the output of a convolutional network over the candidate box's content and, for faces, the box's average skin-color probability. The article says the post-processor is fast largely because it exploits the consistencies of the image set.

In training, an example is positive if, and only if, there is a hand-labeled box overlapping the detected box by at least 10%. The cost function is weighted by area, so a larger box contributes more; according to the authors, this improves the pixel FPR, at the cost of slightly worse per-box precision.

05 · LICENSE PLATES

Same structure, different context

The license plate system has the same general structure as the face system. In the plate system the feature set is reduced, there is no secondary detector and a simplified car detector provides context in place of the color model.

The car boxes used in training are generated automatically, by enlarging the region around the plate box. The two features the car detector provides to the post-processor are the overlap (intersection over union, or IoU) between the ideal car box and the best detected one, and the score of the best box.

06 · METRICS

Measuring what matters and what got through

The face results are presented on two hand-labeled evaluation sets. Three metrics are used, among them the hand-counted per-box recall, which for privacy is the most important number. Manual counting is necessary because the ground-truth boxes tend to be larger than the automatic system's, and a face counts as sufficiently blurred when the facial features are obscured.

Automatic recall considers a ground-truth box recovered if the detected boxes overlapping it cover at least 50% of it. The pixel FPR is the percentage of blurred pixels that fall outside any ground-truth box.

On the evaluation sets, more than 89% of faces are sufficiently blurred. For license plates, the hand-counted recall at the chosen operating point is 96.5% in the US and 93.6% in the EU. The authors say they adopted a conservative notion of identifiable, which lowers recall, and that automatic recall still substantially underestimates plate performance, because fewer pixels need to be detected to blur enough.

On speed, for a 5-megapixel image the primary detector takes an average of 6 to 8 s, the secondary 1 to 2 s and the post-processor about 100 ms, for a total of 7 to 10 seconds depending on the image. The Tsinghua detector takes about 30 s at the same minimum face size.

07 · PUBLISHING

Redacting irreversibly and accepting what slips through

With the final boxes, the camera images are redacted and stitched into the panorama. The redaction method must be irreversible, hide some of the many false positives and make it obvious to the public that redaction took place.

The authors chose a combination of noise and aggressive Gaussian blur, smoothly blended into the background from the edge of the box.

Protection does not end with the automatic system: users report unblurred faces and plates, which are then blurred in the live product, narrowing the gap between automatic performance and 100% recall.