AI Overviews (May 2024): "glue on pizza"

01 · CONTEXT

A generated summary that depends on search

On May 30, 2024, Google published a post explaining why some AI Overviews results became a topic on social media. The feature had been announced weeks earlier for everyone in the US. The post is the source for this case, so what follows is what the company says, without reconstructing behind-the-scenes details.

According to Google, the feature does not generate output from the model's training alone. A custom language model is integrated with the core web ranking systems and was built for tasks such as busca, like identifying relevant results in the index. It should only show information backed by the top results, with links for the person to dig deeper.

1. The query goes through the core web ranking systems, with which the model is integrated.

2. Ranking performs the traditional search task: identifying relevant, high-quality results in the index.

3. The model summarizes what the top results support, instead of answering only from what it learned in training.

4. The answer comes out with links for the person to explore the sources.

02 · WHAT WENT WRONG

Weak sources, unlikely questions

Google attributes the odd answers to reasons that are not pure model invention: misinterpreting the query, misinterpreting a nuance of web language, or there being little good information available. The company says it tested before launch, with red-teaming, evaluations on samples of typical queries, and tests on a portion of traffic, but that nothing compares to millions of people running novel searches.

The clearest example is the question about how many rocks to eat. Before the screenshots went viral, practically nobody asked that question, and there was little serious content about it: a data void. There was, however, satirical content on the topic, and the overview faithfully pointed to one of the few sites that addressed the question.

1. A question almost nobody asked falls into a data void: there is little quality content on the topic.

2. The available content on the topic was satirical.

3. In other cases, the overview surfaced sarcastic content from forums, which led to unhelpful advice, such as using glue to make cheese stick to pizza.

4. The overview reflects what the available sources say: in the rocks example, it faithfully pointed to one of the few sites that addressed the question.

Another pattern was sarcastic content from discussion forums, which can be great sources of first-hand accounts but sometimes lead to unhelpful advice, such as using glue to make cheese stick to pizza. Google also mentions a few cases where the overview misinterpreted the language on pages and presented incorrect information.

03 · WHAT CHANGED

Fewer one-off cases, more trigger rules

The company says it did not fix queries one by one: it updated the systems to help broad sets of queries, including new ones, and cites more than a dozen technical improvements. The post lists a sample of these changes, shown below in an explanatory diagram.

1. Better mechanisms were built to detect nonsensical queries, which should not show an overview.

2. The inclusion of satire and humor content was limited.

3. The use of user-generated content was limited in answers that could give misleading advice.

4. Trigger restrictions were added for queries where the overview was not as helpful.

5. For hard news and health: the goal is not to show an overview on hard news, and for health there were additional trigger refinements.

04 · LESSONS

When not answering is also a design decision

Google reports finding a content policy violation in fewer than one in every 7 million unique queries in which the overview appeared. In that post, violation means an overview with potentially harmful, obscene, or otherwise violating information.