Incident · Mid
Gemini (Feb 2024): image generation paused
Real incident. Key points: side effect of prompt rewriting, over-refusal, and kill switch.
Short answer
Real incident. Key points: side effect of prompt rewriting, over-refusal, and kill switch. The Gemini conversational app gained an image generation feature, with the ability to create images of people, built on an image model. During development, the team tuned the feature to avoid known pitfalls of the technology, such as violent or sexually explicit images and depictions of real…
01 · CONTEXT
A new feature in an AI assistant
The Gemini conversational app gained an image generation feature, with the ability to create images of people, built on an image model. During development, the team tuned the feature to avoid known pitfalls of the technology, such as violent or sexually explicit images and depictions of real people.
The product serves users around the world. For a generic request, such as soccer players or someone walking a dog, the user may want a range of people; for a request for a specific type of person, or in a particular cultural or historical context, the response should reflect exactly what was asked.
02 · WHAT WENT WRONG
Two failures, one in each direction
The company's post points to two causes. The first: the tuning to show a range of people did not account for cases that clearly should not show a range. The second: over time, the model became more cautious than intended and refused certain requests entirely, wrongly treating quite harmless prompts as sensitive.
The user sends the request to the app.
In the diagram, a caution layer evaluates whether the request can be fulfilled.
According to the post, the model became more cautious than intended and refused some requests entirely, treating harmless prompts as sensitive.
In the diagram, another layer adjusts the request to return a range of people, something that may make sense for generic requests.
The adjustment did not account for cases that clearly should not show a range.
Both failures, one in each direction, end in embarrassing and wrong images.
Together, the two failures led the model to overcompensate in some cases and to be too conservative in others, which, in the company's words, resulted in embarrassing and wrong images.
03 · EXPECTED BEHAVIOR
The right rule depends on the request
| Order type | Expected behavior |
|---|---|
| Generic request | Receiving a range of people may be what the user wants. |
| Specific type of person | The response should reflect exactly what was asked. |
| Particular cultural or historical context | The response should reflect exactly what was asked. |
| Harmless request | It should not be treated as sensitive; the refusal was a wrong interpretation by the model. |
The design point is that the right rule depends on the type of request, so a single rule applied to all requests deserves suspicion.
04 · THE RESPONSE
Pause the problematic slice
The company acknowledged that some images were inaccurate or even offensive and paused image generation of people in Gemini while it worked on an improved version. Coverage at the time describes the response as a temporary shutdown of that capability.
Quickly turning off a specific feature through a kill switch (kill switch) is an availability capability worth planning for in the design.
Users report inaccurate or even offensive images.
In the diagram, a switch sits between the app and image generation of people.
The switch is turned off and image generation of people is paused.
Before turning it back on, the company says it will improve the feature significantly, with extensive testing.
The company states that it will improve the feature significantly before turning it back on, with extensive testing, and coverage reports that it plans to reactivate the capability afterward.
05 · DESIGN LESSONS
What to take to your project
When designing an image generation system, it is worth asking where each safety or diversity fix is applied and which requests it should never reach.
It is worth planning a kill switch per feature slice and a test suite that includes requests the fix should not alter and harmless requests that should not be refused.
Frequently asked questions
What is Gemini (Feb 2024): image generation paused?
Real incident. Key points: side effect of prompt rewriting, over-refusal, and kill switch.
What does the Gemini (Feb 2024): image generation paused design cover?
It is organized in these sections: A new feature in an AI assistant, Two failures, one in each direction, The right rule depends on the request, Pause the problematic slice, and What to take to your project.
How long is Gemini (Feb 2024): image generation paused and what level is it?
It takes about 12 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.