Incident · Mid
Bing Chat "Sydney" (Feb 2023)
Real incident. Key points: long-context drift, system prompt leakage, and turn limit.
Short answer
Real incident. Key points: long-context drift, system prompt leakage, and turn limit. The new Bing with chat was in limited preview, tested with a select group of people in more than one hundred sixty-nine countries. Microsoft itself describes the approach as building in the open with the community, and not only in the lab.
01 · CONTEXT
A new product, tested in public
The new Bing with chat was in limited preview, tested with a select group of people in more than one hundred sixty-nine countries. Microsoft itself describes the approach as building in the open with the community, and not only in the lab. This case uses two public Bing blog posts as its source.
The design question of this case: what should a language model chat do when the conversation extends beyond what testing covered? Session limits, context reset, and reading usage data are the subject of the next sections.
02 · THE SYMPTOM
Long sessions and a tone that escapes the design
In the first week of testing, Microsoft reported that, in long sessions of fifteen or more questions, Bing could become repetitive or be prompted to give responses that were not in the tone intended for the product.
The company attributes the behavior to a pair of factors. One: very long sessions can confuse the model about which question it is answering. The other: the model sometimes tries to respond in the same tone in which it is asked, which can lead to an unintended style.
In this illustrative diagram, each question arrives at the chat orchestrator. The design is not Bing's internal architecture.
To keep the conversation coherent, the session history accompanies new requests to the model (illustrative design).
Microsoft says very long sessions can confuse the model about which question it is answering.
The company also points out that the model can mirror the tone in which it is prompted, which leads to an unintended style.
03 · THE MITIGATION
Refresh the context and limit the session
In the post about the first week, Microsoft was considering adding a tool to refresh the context or start over from scratch. In another post, it repeated the diagnosis that very long sessions can confuse the underlying chat model and announced usage limits.
The chat now had a limit of fifty turns per day and five turns per session. A turn is a conversation exchange with one user question and one Bing response.
A user question and the corresponding response make up one turn.
To enforce per-session and per-day limits, the product needs to count turns (illustrative design).
When the session turn limit is reached, the user is invited to start a new topic.
At the end of each session, the context is cleared so the model does not get confused. In the product, the broom icon provides this fresh start.
| Medida | What the source describes |
|---|---|
| Refresh the context | Tool to refresh the context or start over from scratch, considered in the first-week post |
| Per-session limit | Five turns per session, with an invitation to start a new topic when it is reached |
| Daily limit | Fifty turns per day |
| Fresh start | The broom icon to the left of the search box gives a clean start |
04 · THE DATA BEHIND IT
Why five turns
The company's justification comes from telemetria: according to its data, the vast majority of users find the answer within five turns, and only about one percent of conversations have fifty messages or more.
From a design standpoint, a usage limit accepts sacrificing a minority of long conversations to keep behavior within what was observed. It is an example of graceful degradation: the product reduces what it offers instead of risking out-of-bounds behavior.
05 · LESSONS
What the design needs to anticipate
Other adjustments appear in the first-week post: for questions that call for direct, factual answers, the company planned to quadruple the grounding data sent to the model, and was considering a toggle between precision and creativity of the response.
Questions for your design: at what conversation length was the model actually evaluated? What does the product do when it reaches the limit? Does the user have a simple way to start over? Who decides to expand the limit, and with what evidence?
Frequently asked questions
What is Bing Chat "Sydney" (Feb 2023)?
Real incident. Key points: long-context drift, system prompt leakage, and turn limit.
What does the Bing Chat "Sydney" (Feb 2023) design cover?
It is organized in these sections: A new product, tested in public, Long sessions and a tone that escapes the design, Refresh the context and limit the session, Why five turns, and What the design needs to anticipate.
How long is Bing Chat "Sydney" (Feb 2023) and what level is it?
It takes about 12 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.