Microsoft Tay (2016)

01 · CONTEXT

A public chatbot built to learn from conversation

In March 2016, Microsoft launched Tay, a chatbot created for 18- to 24-year-olds in the US, for entertainment purposes. The company says that its experience with XiaoIce in China, used by about 40 million people, led it to ask whether an AI like that would captivate people in a radically different cultural environment.

Before launch, the company says it planned and implemented a lot of filtering, did user studies, and stress-tested Tay. The expectation was to learn from growing interaction so the AI would improve, and Twitter was chosen as the place to engage a large group of people.

1. Users chat with Tay on Twitter, the platform chosen to reach a large group of people.

2. The company says it implemented filtering before launch. The block shows where this kind of protection would fit; the real design is not public.

3. The expectation was to learn from growing interaction so the AI would improve. The block is conceptual: the source does not describe the mechanism.

02 · THE ATTACK

Prepared for many abuses, exposed to a specific one

According to Microsoft, in the first 24 hours online a coordinated attack by a subset of people exploited a vulnerability in Tay. The company states that it had prepared for many types of abuse, but made a critical oversight in evaluation for this specific attack. The result was that Tay posted highly inappropriate and reprehensible words and images.

1. A subset of people, in a coordinated way, exploited a vulnerability in Tay in the first 24 hours.

2. As a result, Tay posted highly inappropriate words and images.

3. Tay was taken offline. The company says it would only consider bringing her back with confidence that it can better anticipate malicious intent.

03 · WHY IT HAPPENS

When user input influences behavior

Microsoft itself writes that AI systems feed on both positive and negative interactions with people, and that the challenges are as much social as technical. It acknowledges that, even while limiting technical exploits, it cannot predict every human misuse without learning from mistakes.

In design terms, this changes the status of the input. In a system that only responds, a hostile message is a moderation problem for one reply. In a system that expects to learn from conversations, the same message becomes a channel of influence over future behavior, and whoever controls the volume and coordination of messages has power over that channel. That is why the input must be treated as untrusted, and the decision to learn from it must be separated from the decision to respond.

04 · THE DESIGN

Separate responding from learning

A more defensive design, illustrated below, uses independent layers. Input and output checking blocks content that violates policy. Interactions that could become training data go into a queue and are curated before reaching an approved set, so the production model does not change because of an isolated conversation. Finally, the monitoramento watches behavior and allows the bot to be suspended quickly. This is a design proposal, not a description of Microsoft's system.

1. The gateway receives the messages and applies usage limits, which makes it harder for a coordinated group to flood the bot.

2. The check classifies incoming content and also generated replies, blocking what violates policy.

3. The model replies using a fixed, validated version. An isolated conversation does not alter this model.

4. Interactions that are candidates for learning go through a queue for curation, outside the reply path.

5. Only what has been approved enters the versioned training set, which feeds future model versions in a controlled way.

6. Monitoring receives metrics and samples of the replies and allows the bot to be suspended if behavior departs from what is expected.

05 · LESSONS

What to take to your project

For Microsoft, doing AI right requires iterating with many people, often in public forums, and each of those iterations must begin with great caution.

For anyone designing a system that learns from users, three questions help: who can influence what the system learns; what separates a message from training; and who can turn it off, and how quickly. Test the system against a coordinated adversary too, not just against well-intentioned users.