Incident · Junior
Moffatt v. Air Canada (2024): chatbot invents a policy
Real incident. Key points: hallucination with legal liability, grounding, source citation, and human handoff.
Short answer
Real incident. Key points: hallucination with legal liability, grounding, source citation, and human handoff. The Moffatt v. Air Canada case deals with liability for a chatbot. Jake Moffatt asked the airline for a refund, saying he had relied on incorrect guidance from its chatbot about bereavement fares.
01 · CONTEXT
A chatbot answers about a bereavement fare
The Moffatt v. Air Canada case deals with liability for a chatbot. Jake Moffatt asked the airline for a refund, saying he had relied on incorrect guidance from its chatbot about bereavement fares.
The chatbot suggested that the bereavement fare could be requested retroactively, and another page on the airline's own site contradicted this. The tribunal ruled that Air Canada was responsible for the misleading information given by the chatbot.
The customer asks the website chatbot about bereavement fares.
The chatbot suggests that the bereavement fare can be requested retroactively, and the tribunal concluded that the information was incorrect.
Another page on the airline's site contradicted what the chatbot had said.
02 · THE DECISION
The company answers for what the chatbot says
Air Canada argued that the chatbot was a separate entity, which would exempt it from liability for its inaccuracies. The tribunal rejected the argument: the chatbot is just a part of the website, and the company is responsible for all the information on it.
| Company's argument | Tribunal's position | |
|---|---|---|
| Nature of the chatbot | It would be a separate entity, exempting the company | It is just a part of the airline's website |
| Correct page on the site | The right information was on another page | The company did not explain why that page would be more trustworthy than the chatbot |
| Verification by the customer | The customer could check another part of the site | The company did not explain why the customer would have to verify the information in another part of the site |
The tribunal applied a standard of care that requires the company to take reasonable care to ensure its statements are accurate and not misleading, and concluded that Air Canada did not take that care with the chatbot's accuracy. It also considered it reasonable for the customer to rely on the information the chatbot gave.
03 · WHAT REMAINED OPEN
What the case does not clarify about the system
The decision says little about the chatbot itself: Air Canada did not provide information about its nature. The commentator who serves as the source laments the lack of information about why the tribunal found the company negligent, and notes that it could have tried to defend itself by explaining how the chatbot was trained and tested.
For that reason, this article does not describe the chatbot's real architecture: the source does not describe it. The design in the next section is an explanatory proposal, not an account of what the airline had.
04 · SAFER DESIGN
Ground the answer in policy and know when to stop
A chatbot that speaks on behalf of the company needs a single source for what it states. One approach is to answer only with snippets retrieved from the current policy, cite the source in the answer, and hand off to a person when no snippet supports what the customer asks. This design ties into the topic of regulatory compliance in system design, where legal liability becomes a design requirement.
The customer asks the chatbot; it does not answer anything yet.
The chatbot looks up the current policy snippet in the knowledge base and builds the answer from it alone, showing the source.
With no snippet to support the answer, or faced with a refund request, the chatbot hands the conversation off to a person through the queue.
Every conversation is logged for audit and sampled review.
| Risco | Design control | |
|---|---|---|
| Unsupported answer | The policy text does not support the answer | Answer only with a snippet retrieved from the current version and refuse when there is none |
| Opaque origin | The customer does not know where the answer came from | Show the source and the policy page link in the answer itself |
| High-impact request | Refund or exception to the rule | Hand off to human support instead of promising |
| Error found late | Nobody reviews the conversations | Log conversations and review samples and complaints |
05 · LESSONS
What to take into your design
The commentator concludes that the decision reinforces the importance of ensuring the accuracy of information across all customer-facing interfaces. The chatbot is one of those interfaces.
Frequently asked questions
What is Moffatt v. Air Canada (2024): chatbot invents a policy?
Real incident. Key points: hallucination with legal liability, grounding, source citation, and human handoff.
What does the Moffatt v. Air Canada (2024): chatbot invents a policy design cover?
It is organized in these sections: A chatbot answers about a bereavement fare, The company answers for what the chatbot says, What the case does not clarify about the system, Ground the answer in policy and know when to stop, and What to take into your design.
How long is Moffatt v. Air Canada (2024): chatbot invents a policy and what level is it?
It takes about 9 minutes to read and is rated Junior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.