Real architecture · Senior
Booking.com: 150 models, 6 lessons
Real production architecture. Key points: offline metric does not predict business impact, and A/B as the source of truth.
Short answer
Real production architecture. Key points: offline metric does not predict business impact, and A/B as the source of truth. This case starts from a public summary of the Booking.com paper presented at KDD. The summary covers the lessons from integrating about 150 customer-facing machine learning applications.
01 · CONTEXT
A summary about many models, not one
This case starts from a public summary of the Booking.com paper presented at KDD. The summary covers the lessons from integrating about 150 customer-facing machine learning applications.
According to the summary, the six lessons in the title are not listed in the body of the paper, but can be inferred from the division into sections. The main conclusion cited is that an iterative, hypothesis-driven process integrated with other disciplines was fundamental to building 150 machine-learning-enabled products.
The summary also recalls why recommending is hard in this domain: booking the wrong lodging is much worse than watching a movie you do not like.
02 · MODELS IN PRODUCTION
Many models, many roles
With about 150 models in production, machine learning touched many parts of the Booking.com experience. Some models are specific to a use case in one context; others form a semantic layer, modeling concepts that are useful across several contexts. The summary groups the models into six broad categories.
Among them are the interface optimization models. In these, according to the summary, a single value is rarely optimal for everyone, so the models consider context and user information to decide the best interface.
In the first lesson, the summary states that, compared with other successful projects that did not use machine learning, projects with machine learning tend to deliver higher returns.
03 · METRIC AND BUSINESS
A better model is not a better business
Booking.com estimates the value delivered by a model through randomized controlled experiments, which measure the impact on business metrics. A finding highlighted in the summary is that improving a model's performance does not necessarily translate into a gain in business value.
| Possible reason | What it means |
|---|---|
| Value saturation | There is no more value to extract, whatever you do. |
| Segment saturation | Smaller populations are exposed to the treatment, and the old and new models largely agree. |
| Optimizing a proxy | The proxy, for example clicks, does not convert into the desired business metric, for example conversion. |
04 · PROBLEM AND LATENCY
Defining the problem well and serving fast
The third lesson is to define the problem carefully before building models. The summary highlights that some of the biggest improvements come from changing the formulation, not from improving the model within a given formulation. The example cited is replacing a user preference model based on click data with a natural language processing problem based on guest reviews. According to the summary, the best problem is often not the one that comes to mind first.
The fourth lesson is that prediction latency matters. In an experiment with synthetic latency, the summary reports that an increase of about 30% in latency cost about 0.5% in conversion rates.
The user's request reaches the application, which needs predictions to decide what to show.
Precomputation and caching are two of the tactics cited in the summary to reduce model latency; here the cache holds predictions that have already been computed.
Copies of the model distributed horizontally, behind a load balancer, are another tactic cited in the summary.
Favoring models with fewer parameters and using a custom linear prediction engine are also on the summary's list of tactics.
The effort is justified because, in the synthetic latency experiment, more latency cost conversion.
05 · EARLY FEEDBACK
Knowing early if the model is doing badly
The fifth lesson is to get early feedback on model quality. With the model serving requests, monitoring the quality of its output is crucial and brings at least two challenges. The first is incomplete feedback, because of the difficulty of observing the true labels. The second is delayed feedback: the prediction made at booking time about whether the user will leave a review can only be checked after the trip.
For binary classifiers, one tactic is to look at the distribution of the model's responses. According to the summary, smooth bimodal distributions, with a clear stable point, are a sign of a model that separates the two classes well, and other shapes may indicate a model that is struggling. Analyzing the distribution of responses proved very useful for detecting model defects very early.
The application requests the prediction and the model returns a score for each request.
Each score is recorded without waiting for the true label, which can take a long time to arrive.
The analyzer builds the distribution of the scores and looks for the smooth bimodal shape; other shapes suggest a model that is struggling.
The result goes to monitoring, which makes it possible to detect defects early.
06 · EXPERIMENTS
The proof is the controlled experiment
The sixth lesson is to test business impact with randomized controlled experiments. According to the summary, the vast majority of the success cases studied were made possible by sophisticated experiment designs, whether to guide development or to detect impact.
When not everyone can be exposed to the change, the treatment and non-treatment groups are created within the eligible subset.
When comparing models, the interest is in the cases where the two disagree, and the baseline is a control group that invokes the current model.
To evaluate the impact of performance, a third control group is added in which the model is not invoked.
What is compared are business metrics, which is what controlled experiments measure at Booking.com.
Frequently asked questions
What is Booking.com: 150 models, 6 lessons?
Real production architecture. Key points: offline metric does not predict business impact, and A/B as the source of truth.
What does the Booking.com: 150 models, 6 lessons design cover?
It is organized in these sections: A summary about many models, not one, Many models, many roles, A better model is not a better business, Defining the problem well and serving fast, Knowing early if the model is doing badly, and The proof is the controlled experiment.
How long is Booking.com: 150 models, 6 lessons and what level is it?
It takes about 17 minutes to read and is rated Senior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.