Netflix: artwork personalization (contextual bandits)

01 · CONTEXT

Recommending the title is not enough: you also have to convince

For many years, the goal of Netflix's recommendation system was to put the right titles in front of each member at the right time. The blog about artwork personalization starts from a next question: why should the member care about a recommended title? One of the ways to answer is the image used to portray the title.

Before, Netflix already used non-contextual bandits to find the winning image for each title, regardless of context. The step described here is choosing the best image for each member, motivated by the enormous diversity of tastes and preferences.

02 · CHALLENGES

Why choosing an image is different from recommending

A title's play can only come from the image that was chosen to show the member. So the image choice is a closed-loop chicken-and-egg problem.

Each title's set of images also matters: it must be diverse enough to cover audiences interested in different aspects, and each image must be engaging, informative and representative to avoid clickbait.

03 · CONTEXTUAL BANDITS

From batch learning to online learning

In the traditional mode, a batch of usage data is collected and a new algorithm is run over it. The text notes that this batch path generates regret: many members, for a long period, did not benefit from the better experience.

For artwork personalization, the framework is contextual bandits, a class of online learning algorithms that balances the cost of collecting training data with the benefit of applying the learned model to each member's context. Here the member is the context, because different members are expected to respond differently to images.

The QCon talk page adds that bandits bring technical challenges because they require closing a feedback loop. The diagram shows this loop illustratively.

1. The member only sees the image the selector chose for the title, and any play comes from that image.

2. The selector starts from the title's candidate images, up to a few dozen, and the model estimates the quality-play probability of each in the member's context; the one with the highest probability is chosen.

3. Each randomized selection is logged along with information about the randomness used, which makes it possible to correct biased selection propensities.

4. Each impression and its outcome, played or not played, feed the training set by member, title and image tuple.

5. When the data comes back, the models are updated and put into use again, in a continuous improvement loop.

04 · EXPLORATION

Exploring without hurting the experience

The training data of a contextual bandit comes from injecting controlled randomness into the model's predictions. The schemes range from epsilon-greedy, with uniform randomness, to closed-loop schemes that vary the degree of randomness according to the model's uncertainty.

Each randomized selection must be logged. This log makes it possible to correct biased selection propensities and to evaluate models offline without bias.

Exploring has a cost, because the image chosen in a session may not be the one predicted to be best. With a very large member base, the text states that this regret is usually very small and gets diluted across them.

Two precautions improve the data. Exploration can be controlled so that the image does not change too often, which gives cleaner engagement attribution. And the label is defined by engagement quality, to avoid a model that recommends clickbait images.

05 · MODEL

Predicting quality play by member and image

The training set records, for each (member, title, image) tuple, whether that selection resulted in a play of the title. There are typically up to a few dozen candidate images per title, and the problem is simplified by ranking a member's images independently across titles.

The model predicts, for each tuple, the probability that the member has a quality engagement. They can be supervised models or contextual-bandit equivalents with Thompson Sampling, LinUCB or Bayesian methods, which balance the best prediction with data exploration.

Among the member attributes that can serve as context are the titles the member played, genre, interactions with the title, country, language, device, time of day and day of the week, plus signals of what the recommendation algorithms think of the title, regardless of the image.

Some images are naturally better than others. An image's take rate is the number of quality plays divided by the number of impressions, and personalization still recovers selections that agree, on average, with the ordering of the non-personalized model.

At selection time, the candidate images are ordered by these probabilities and the one with the highest probability is presented to the member.

06 · EVALUATION

Offline replay and online A/B test

Before deploying the algorithms to real members, offline evaluation uses a technique called replay. It answers counterfactual questions from the logged exploration data and makes it possible to compare, without bias, what would have happened in historical sessions with different algorithms.

1. In exploration, a random image is assigned to each member, and the system logs the impression and whether the profile played the title.

2. Replay confronts the candidate model with the log, and only profiles where the randomly drawn image and the one chosen by the model coincide count.

3. Over this subset, the play fraction is computed, which is the number of quality plays divided by the number of impressions.

4. Models with a substantial increase in replay move on to an A/B test against non-personalized bandits.

5. In the online test, a reasonable correlation was seen with what replay had measured.

In the text, the contextual bandit algorithms raise the catalog's average play fraction relative to random selection and to non-contextual bandits.

After testing many models offline and finding those with a substantial increase in replay, the team ran an A/B test of the most promising contextual bandits against non-personalized bandits. Personalization worked and generated a significant increase in the core metrics, with a reasonable correlation between replay and the online result.

The increase was larger when the member had no prior interaction with the title, which makes sense because the image tends to weigh more for those less familiar with it.

07 · LESSONS

What to take into the interview

The blog describes the project as the first time Netflix personalizes not only what it recommends, but also how it recommends. Among the next steps listed are handling cold-start for new images and titles and extending the approach to other types of artwork and other evidence, such as synopses, metadata and trailers.

The central idea is to treat the choice as a closed loop: what you show determines what you observe. Logging the impressions, the outcomes and the randomness used becomes part of the system's design, not an instrumentation detail.