Airbnb search ranking (GBDT → DL, Experiences)

01 · CONTEXT

A two-sided marketplace and a short list

Airbnb is a two-sided marketplace: hosts offer their spaces, called listings, to be booked by guests from all over the world. The task of search ranking is to respond to the guest with an ordered list of a handful of listings, chosen from the thousands available in the inventory.

The model in focus is part of an ecosystem of models that contribute to deciding which listings to show. Within this ecosystem, it is the one that orders the available listings by the probability that the guest will book.

The first version of ranking was a hand-written scoring function. Replacing it with a gradient boosted decision tree (GBDT) model brought one of the biggest booking jumps in the company's history. Over time, however, gains in online bookings saturated, with long periods of neutral experiments, and this made the moment ripe to try broad changes to the system.

Training starts from the logs: the new model learns a scoring function that places, as high as possible, the impressions of the listings that ended up booked. It is then tested online in an A/B testing framework, to see whether it achieves a statistically significant increase in conversions relative to the current model.

02 · MODEL EVOLUTION

From a simple network to a DNN

The transition to deep learning was the culmination of a series of iterative refinements. The team started with customized, intricate architectures and found itself overwhelmed by the complexity; the authors then adapt Karpathy's advice to something like not being a hero at the start.

The first model to go live was a simple network with a single hidden layer, which was neutral on bookings against the GBDT. It used the same features as the GBDT, and the value of the exercise was to validate that the neural network pipeline was ready for production and able to serve real traffic.

The first breakthrough came from combining the network with the Lambdarank idea, to directly optimize NDCG. The formulation became pairwise preference: with the listings seen by the person who booked, pairs of booked and non-booked listings are built. Each pair loss is weighted by the NDCG difference that would result from swapping the positions of the two listings, which prioritizes moving the booked listing to the top of the list, rather than the bottom.

In another variant, the final FM prediction entered as a feature of the network and, from the GBDT, the index of the activated leaf in each tree entered as a categorical feature. The alternative models had performance comparable to the network in testing, but the listings they prioritized were very different.

03 · FEATURES AND SIGNALS

Features the network can take advantage of

The features that feed the DNN are, for the most part, simple properties of the listings, such as price, amenities, and booking history, passed directly with minimal engineering. The exceptions are outputs of other models: the price of listings that use the Smart Pricing feature, coming from a specialized model, and the listing's similarity to what the user has already seen, computed with co-view embeddings.

1. Simple listing properties enter the DNN almost without transformation.

2. The price of listings with Smart Pricing is supplied by a specialized model.

3. The similarity to what the user has already seen comes from co-view embeddings.

4. The DNN uses these signals to order the listings by booking probability.

Feeding the network the GBDT features, untreated, went very wrong: the loss saturated in the middle of training and additional steps had no effect. For decision trees, the exact numeric values matter little as long as the relative order makes sense; neural networks, on the contrary, are quite sensitive to them.

The smoothness of the feature distribution proved to be an invaluable tool for finding bugs, because errors tend to stand out from the typical distribution, and one example was currency bugs in the prices recorded for certain geographies.

For location, instead of raw latitude and longitude, the offset relative to the center of the map shown to the user is used, which lets the model learn global distance-based properties, rather than properties of specific geographies.

As for neighborhood preference, in the GBDT it came from a heavily engineered pipeline. In the network, it was enough to create a categorical feature that combines the query's city with the listing's S2 cell, mapped to an integer by a hash function. The embedding of this feature is learned by backpropagation and encodes the location preference by neighborhood.

04 · WHAT DID NOT WORK

Attempts that ran into the marketplace

Failed attempts outnumber successful ones. Two of them illustrate how a technique that is effective and popular outside can work worse when brought home.

The first was to use each listing's ID as the index of an embedding, learning one representation per listing, a technique that succeeded in other applications. In the variations tested, however, the listing ID mainly led to overfitting.

The cause lies in the marketplace. Listings are subject to physical-world constraints: even the most popular one can only be booked a limited number of times per year. This limitation generates very sparse data per listing, and overfitting is a direct consequence of it.

The second attempt was multitask learning: the model predicted the probability of booking and of a long view at the same time, and the ID embedding was shared across the tasks. Online, long views rose substantially, but bookings were neutral. Manual inspections pointed to possible reasons, such as expensive high-end listings or ones with long, hard-to-read descriptions.

Dropout did not help either: all the dropout variations tested slightly degraded the offline metrics.

05 · SYSTEM AND TRAINING

Training and serving the DNN

A guest search arrives at a server that does retrieval and scoring, and that also produces logs. A pipeline processes these logs to create the training data, training is done in TensorFlow, and the models are then sent to the server.

1. The guest search arrives at the server, which does retrieval and scoring.

2. To meet the strict latency requirement of search, the network scoring uses a custom Java library.

3. The server also produces logs.

4. A pipeline processes the logs to create the training data.

5. Model training is done in TensorFlow with this data.

6. The trained models are sent to the retrieval and scoring server.

When they started taking TensorFlow models to production, the team found no efficient solution to score them in a Java stack. Converting data back and forth between languages introduced latency that was a blocker, so they built their own neural network scoring library in Java.

The data pipeline also hid a bottleneck. Most of the training time went to parsing CSV and copying data; rebuilding the pipeline with Protobufs and the Dataset API sped up training, raised GPU utilization, and made it possible to scale training data from weeks to months.

Another bottleneck came from nearly static features. Many were listing properties that rarely changed, and reading them for every training example created an input bottleneck.

1. Reading the listing's nearly static properties for every training example created an input bottleneck.

2. To eliminate this disk traffic, the example now carries only the listing ID as a categorical feature.

3. The nearly static features were packed into a non-trainable embedding indexed by the listing ID.

4. Resident in GPU memory, the embedding avoids loading several kilobytes of data per example from disk through the CPU.

5. For features that changed during the training period, a small amount of noise was traded for training speed.

06 · INTERPRETABILITY AND LESSONS

Understanding the network and what changes in the roadmap

Feature importance estimation and interpretability took a step back with the move to networks. The strength of networks is finding nonlinear interactions between features, which also makes it harder to study a feature in isolation.

For each test query, TopBot ranks the listings with the model and compares the distribution of feature values between the listings at the top and those at the bottom. In one example, the distribution of review ratings was very similar between top and bottom, which indicated that that version of the model was not using reviews as expected.

In retrospect, the change in direction is clear: before, the focus was feature engineering, and after the migration, trying to manually do better math on the features lost its shine. This freed the team for higher-level problems, such as improving the optimization objective.