Airbnb: home valuation

01 · CONTEXT

Predicting the value of a listing before it earns revenue

Airbnb recognized that data products are expensive to build. Projects such as personalized search ranking and smart pricing each required a lot of dedicated data science and engineering time.

The post's case is customer lifetime value (LTV) prediction, which captures the projected value of a user over a fixed time horizon, usually in dollars. In marketplaces, knowing LTV makes it possible to allocate the budget across marketing channels more efficiently. The team went beyond history and predicted the LTV of new listings with machine learning.

02 · OVERVIEW

Four steps, one tool for each

Data scientists are usually used to feature engineering, prototyping, and model selection, but taking a prototype to production requires data engineering skills they may not have. The post is organized into four topics, each with the tool used.

In Airbnb's practice, feature engineering usually meant writing Hive queries from scratch, and these pipelines were rarely reusable. Zipline gathers features validated by other people in previous projects and, when several go into training, performs the joins by key and fills in the history of the training set.

1. The data scientist starts from the feature repository and reuses the ones that have already been validated.

2. The repository performs the joins by key and fills in the history to assemble the training set.

3. In the notebook, a pipeline specifies the transformations by feature type and the model to train.

4. AutoML tools come in to speed up model selection.

5. With the notebook integrated, the ML Automator converts it into a production pipeline.

03 · PROTOTYPE AND MODEL

Pipelines, AutoML, and the choice of XGBoost

The model used more than 150 features. Among them, the price features include nightly rate, cleaning fee, and price relative to similar listings; the quality features include review ratings, number of reviews, and amenities.

Pipelines ensure that data is transformed consistently in training and scoring, which solves a common inconsistency problem when translating a prototype into production. They also separate the transformations from the model fitting, and swapping the estimator is the path to model selection.

Insurance and credit analysis require interpretability to avoid discriminating against customers. In the LTV case, the team used AutoML tools to speed up selection and saw XGBoost significantly outperform baseline models such as average response, ridge regression, and a single decision tree. Since the goal was to predict values, they accepted a model that favors flexibility over interpretability.

04 · PRODUCTION

From notebook to daily pipeline

A production pipeline is quite different from a prototype on a laptop: questions arise about periodic retraining, efficient scoring of many examples, and monitoring performance over time. For that, Airbnb built the ML Automator, which translates a Jupyter notebook into an Airflow data pipeline pipeline, designed for people with little data engineering experience.

The contract is small. A model configuration says where the training table is, how much compute to allocate, and how the scores are computed; the fit function defines training and the transform function becomes a Python UDF for distributed scoring.

1. The data scientist delivers a notebook with the model configuration and the fit and transform functions.

2. After the notebook is integrated, the ML Automator wraps the trained model in a Python UDF and creates the Airflow pipeline.

3. The pipeline runs as a daily batch job, which encapsulates data serialization, retraining scheduling, and distributed scoring.

4. Training reads the training table indicated in the configuration, with the compute defined there.

5. The transform function, wrapped as a UDF, scores the examples in a distributed way when needed.

6. The scores feed decisions such as budget allocation across marketing channels.

05 · LESSONS

Lower cost and limits of the account

According to the author, combining the tools greatly shortened the development cycle, and the cost of developing models dropped significantly. The notebook-driven design lowers the barrier to entry, and notebooks used in production are guaranteed to be correct, self-documenting, and up to date.

At the time of the post, other teams were already exploring ML product ideas with the same approach.