Incident · Mid
Zillow Offers (2021): pricing model leads to the shutdown of the business
Real incident. Key points: concept drift, uncertainty, and the loop between model and market.
Short answer
Real incident. Key points: concept drift, uncertainty, and the loop between model and market. Zillow shut down Zillow Offers, its home buying and reselling program (iBuying). The WIRED report, the public source for this case, gathers what the company said about the reason. The topic is a machine learning model that underpinned the business's purchasing decisions.
01 · CONTEXT
A price model at the center of the business
Zillow shut down Zillow Offers, its home buying and reselling program (iBuying). The WIRED report, the public source for this case, gathers what the company said about the reason. The topic is a machine learning model that underpinned the business's purchasing decisions.
According to the report, Zillow believed it had the secret to iBuying in the Zestimate, an algorithm launched in 2006 and trained on millions of US home valuations before being used to estimate the price of homes Zillow itself bought.
In the simplified design, the seller asks for an offer and the offers service queries the price model, which estimates the home's value.
Once the purchase closes, the home goes into the company's inventory and stays there while it is renovated and sold.
The business premise was to predict the resale price three to six months ahead, the time to renovate and sell.
02 · THE PREMISE
What the business demanded of the model
Executive Rich Barton told analysts that Zillow's iBuying premise was to accurately predict home prices three to six months ahead, and that this horizon reflected the time to renovate and sell the purchased homes.
The margin also weighed: according to the report, Zillow believed that, for the program to be profitable, the estimates needed to be more precise, with an error of a few thousand dollars.
In defense of the algorithm, a company spokesperson said it remained confident in the Zestimate and cited a median error of 1.9% for listed homes and 6.9% for off-market homes.
03 · THE DRIFT
When the market changed and the forecast did not keep up
According to the report, the predictions proved inaccurate in the turbulent 2021 housing market.
| Trimestre | Sales versus forecast |
|---|---|
| Second quarter | 5.8% above expectations |
| Third quarter | 5 to 7% below the forecast |
The term that usually describes this decoupling, in which the current market stops resembling what the model learned, is concept drift; the report does not use the expression, and the connection is ours.
Resale came in below forecast: in the third quarter, Zillow sold for 5 to 7% less than it had predicted.
On the buying side, the report notes that Zillow was paying above the market average in September and kept up the pace in October, when competitors slowed down.
In Phoenix, Zillow had its most active buying week in early October 2021, within the goal of buying 5,000 homes a month by 2024. Then, suddenly, it stopped buying. An Insider analysis from October 2021 found that nine out of ten homes bought there were listed at a lower price than the purchase price.
04 · SCOPE AND UNCERTAINTY
Outside the model's limits
Analyst Tomasz Piskorski says that Zillow, in seeking market share quickly, may have gone beyond the limits of the algorithm's capability, and that there was no problem as long as the business stayed within the limits of the business model and bought standardized homes, which are easier to sell.
| Fator | What the report records |
|---|---|
| Volatile market | Inaccurate predictions in the turbulent market |
| More complex homes | More risk in lower-quality or more complex homes, with more noise in the data |
| Contractor shortage | Difficulty reselling at the expected pace |
| Buying pace | Competitors slowed down; Zillow kept up the pace |
05 · OUTCOME
The shutdown of the program
Barton stated that the unpredictability in price forecasting exceeded what the company anticipated and that continuing to scale Zillow Offers would bring too much volatility to earnings and the balance sheet.
He admitted that the observed error rate was much more volatile than the company thought possible. Zillow shut down the iBuying arm and said it would cut 25% of its workforce.
06 · DESIGN LESSONS
Don't depend on a single number
The facts above suggest a design question: what measures the gap between the forecast and the price the market actually paid, and with what delay? When that answer is slow to arrive, early signals and an automatic brake on the buying pace are candidates for handling the delay, with observabilidade by home segment.
In this example design, the model delivers to the buying policy the prediction and also an uncertainty band.
Each decision is recorded along with the prediction that motivated it, so it can be compared later.
When the home is sold, the realized price is compared with the original prediction, by segment.
The measured error becomes an alert and also brakes the buying policy when it falls outside the tolerated band.
| Sinal | What decides |
|---|---|
| Predicted versus realized error, by segment | Reduce or pause purchases in that segment |
| Prediction uncertainty band | Limit how much inventory the company takes on |
| Performance outside the training range | Require human review before buying |
| Homes stuck in inventory | Trigger a tied-up capital alert |
Frequently asked questions
What is Zillow Offers (2021): pricing model leads to the shutdown of the business?
Real incident. Key points: concept drift, uncertainty, and the loop between model and market.
What does the Zillow Offers (2021): pricing model leads to the shutdown of the business design cover?
It is organized in these sections: A price model at the center of the business, What the business demanded of the model, When the market changed and the forecast did not keep up, Outside the model's limits, The shutdown of the program, and Don't depend on a single number.
How long is Zillow Offers (2021): pricing model leads to the shutdown of the business and what level is it?
It takes about 12 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.