DoorDash Sibyl (prediction service)

01 · CONTEXT

A prediction service for the whole platform

DoorDash's platform uses data to power models that curate search results, assign couriers, and recognize fraud. These models serve several microservices that depend on data-driven predictions, which requires a robust prediction service.

Sibyl was designed to handle all real-time predictions and focus on predicting. Feature computation, training pipelines, and storage of features and models live in other services and repositories.

The expectation was hundreds of thousands of predictions per second. Moreover, to convince other services to call it, Sibyl had to be so fast that calling it was better than each service making its own predictions.

02 · ARCHITECTURE

The lifecycle of a request

Clients talk to Sibyl over gRPC. The service obtains models and features from independent repositories and can, optionally, send predictions to Snowflake for offline evaluation. The path below follows a typical request.

1. A client service sends the request over gRPC, with one or more sets of features to predict on.

2. For each model in the request, Sibyl fetches the model and its configuration (required features, default values, and type) from an in-memory cache, loaded when the service started.

3. The service looks for feature values that did not come in the request, for all models and sets at once.

4. Each missing feature is fetched from the feature store, a Redis cache; if it still does not exist, the default value from the model's configuration applies.

5. With all features gathered, Sibyl makes one asynchronous prediction per set and fires off a coroutine per shadow model, without waiting for them to finish.

6. Finally, it builds the protobuf object with the predictions and returns it to the client.

The post also mentions, as a possible future expansion, a separate model evaluator for computing predictions of complex models; in the initial version, this function would stay inside Sibyl itself.

03 · IMPLEMENTATION DECISIONS

Speed comes from native code, parallelism, and caching

To optimize speed, models are stored in their native format and the prediction calls to the frameworks are made in C++. The service, written in Kotlin, uses the Java Native Interface (JNI) to call the LightGBM and PyTorch predictions implemented in C++.

Another priority was for predictions to run in parallel and for threads to be doing computation while features were being fetched. Kotlin coroutines help with this: they are not tied to a thread and suspend while waiting, freeing the thread for other work.

Models are fetched when the service starts and kept in cache in memory, without needing to be loaded on every request. Features, by contrast, are fetched on every prediction.

04 · BATCHES AND SHADOW

Batch predictions and candidate models

Each request can carry a variable number of feature sets. Batch predictions let you send and receive from 1 to N predictions at once, which greatly reduces the number of calls.

With shadow predictions, a team uses one model for the official predictions and, asynchronously, predicts the same data with other models. This way you can analyze the effectiveness of the candidates without swapping the official model.

For the test, the team split the request with all the stores into batches of reasonable size. The optimal size turned out to be around 100 to 200 stores, and batches of 10 and 20 stores worsened latency. The comparison below summarizes what the post observed and the team's hypotheses.

The team also evaluated compressing requests, to reduce the number of packets on the network.

05 · ROLLOUT

From load test to production

In the search service, every restaurant displayed needs to be scored and ranked first. The post uses this service, one of the company's most demanded, as the basis of the load test.

1. In the load test, when the search service was about to rank a restaurant with its own logic, it also fired an asynchronous thread that called Sibyl.

2. The asynchronous calls did not slow down the search endpoints, and the result returned by Sibyl was never used.

3. In this test, Sibyl handled more than 100,000 predictions per second.

4. Production began with fraud and courier payment models, with lower estimated QPS and less strict latency.

5. Search was not among the first to migrate: its models would still require migrating the feature values to Sibyl's feature store.

The post was published on June 29, 2020 and says that, since March of that year, the fraud and courier payment models officially use Sibyl. After adoption, the team observed a 3x drop in latency compared with the old prediction service.

According to the post, the migration was close to finished: all models but five already called Sibyl. The team was still adding support for different types of features and models, such as embedded features, used mainly in neural networks, and composite models, which are chains of submodels and expressions called compute nodes.