iFood: recommendation and inference platform

01 · CONTEXT AND PROBLEM

Recommending on the home of a delivery app

iFood receives thousands of simultaneous visits and, on each visit, the app loads dozens of pieces of content, among restaurants, dishes, grocery stores, and products, to guide the choice of the person ordering. The recommendation systems that serve this environment in real time must support a high load with high availability and resilience.

Cost is the central problem. According to the team, even a simple model, such as a matrix factorization, can require a large number of machines to serve recommendations in real time. In addition, customers change preferences depending on context, such as location and time of day, and this requires extra effort per model and service, increasing cost. The team's question is how to scale dozens of different models and combine them with the customer's context.

iFood's answer is to pre-index the predictions of multiple models, which allows combining them with other sources of information in real time.

02 · INGESTION

From candidate to stored prediction

The platform is an architecture of microservices separated into two domains, Ingestion and Server. We start with Ingestion, which produces the predictions before any customer visit.

1. The candidate generation pipeline filters what is relevant and leaves the candidates in files on S3.

2. The candidates are processed by multiple models on SageMaker, which uses Batch Transform to scale the predictions.

3. The prediction output is written to multiple files on S3.

4. Creating the files triggers the Lambda function Feature Publisher, which reads the predictions.

5. The Feature Publisher publishes the predictions to a Kafka topic.

6. The Feature Consumer consumes the topic and inserts the predictions into the Personalization Store.

A prior candidate generation flow filters the relevant entities for each context. The same pipeline selects the customers that must be updated daily and a reduced set of restaurants for each one.

Each model has independently automated training and prediction with SageMaker Pipelines, and Batch Transform scales the predictions. The messages go through a Kafka topic, a message queue that decouples whoever generates the predictions from whoever writes them.

The Go consumer reads tens of millions of messages from the topic and writes to the Personalization Store, a NoSQL database that holds about ten billion predictions.

03 · SERVER

Read what is already ready and reorder on the spot

The Personalization Store uses a well-partitioned schema to ensure read scalability. This is the read that the server performs on each app session.

1. The server receives all the app's calls.

2. For each user, the server retrieves the predictions of the various models stored in the Personalization Store.

3. The response is enriched with the precomputed predictions and goes through ranking models applied on the spot.

4. The architecture allows combining the predictions with other sources of information in real time.

04 · ML PLATFORM

How models reach production

Another AWS post, whose title mentions hundreds of models, describes the other side of the problem. Previously, data scientists created models in notebooks and published them to services, and engineering teams struggled to integrate them into production systems. To overcome this, iFood built an internal ML platform.

In this platform, ML Go! is responsible for deploying SageMaker pipelines and endpoints. See the path from code to prediction.

1. The flow starts with the ML Go! CI/CD pipeline sending the training and deployment code artifacts.

2. Each model available for deployment is automatically registered in the Model Registry, with versioning and tracking.

3. In batch prediction cases, the pipeline creates a batch transform job.

4. In real-time inference, the pipeline deploys the model to an endpoint.

5. The Gateway is a single abstraction for connecting to SageMaker endpoints and jobs.

In the most recent architecture, the separation of responsibilities inside the endpoints, with the Inference Components feature, makes serving faster and more efficient.

05 · RESULTS AND TRADE-OFFS

What iFood reports and what to ask

iFood reports 6x more scalability, from 500 to more than 3000 requests per second. It also reports savings of about 50% in the cost of serving content on the home, since most of the information is pre-indexed, and an improvement of about 10% in list conversion.