Lyft: fraud, from shallow to deep

01 · CONTEXT

Fraud as classification, and a bug that exposed the system

In fraud, Lyft's main interest is in classification algorithms that distinguish good users from fraudulent users on the platform. The case follows the evolution of these models and of the engineering around them, told by a member of the fraud team, covering logistic regression, GBDT and deep learning.

The author says he deployed the fraud decision service along with his first pull request, which removed a duplicated line of coefficients from a hand-written logistic regression. That small bug exposed limitations of a system built for simple, readable business rules.

Inconsistencies in model and feature definitions between the offline prototypes, built in notebooks, and the production service happened more often. These inconsistencies led to months-long back-and-forth cycles between engineering and science, which ended up blocking production.

1. The diagram shows the offline prototype, the decision service and the application connected to it.

2. The production service's model and feature definitions were inconsistent with those of the offline prototype.

3. The decision service was monolithic: it combined feature ingestion and preprocessing, model execution and anti-fraud challenges.

02 · THE BASELINE

Logistic regression: simple, readable and limited

Logistic regression remains the basic classifier in many industries. Its coefficients indicate how much a feature correlates with the probability of fraud, and, when diagnosing errors, the features that contribute most are those with the largest terms in the logit. The team also logged each prediction's feature importances to debug and evaluate performance.

The cost of this simplicity was performance: linear models did not capture all the information the features carried through linear correlations. To capture higher-order interactions between features, they would have to be created by hand and added as new features.

03 · GBDT

Natural interactions and precision at fixed recall

Among the options explored, the team chose gradient-boosted decision trees (GBDT), trained with the XGBoost library, for ease of use and efficiency.

In logistic regression, the decision boundary is a hyperplane in feature space, which means good features need to have a strong linear relationship with the probability of fraud. In GBDT, the boundaries are smoothed versions of a set of boxes in higher dimensions, which allows encoding more complicated interactions.

The measured gain was in relative precision: GBDT had more than a 60% relative precision gain over the previous model, at the same operating recall and with the same feature set.

04 · FROM NOTEBOOK TO PRODUCTION

Interpreting, serializing and versioning

Ensembles are generally harder to interpret and analyze than simple logistic regression. The team found substitutes in boosted-tree-specific feature importances and in model explainers: the former help prioritize what to take to production, and the explainers show the most influential features in a specific prediction.

Another obstacle was productionizing the GBDTs, for which the team did not have a good way. It evaluated tools such as PMML, but most did not support cutting-edge models and did not capture details such as feature encoding and transformations in the pipelines. The team built a library based on the standardized scikit-learn API and on pickle serialization, with extra safeguards on package requirements as a simple attempt at model versioning.

The text says it uses TensorFlow deep learning models in production, for performance and because they work with signals that are hard to turn into features. GBDTs do not handle sequential inputs well, such as histories of transactions, rides and app activity.

05 · THE PLATFORM

From monolith to services around the models

The text also describes an architecture change: the model execution and feature serving moved out of the original decision service, and services built around them meet the needs of other teams.

1. The diagram starts from the application, which calls the decision service.

2. Model execution and feature serving moved out of the original decision service.

3. The services built around them meet the needs of other teams.

4. Most models run synchronously, with hard execution time limits; at some points, asynchronous execution unlocked the use of more complicated features and models.

The author also redesigned the holdback and experimentation framework to measure the system's impact hierarchically. Before, measurements were rule-based and it was impractical to evaluate the system as a whole; with the current framework, it can be evaluated at the level of the anti-fraud challenge and of each rule or model.

There is also an organizational dimension: a siloed mindset, in which the scientist stops working after the feature and model prototype is built. The author says he abandoned the working model in which all feature and model production was handed off to engineering.