Real architecture · Senior
Stripe Radar (payment fraud)
Real production architecture. Key points: network of signals across merchants, decision in under 100 ms, explainability, and rules + ML.
Short answer
Real production architecture. Key points: network of signals across merchants, decision in under 100 ms, explainability, and rules + ML. When someone clicks to buy, Radar, Stripe's fraud prevention solution, kicks in until the transaction is confirmed. It evaluates more than a thousand features of the transaction to estimate the probability of fraud, lets the good ones through and blocks…
01 · CONTEXT
Deciding in a fraction of a second
When someone clicks to buy, Radar, Stripe's fraud prevention solution, kicks in until the transaction is confirmed. It evaluates more than a thousand features of the transaction to estimate the probability of fraud, lets the good ones through and blocks the risky ones or diverts them to additional checks. It is a typical case of ML system design with decision in real time: the response comes back in under a hundred milliseconds.
Two factors make the problem hard. The tool has to be accurate, fast and cheap per transaction, balancing the blocking of bad transactions against false positives, that is, good payments that get blocked. In addition, fraud is rare, on the order of one in every thousand payments. Among the billions of legitimate payments made on Stripe, Radar incorrectly blocks only 0.1%.
02 · THE DECISION
From click to decision
According to the author, the team relies on the breadth of Stripe's network, described as its greatest strength, to identify fraudulent transactions.
The buyer confirms the purchase and the payment request reaches the API.
Before the transaction is confirmed, the API queries the scoring service, which evaluates the transaction's features to estimate the probability of fraud.
The design includes the custom allow or block rules that the merchant creates to tailor Radar to their business.
The result is one of three outcomes: let the transaction through, block it or divert it to additional security checks.
In the diagram, model training uses the transaction history and delivers the model to the scoring service, outside the payment request path.
After understanding a payment's risk, whoever uses Radar can improve the quality of the data they send or create custom allow or block rules for their business.
03 · MODEL EVOLUTION
From logistic regression to a single neural network
The team started with relatively simple ML models, such as logistic regression, and moved to more complex models, such as deep neural networks, as Stripe's network grew and ML technology advanced.
The most recent change described in the text came in mid-2022: from a Wide & Deep ensemble, made up of an XGBoost model and a deep neural network, to a deep-neural-network-only model. According to the team, XGBoost was incompatible at scale with more advanced ML techniques it wanted to adopt and slowed the retraining pace because it is not very parallelizable.
Simply removing XGBoost would have caused a 1.5% drop in recall, a regression considered unacceptable. Making the network too deep risks overfitting, in which the model memorizes random noise in the features. The team adopted a multi-branch neural network architecture inspired by ResNeXt.
| Criterion | With XGBoost (Wide & Deep) | Deep neural network only |
|---|---|---|
| Arquitetura | Ensemble of an XGBoost model with a deep neural network | Multi-branch deep neural network inspired by ResNeXt |
| Treino | Slower retraining, since XGBoost is not very parallelizable | Training time reduced by more than eighty-five percent, to under two hours |
| More training data | No disproportionate quality improvements with XGBoost | Each increase in the training set yields disproportionate quality improvements |
04 · EXPLAINABILITY
Explaining matters as much as detecting
All ML models are black boxes to some extent, and deep neural networks even more so. That is why it is hard to explain to users why Radar scores transactions the way it does, a trade-off the team accepted when choosing neural networks over simpler techniques.
In 2020, risk insights was created, which shows the user which features of a transaction contributed to it being declined. The factors can include whether the cardholder's name matches the email provided and how many cards have already been associated with an IP address.
Scoring produces the transaction's outcome and the factors that weighed most in it.
Risk insights shows the merchant which features of the transaction contributed to the decline.
With the explanation, the merchant can create custom allow or block rules.
A table view of the features that most raised and lowered the score serves engineers to debug support cases.
According to the text, this table view shows the features that contributed most to raising and lowering the fraud score, and engineers use it internally to debug support cases.
05 · FEATURES AND DATA
More signals, more data
The team reviews past fraud attacks in detail, with investigation reports that try to get inside fraudsters' minds. Not every idea works: a boolean indicator for the business being under a distributed fraud attack did not improve the model's performance as much as expected.
To test whether more data is enough, the team increased the training transaction data tenfold and still saw significant improvements in the model. Training time grows linearly with data size, but the neural-network-only architecture, faster to train, made this less of a problem.
Frequently asked questions
What is Stripe Radar (payment fraud)?
Real production architecture. Key points: network of signals across merchants, decision in under 100 ms, explainability, and rules + ML.
What does the Stripe Radar (payment fraud) design cover?
It is organized in these sections: Deciding in a fraction of a second, From click to decision, From logistic regression to a single neural network, Explaining matters as much as detecting, and More signals, more data.
How long is Stripe Radar (payment fraud) and what level is it?
It takes about 17 minutes to read and is rated Senior level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.