Bot detection

01 · CONTEXT AND REQUIREMENTS

An adversarial problem with few labels

Bots are malicious automated actors that do many things on a platform, such as creating fake accounts, sending friend requests and messages, posting duplicate or harmful content, and spreading misinformation. This case designs a bot detection system for a social network, with the goal of separating legitimate user activity from automated or malicious activity.

The interviewer's answers define the design. They may say that, without remediation, bots account for 50% or more of all actions, but that simple heuristics reduce that to under 1%. The assumed scale is 500M daily active users, with 1-2 sessions per day. Classifying a real user as a bot drives away good users and reduces their engagement, and although appeals exist, reviewing them is expensive.

Some constraints dominate the design. The problem is adversarial: bot authors constantly adapt to evade detection, and the newest bots are far more sophisticated and harder to detect than those that sent requests at a much higher frequency than humans. Prevalence is high and the system sits in front of many interaction surfaces, so compute is a central constraint. Finally, labeled data is scarce and expensive, and investigators manually verify accounts in the hundreds per week.

02 · OBJECTIVE

What to optimize: impact, not count

Before choosing the model, you need to translate the problem into a business objective, which is distinct from the loss function and the ML objective. The source compares possible formulations of the objective.

The ML objective, in turn, is straightforward: at its core it is a binary classification problem, in which an account is classified as bot or not bot.

03 · DATA AND FEATURES

Scarce labels, diverse signals

Labels come from sources of different strength. Investigator labels are the highest-quality source, but because they are few, it makes sense to send accounts near decision boundaries or with new patterns to manual review. User reports give noisy but scalable labels, and successful appeals give high-confidence negative labels.

It is also possible to propagate labels through the social graph: known malicious IPs provide additional positive labels, and mass account creation from similar origins indicates bot farms. The source itself warns that you need to be careful not to propagate errors through the graph.

Features are organized into groups. Activity patterns include posting cadence, measured by the distribution of time between events, and burst detection, which are spikes of activity beyond human capacity; these patterns are computed over several time windows, such as hour, day, and week. Network signals include the ratio of followers to following, which is often skewed in bots, the speed of connection formation, and the clustering coefficient. Account metadata covers age, verification, profile completeness, login frequency, and device diversity.

Content signals are among the weakest, because they are the easiest to fake and the most visible: if the system always takes down one type of content and not another, adversaries start producing more of the other. Temporal activity patterns suffer less from this.

Not all features are available right at signup. The model should handle missing data well and possibly use different feature sets for new and established accounts.

04 · MODELING

Start simple and join graph with sequence

The design combines two approaches. For obvious bots or already-observed signatures, supervised techniques and nonparametric approaches are used, with high precision. For the hardest-to-detect or never-detected ones, unsupervised techniques are used. Responses combine bans, demotions, and limits: when confident, the account is removed; when uncertain, visibility or the volume of interactions with other users is reduced.

For modeling, the source proposes starting with something simple, seeing where the gaps are, and adding sophistication where it matters. The baseline is not throwaway work: it becomes the canary that warns when a future version gets worse.

Bot detection is more a challenge of evaluating behavior than of judging content alone. The proposal joins two complementary views: the graph shows who the account connects with and how densely, and the sequence shows what the account does over time. The source chooses a GRU over a transformer because the target is suspicious behavior, not the nuance of language.

1. The main trunk splits into two specialized branches, one for the event sequence and one for the graph, and then joins each account's views.

2. The graph branch uses GraphSAGE, which is inductive, over a compact k-hop subgraph, typically k=2. Neighbor sampling limits the fan-out of celebrity accounts.

3. The sequence branch takes about 200 recent events, buckets the timestamps into five-minute intervals, embeds the event type, and passes the sequence through a bidirectional GRU with a small hidden state.

4. The output vectors of the two branches meet in a single cross-attention layer.

5. The representation goes through a tiny three-layer MLP that emits a raw risk score.

6. The raw score is calibrated to become a bot probability; one option is to fit a sigmoid to the raw score, with two parameters learned on a hold-out set.

Training a large model with little data and a single binary objective is a recipe for overfitting and unstable training, which is why the source turns to pretraining. With much of the work done through self-supervision, fine-tuning is light: most layers are frozen, a few thousand reliable labels are used, and the final score is calibrated to read as a bot probability.

The loss function is weighted binary cross-entropy. Each sample's weight reflects how many real users the account could have bothered, and the weights are capped to avoid runaway gradients.

05 · INFERENCE AT SCALE

A lightweight filter in front of the expensive model

The main supervised model is expensive and probably requires GPUs. To reduce compute, the model is optimized, for example with quantization-aware training, which reduces memory while maintaining accuracy, and computations are reused: account graph embeddings can be stored for a short time and activity-pattern encodings can be reused, in a cache.

In addition, instead of running the expensive model on every account, a two-stage architecture is adopted: a lightweight filter, which is a fast logistic regression with basic signals, and the full heavy model, reserved for accounts that pass the initial filter. The lightweight filter is trained in a teacher-student fashion to approximate the output of the heavy model.

The cascade reduces compute by 80-90% while maintaining detection quality, and, if the system is short on resources, the lightweight model's threshold is raised so the heavy model runs on fewer accounts, at some cost in effectiveness.

1. In the diagram, account actions pass through the platform surfaces and reach the event queue. Account signup and user reports are among the triggers that fire the lightweight model.

2. The lightweight filter is a fast logistic regression, with basic signals such as posting frequency, account age, and device patterns, and it quickly identifies the obviously legitimate accounts.

3. The full graph-and-sequence model is reserved for accounts that pass the initial filter.

4. Account graph embeddings can be stored for a short time, and activity-pattern encodings can be reused.

5. Unsupervised techniques cover the bots that are hardest to detect or not yet detected.

6. When confident, the account is removed; where there is no certainty, the account's visibility is reduced or the volume of interactions with other users is limited.

The system needs to handle account evaluation in real time and updates as patterns evolve. The lightweight model is triggered by several events: account signup, crossing of suspicious-activity thresholds, network changes, and user reports. The caching strategy is suitable for re-evaluation, as long as the account's neighborhood in the network has not changed much.

When re-evaluating, there are data challenges. In positive suppression, because effective detection removes accounts before they show their full behavior, the model risks learning that behavioral data indicates legitimacy. In survivorship bias, the remaining bots become more sophisticated and require constant adaptation. To deal with this, the source uses holdouts, which collect valuable data and hide from adversaries the information that the system's actions would reveal, but care is needed so that bot authors do not exploit them to get past detection.

06 · EVALUATION AND DRIFT

Measuring without enough labels and without looking only at the past

In online evaluation, the candidate and control models run side by side, acting on only one of them, which requires additional labeling because the models flag different populations of accounts. With low prevalence, random sampling would require huge samples, so importance sampling based on scores is used: accounts near decision boundaries where the models disagree are sampled more, and obvious cases are subsampled, with reweighting.

1. With low bot prevalence, random sampling would require huge samples; therefore, the choice of which accounts to label starts from the models' scores.

2. Sampling gives more weight to accounts near decision boundaries where the models disagree, and subsamples the obvious cases, with scores near 0.0 or 1.0, with reweighting.

3. Investigators manually verify accounts in the hundreds per week, and these labels are the highest-quality source.

4. User reports give noisy but scalable labels, and successful appeals give high-confidence negative labels.

5. The system should distinguish investigator labels from system-generated labels and weight them accordingly.

6. Labels serve, among other uses, to build test sets that reflect the real sophistication of bots and to calibrate confidence thresholds.

The main online indicator is the reduction in bot interactions per legitimate user, subject to keeping the false-positive rate below 1% in account restrictions. To iterate faster, proxies are used: user reports, appeal success rate, and network effects, such as friend-request acceptance and message response rate.

Offline, the primary metric Precision@Recall90 is aligned with the operating threshold of automated actions, PR-AUC is a stable indicator of overall discriminative power, and impact-weighted metrics use the potential exposure of users. As diagnostics, there is the early detection rate, which is the fraction of bots caught in the first 24 hours of activity, and network coverage. Performance across demographics and geographies is also evaluated to identify bias.

One of the biggest challenges in an adversarial setting is drift: future bot behavior is expected to differ from what has already been seen. One of the simplest ways to reflect this offline is to stratify the validation sets by time. The source also regularly tests against simulated behaviors and known evasion techniques, maintaining a red team dataset.

On calibration, uncalibrated scores make actions diverge greatly from a naive probability, since 0.6 does not mean the account is a bot 60% of the time; calibrating helps respect the guardrails and stabilizes the system across migrations and new deploys, and the source considers Platt scaling a great option, given the expensive labels and the few relevant points on the curve.

In anomaly detection, isolation forests isolate anomalies with fewer random splits than normal data and are good for point anomalies, but struggle with contextual anomalies. Autoencoders cost more, but capture temporal dependencies and sequential patterns that isolation forests miss. The proposal is an ensemble that uses isolation forests for fast triage of obvious outliers and autoencoders on the accounts that get past that filter.

07 · COUNTERPOINT IN PRODUCTION

Cloudflare: the same problem, request by request

There are systems like this in production. Cloudflare describes a score from 0 to 100 for each request, indicating whether it came from a human (high score) or an automated program (low score). The scope differs from the Hello Interview case, in which the target is to classify an account as bot or not bot.

Cloudflare's technical requirements are demanding: compute the score on every request, even above 10 million per second, without delaying processing by more than 100 microseconds.

The platform uses five complementary detection mechanisms, each producing its own score, and combines them into a single score that goes to the Firewall.

Two platform details deserve a note. First, because of the high certainty, requests classified by heuristics train the ML models. Second, several CatBoost models run in shadow mode on every request, while one runs in active mode and influences the final score. New models are deployed in seconds through the Quicksilver configuration database.