Harmful content detection

01 · CONTEXT AND REQUIREMENTS

What the system needs to decide

Social networks moderate harmful content to comply with the law and keep a good experience for users, a topic tied to regulatory compliance. The exercise starts from a social network of posts and asks for a system that detects all harmful content, from nudity and violence to terrorism and human trafficking.

The scope is text and image, which makes the case, according to the script, a multimodal content classification problem, typical of ML system design.

The system can automatically remove what it considers harmful with confidence and demote content that is likely harmful. There is a small team of moderators with limited capacity. For automatic removal, the prompt does not accept removing anything without 95% confidence that the content is harmful.

Errors are costly in both directions: someone whose content is removed by mistake is unhappy and may leave the platform, and someone exposed to too much harmful content may also leave.

The prompt assumes 1B posts per day, with fewer than 1% harmful, and 50k labeled examples for initial training, half harmful and half benign.

02 · GOAL AND METRIC

What the platform cares about

The script reminds us that the business goal is distinct from the loss function. Maximizing removed content encourages flagging as much as possible, and maximizing accuracy is a poor proxy for harm, subject to class imbalance.

The script adopts minimizing the number of views of harmful content, subject to precision guarantees. This goal answers the question of classifying right away or waiting: the longer you wait, the more views accumulate. The script also notes that harmful content becomes easier to detect the more people are exposed to it.

03 · DATA AND SIGNALS

Scarce labels, abundant signals

Data comes in layers. Supervised labels are few and the most valuable set. User reports work as an approximate label, probably correlated with the true one. Comments allow training a model that predicts comments from the post body, which produces post representations more useful than the body alone.

Because harmful content is rare, training on the raw distribution would make the model learn to predict "not harmful" for everything and still reach 99% accuracy. The script's answer combines balanced sampling in training with loss weighting to tune the trade-off between precision and recall.

The most informative features are the text and images of the post. Behavioral ones, such as the ratio of shares per view, change quickly, and so the post needs to be classified again after it is created. Creator features combine embeddings, rich but slow, with real-time counts, fast but noisy.

04 · MODEL

Multimodal fusion and multiple tasks

The starting point is a simple baseline: according to the script, a logistic regression with the numeric and embedding features is already a good start.

The script adopts the multimodal transformer and leaves cost for the inference discussion.

Training uses two separate heads: the main task, classifying harmful or not, and predicting the probability of reports. In production, only the main task head is used. The main loss includes a small logarithmic term over views, because views follow a power-law distribution and heavily viewed content would dominate the loss.

Instead of a softmax across content types, the script prefers a multi-label formulation, because the same harmful content can have several classes at once.

05 · INFERENCE AND HUMAN REVIEW

From post to action

With 1B posts per day, the multimodal transformer is large and expensive, probably requiring GPUs. The script proposes a lightweight, distilled model with fewer features, which quickly filters what obviously does not violate, leaving the heavy model for the rest. This also provides a knob to increase or reduce resource consumption.

1. The author publishes a post and the post service receives it.

2. Every new post enters the queue and first goes through the lightweight classifier.

3. If the filter score is high enough, the post goes on to the heavy classifier, which stores and reuses encodings in the cache.

4. The classifier output goes through a calibration layer that guarantees the required precision.

5. The decision reaches the action service, which can delete the post, flag it for review, or act otherwise.

6. Flagged posts wait in the review queue, and moderators review them with limited capacity.

Many posts use identical images or text, and storing and reusing the encodings avoids redundant passes through the model.

After calibration, the action system can delete the post, flag it for review, or act otherwise, and moderators have limited capacity. In the design, flagged posts wait in a queue, a topic of message queues.

The model is also triggered again when important events happen, such as reports, many views, or dislike comments. A lightweight update network merges the behavioral signals with the heavy embeddings.

06 · EVALUATION AND DRIFT

Measure and track what changes

In online evaluation, two models run side by side, candidate and control, and only one acts. Sampling posts at random would require many examples to measure a prevalence below 1%. So each model's score guides the subsampling of extreme cases, which are reweighted to avoid bias.

Offline, the suggested metrics are PR-AUC and Recall@Precision95: the first is a stable indicator of the model's overall strength, and the second aligns with the action threshold, with more variance.

1. Reports, reactions, and comments arrive as signals and the trigger decides to reclassify the post.

2. The trigger reuses the image and text encodings from the cache, which do not need to be recomputed.

3. A lightweight network merges the new behavioral signals into the heavy embedding, and calibration must handle repeated classifications of the same content.

4. In the design, posts under review reach the moderators, whose labels are recorded and replicated into the training set.

5. Training uses the training set and publishes a new model to the update network, while the monitor receives the decisions to track prevalence and drift.

Harmful content evolves quickly, partly because of people looking for new ways to get around the systems. Meta reports that, in traditional approaches, it takes several months to collect and label examples of each new type. The Few-Shot Learner (FSL) is described as a large, multimodal, multilingual model, trained on few or no examples, that adapts to new types in weeks instead of months.

FSL learns multimodal information from the whole post, with text, image, URL, and more. It can also be used to test a new set of likely violations and understand whether the proposed definitions make sense.

Meta evaluated the effect by the difference in prevalence of harmful content, that is, the percentage of views of violating content that people see, which matches the goal of minimizing views.